Skip to content

Latest commit

 

History

History
157 lines (118 loc) · 3.81 KB

BUILD.md

File metadata and controls

157 lines (118 loc) · 3.81 KB
Raw

Build from source

1. Linux

1.1 Use prebuilt container image for developers

Docker is the future

build/run make -j$(nproc)

1.2 Use customized container image for developers

Configure container repository and tag:

export REPO=myhost/myns/myimage
export TAG=deeprec-py3.6-cu114-ubuntu18.04

Build and push customized developer image:

build/docker-build-dev \
--build-arg TF_REPO=https://github.com/alibaba/DeepRec.git \
--build-arg TF_COMMIT=b73e41b8399038373da0f94d204673c911c4dbc1
docker push myhost/myns/myimage:developer-deeprec-py3.6-cu114-ubuntu18.04

Build HybridBackend on customized developer image:

TAG={tag} build/run make -j$(nproc)

NOTE

docker-ce >= 20 is required for BuildKit support.

For better debugability:

  1. Edit /etc/systemd/system/multi-user.target.wants/docker.service
  2. Under the [Service] tag, put those lines:
Environment="BUILDKIT_STEP_LOG_MAX_SIZE=1000000000"
Environment="BUILDKIT_STEP_LOG_MAX_SPEED=10000000"
  1. Then restart docker daemon:
systemctl daemon-reload
systemctl restart docker.service

1.3 Build from scratch

Requirements:

  • Ubuntu 18.04 or later (64-bit)
  • Python 3.6 or later
  • Pip 19.0 or later
  • TensorFlow 1.15 or TensorFlow 1.14
  • For GPU support, CUDA SDK 11.3 or later is required

Build & install arrow:

cd build/arrow/
ARROW_USE_CXX11_ABI=${HYBRIDBACKEND_USE_CXX11_ABI} \
ARROW_HDFS=ON \
ARROW_S3=ON \
./build.sh

Install TensorFlow and other requirements, see Dockerfiles for more detail.

Configure & build:

# Build GPU releated functions.
export HYBRIDBACKEND_WITH_CUDA=ON
# For TensorFlow 1.15, zero-copy is supported.
export HYBRIDBACKEND_WITH_ARROW_ZEROCOPY=ON
# Use below command to verify C++ ABI of installed TensorFlow.
python -c 'import tensorflow as tf; print(tf.sysconfig.get_compile_flags())'
# Must be consistent with installed TensorFlow.
export HYBRIDBACKEND_USE_CXX11_ABI=0

make -j$(nproc)

2. macOS

Requirements:

  • macOS 11.0 or later (x86 64-bit)
  • Python 3.7 or later
  • Pip 19.0 or later
  • Tebnsorflow 1.15 or TensorFlow 1.14
  • Other libraries installed by brew

Build & install arrow:

cd build/arrow/
ARROW_USE_CXX11_ABI=${HYBRIDBACKEND_USE_CXX11_ABI} \
ARROW_HDFS=ON \
ARROW_S3=ON \
./build.sh

Install TensorFlow and other requirements:

brew install wget [email protected] [email protected] utf8proc zstd snappy re2 thrift zlib
brew uninstall grpc abseil || true
export PATH="/usr/local/opt/[email protected]/bin:$PATH"
export LDFLAGS="-L/usr/local/opt/[email protected]/lib"
export PKG_CONFIG_PATH="/usr/local/opt/[email protected]/lib/pkgconfig"

pip3.7 install -i https://mirrors.aliyun.com/pypi/simple/ \
    tensorflow==1.14 \
    "pybind11[global]"

Configure & build:

# Only build CPU releated functions.
export HYBRIDBACKEND_WITH_CUDA=OFF
# For TensorFlow 1.14, zero-copy is not supported.
export HYBRIDBACKEND_WITH_ARROW_ZEROCOPY=OFF
# Use below command to verify C++ ABI of installed TensorFlow.
python -c 'import tensorflow as tf; print(tf.sysconfig.get_compile_flags())'
# Must be consistent with installed TensorFlow.
export HYBRIDBACKEND_USE_CXX11_ABI=0

# Set path of thridparty libraries.
export PYTHON=python3.7
export PYTHON_INCLUDE=/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/Current/include
export PYTHON_LIB=/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/Current/lib
export PYTHON_IMPL=python3.7
export PYTHON_IMPL_FLAG=m
export SSL_LIB=/usr/local/opt/[email protected]/lib
export RE2_LIB=/usr/local/opt/re2/lib
export THRIFT_LIB=/usr/local/opt/thrift/lib
export UTF8PROC_LIB=/usr/local/opt/utf8proc/lib
export SNAPPY_LIB=/usr/local/opt/snappy/lib
export ZSTD_LIB=/usr/local/opt/zstd/lib
export ZLIB_LIB=/usr/local/opt/zlib/lib

make -j$(nproc)