running error for deepmd-gpu

[zhangyongsdu]

Hi,

I have followed the instruction (https://github.com/deepmodeling/deepmd-kit/blob/master/doc/install-tf.1.12.md) to install C++ interface for Tensorflow with gpu support. The installation processes for both Tensorflow-gpu=1.12.0 and deepmd-kit are successful. However, once I try to run the installed dp_train, I will get the following errors. I noticed the errors were mentioned early, but the fixed proposed (#10 and #20) are can not fix my issues. I am not sure if the error is correlated with the version of protobuf. In installing Python interface of tensorflow, I have to install protobuf-3.7.0 (at least >3.6.1), but I have to download 3.6.0 when compiling protobuf. This is because if I use >3.6.1, errors will occur during install deepmd-kit say something about protobuf is not compatible. As as result, if protobuf version is the reason for the error, how can I deal with it?

Thanks for your help.

errors:

Traceback (most recent call last):
File "/home/yzhang1/lj94/deepmd_root/bin/dp_train", line 16, in
from deepmd.Model import NNPModel
File "/projects/lj94/deepmd_root/bin/../lib/deepmd/Model.py", line 21, in
op_module = tf.load_op_library(module_path + "libop_abi.so")
File "/home/yzhang1/lj94/tensorflow_venv/lib/python3.6/site-packages/tensorflow/python/framework/load_library.py", line 60, in load_op_library
lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: /projects/lj94/deepmd_root/lib/deepmd/libop_abi.so: undefined symbol: _ZN10tensorflow12OpDefBuilder4AttrENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE

installing process

1. module load: cuda/9.0 python/3.6.6-gcc5 cmake/3.10.2-gcc5
   unset PYTHONPATH

2. virtualenv --no-site-packages -p python3 /home/yzhang1/lj94/tensorflow_venv

3. source /home/yzhang1/lj94/tensorflow_venv/bin/activate

4. pip install --upgrade tensorflow-gpu==1.12.0

5. python -c "import tensorflow as tf; tf.enable_eager_execution(); print(tf.reduce_sum(tf.random_normal([1000, 1000])))"

6. wget https://github.com/bazelbuild/bazel/releases/download/0.15.0/bazel-0.15.0-dist.zip

7. mkdir bazel-0.15.0
   cd bazel-0.15.0

8. unzip ../bazel-0.15.0-dist.zip

9. ./compile.sh

10. export PATH=pwd/output:$PATH

11. cd ../

12. git clone https://github.com/tensorflow/tensorflow tensorflow12 -b v1.12.0 --depth=1

13. cd tensorflow12

14. ./configure

Please specify the location of python. [Default is /home/yzhang1/tensorflow_venv/bin/python]:
Please input the desired Python library path to use. Default is [/home/yzhang1/tensorflow_venv/lib/python3.6/site-packages]
Do you wish to build TensorFlow with Apache Ignite support? [Y/n]: n
Do you wish to build TensorFlow with XLA JIT support? [Y/n]: n
Do you wish to build TensorFlow with OpenCL SYCL support? [y/N]: n
Do you wish to build TensorFlow with ROCm support? [y/N]: n
Do you wish to build TensorFlow with CUDA support? [y/N]: y
Please specify the CUDA SDK version you want to use. [Leave empty to default to CUDA 9.0]:
Please specify the location where CUDA 9.0 toolkit is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: /usr/local/cuda/9.0
Please specify the cuDNN version you want to use. [Leave empty to default to cuDNN 7]:

Please specify the location where cuDNN 7 library is installed. Refer to README.md for more details. [Default is /usr/local/cuda/9.0]:

Do you wish to build TensorFlow with TensorRT support? [y/N]: n
Please specify the NCCL version you want to use. If NCCL 2.2 is not installed, then you can use version 1.3 that can be fetched automatically but it may have worse performance with multiple GPUs. [Default is 2.2]: 1.3
Please note that each additional compute capability significantly increases your build time and binary size. [Default is: 3.5,7.0]: 3.7,6.0,7.0
Do you want to use clang as CUDA compiler? [y/N]: n
Please specify which gcc should be used by nvcc as the host compiler. [Default is /usr/local/gcc/5.4.0/bin/gcc]:
Do you wish to build TensorFlow with MPI support? [y/N]: n
Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -march=native]:
Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]: n

16. bazel build --config=opt --config=cuda --verbose_failures //tensorflow:libtensorflow_cc.so //tensorflow:libtensorflow_framework.so

rm -r -f /home/yzhang1/lj94/tensorflow_root

17. mkdir -p /home/yzhang1/lj94/tensorflow_root
18. mkdir -p /home/yzhang1/lj94/tmp/proto
19. sed -i 's;PROTOBUF_URL=.*;PROTOBUF_URL="https://mirror.bazel.build/github.com/google/protobuf/archive/v3.6.0.tar.gz\";g' /home/yzhang1/lj94/tensorflow12/tensorflow/contrib/makefile/download_dependencies.sh
20. run the followring command under

cd /home/yzhang1/lj94/tensorflow12

./tensorflow/contrib/makefile/download_dependencies.sh

21. cd /home/yzhang1/lj94/tensorflow12/tensorflow/contrib/makefile/downloads/protobuf/

22. ./autogen.sh

23. ./configure --prefix=/home/yzhang1/lj94/tmp/proto/

24. make

25. make install

26. mkdir /home/yzhang1/lj94/tmp/eigen

27. cd /home/yzhang1/lj94/tensorflow12/tensorflow/contrib/makefile/downloads/eigen

28. mkdir build_dir

29. cd build_dir

30. cmake -DCMAKE_INSTALL_PREFIX=/home/yzhang1/lj94/tmp/eigen/ ../

31. make install

32. mkdir /home/yzhang1/lj94/tmp/nsync

33. cd ../../nsync

34. mkdir build_dir

35. cd build_dir

36. cmake -DCMAKE_INSTALL_PREFIX=/home/yzhang1/lj94/tmp/nsync/ ../

37. make

38. make install

39. cd ../../absl

40. bazel build

41. mkdir -p /home/yzhang1/lj94/tensorflow_root/include/

42. rsync -avzh --include '/' --include '.h' --exclude '*' absl /home/yzhang1/lj94/tensorflow_root/include/

43. cd ../../../../..

mkdir /home/yzhang1/lj94/tensorflow_root/lib
cp bazel-bin/tensorflow/libtensorflow_cc.so /home/yzhang1/lj94/tensorflow_root/lib/
cp bazel-bin/tensorflow/libtensorflow_framework.so /home/yzhang1/lj94/tensorflow_root/lib/
cp /home/yzhang1/lj94/tmp/proto/lib/libprotobuf.a /home/yzhang1/lj94/tensorflow_root/lib/
cp /home/yzhang1/lj94/tmp/nsync/lib64/libnsync.a /home/yzhang1/lj94/tensorflow_root/lib/

mkdir -p /home/yzhang1/lj94/tensorflow_root/include/tensorflow
cp -r bazel-genfiles/* /home/yzhang1/lj94/tensorflow_root/include/
cp -r tensorflow/cc /home/yzhang1/lj94/tensorflow_root/include/tensorflow
cp -r tensorflow/core /home/yzhang1/lj94/tensorflow_root/include/tensorflow
cp -r third_party /home/yzhang1/lj94/tensorflow_root/include
cp -r /home/yzhang1/lj94/tmp/proto/include/* /home/yzhang1/lj94/tensorflow_root/include
cp -r /home/yzhang1/lj94/tmp/eigen/include/eigen3/* /home/yzhang1/lj94/tensorflow_root/include
cp -r /home/yzhang1/lj94/tmp/nsync/include/*h /home/yzhang1/lj94/tensorflow_root/include

cd /home/yzhang1/lj94/tensorflow_root/include
find . -name "*.cc" -type f -delete

rm -fr /home/yzhang1/lj94/tmp/proto /home/yzhang1/lj94/tmp/eigen /home/yzhang1/lj94/tmp/nsync

Install deepmd-kit

1. cd /home/yzhang1/lj94

2. git clone https://github.com/deepmodeling/deepmd-kit.git deepmd-kit

3. cd deepmd-kit
   deepmd_source_dir=pwd

cd $deepmd_source_dir/source
mkdir build
cd build

5. cmake -DTENSORFLOW_ROOT=/home/yzhang1/lj94/tensorflow_root -DCMAKE_INSTALL_PREFIX=/home/yzhang1/lj94/deepmd_root ..

6. make

7. make install

[dingye18]

Did you checked the library links? Followed the instructions of #20 .
Also, you can recompile tensorflow for pip package with the configurations before. And install it to your python library.

[zhangyongsdu]

Thanks, fixs for #20 work here.