Trouble compiling NWChem with CUDA enabled


Gets Around
I am trying to compile NWChem 6.3 (2014-01-28) on a CentOS 6.5-x86_64 system with CUDA-5.5 installed. The 2.08MB make.log file ends with the following 8 lines:
/usr/local/nwchem-6.3/lib/LINUX64/libtce.a(hybrid.o): In function `check_device_':
hybrid.c:(.text+0x2): undefined reference to `armci_me'
hybrid.c:(.text+0x8): undefined reference to `armci_master'
/usr/local/nwchem-6.3/lib/LINUX64/libtce.a(hybrid.o): In function `device_init_':
hybrid.c:(.text+0x2f): undefined reference to `armci_me'
hybrid.c:(.text+0x35): undefined reference to `armci_master'
collect2: ld returned 1 exit status
make: *** [all] Error 1

This version of CentOS/NWChem compiles correctly, and runs, w/o CUDA. Cuda 5.5 installed correctly and runs cuda samples so I assume that is working.
Here are the environment variables that were set:
export NWCHEM_TOP=/usr/local/nwchem-6.3
export NWCHEM_TARGET=LINUX64
export USE_MPI=y
export USE_MPIF=y
export LIBMPI="-pthread -m64 -lmpi_f90 -lmpi_f77 -lmpi -ldl"
export MPI_LOC=/usr/lib64/openmpi
export MPI_LIB=/usr/lib64/openmpi/lib
export MPI_INCLUDE=/usr/include/openmpi-x86_64
export NWCHEM_MODULES=all
export LARGE_FILES=TRUE
export USE_NOFSCHECK=TRUE
export TCE_CUDA=y
export CUDA_LIBS="-L/usr/local/cuda-5.5/lib64 -lcudart"
export CUDA_FLAGS="-arch sm_20 "
export CUDA_INCLUDE="-I. -I/usr/local/cuda-5.5/include"
export CUDA=nvcc
export PATH=$PATH:/usr/local/cuda-5.5/bin
export FC=gfortran
export CC=gcc

Then compiled with make nwchem_config, and make >& make.log
I guess the big question is, what are the references to "armci" ?

Forum Vet
Jkweller,
This is a know problem of the Cuda part of NWChem (it works straight out of the box only for certain values of ARMCI_NETWORK)
The next release will contain a fix for it.
If you are interested in testing the patch for this, I can pass it to you in a few days.
Edo

Gets Around
Edo,
Thanks for the offer. I would be glad to test a CUDA patch on the system I mentioned. This is a fairly typical setup, so others may also find it useful if it works.
John Keller

Gets Around
Not having received any further information on a patch, can you suggest values for ARMCI_NETWORK that, if set, will give a successful compilation along the lines of my original post?

Forum Regular
In your original settings I didn't see any ARMCI_NETWORK setting at all (when building with the build_nwchem script that is fine but otherwise it isn't). In any case you seem to have MPI so you could try ARMCI_NETWORK=MPI-TS. I have no way of testing whether that will actually work in your setup, but it asks ARMCI to build using MPI as the communication fabric. I would be surprised if that wouldn't work.

Huub

Forum Vet
Quote:Jwkeller Apr 17th 4:44 pm
Not having received any further information on a patch, can you suggest values for ARMCI_NETWORK that, if set, will give a successful compilation along the lines of my original post?


Jwkeller

I am not quite ready with the patch yet

I might be able to pass it to you in few days.

Cheers, Edo

Gets Around
Huub,
Adding the line "export ARMCI_NETWORK=MPI-TS " to list in the first box above, and using the same make commands, gives exactly the same make.log ouput and no nwchem executable.
John Keller

Gets Around
...and using the build_nwchem script also give the same error messages.
John K.

Forum Vet
Quote:Jwkeller Apr 28th 11:07 pm
...and using the build_nwchem script also give the same error messages.
John K.

Yes, This is the expected behavior.
If you can wait a few more days, I should be able to issue a patch before the end of next week.
Edo

Forum Vet
John
I have just uploaded a patch that should allow to use the Cuda part of NWChem (TCE/CCSD(T))

http://nwchemgit.github.io/images/Cuda.patch.gz

To patch your NWChem 6.3 source, you may use the following steps

1) cd $NWCHEM_TOP/src
2) wget http://nwchemgit.github.io/images/Cuda.patch.gz
3) gzip -d Cuda.patch
4) patch -p0 < Cuda.patch
5) recompile the directories util and tce/ccsd_t
6) relink

Gets Around
Thank you, Edo. I will try this. John Keller

Just Got Here
This worked great for me. Thanks!


Forum >> NWChem's corner >> Compiling NWChem