compile nwchem-6-3 with openmpi1.6.5 & ifort

Clicked A Few Times

8:38:44 AM PDT - Mon, Jun 2nd 2014

Dear all,

I have been fighting like an animal to compile nwchem on my workstation (intel xeon, centos 6.5)
I have installed intel fortran compiler, intel cc, mkl.
I have compiled openmpi 1.6.5 with CC=icc CXX=icpc F77=ifort FC=ifort

But, when I try to compile nwchem - I see the following error:

checking for mpi.h... yes
checking for library containing MPI_Init... no
configure: error: test for COMEX_NETWORK=MPI_TS failed
configure: error: ../../ga-5-2/comex/configure failed for comex
make[1]: *** [build/config.status] Error 1
make: *** [libraries] Error 1

My script is following:

!/bin/tcsh

cd nwchem-6.3
setenv NWCHEM_TOP `pwd`
setenv NWCHEM_TARGET LINUX64

setenv USE_MPI y
setenv USE_MPIF y

openmpi version 1.6.5

setenv MPI_LOC /home/minenky/programs/openmpi_intel

setenv MPI_LIB $MPI_LOC/lib
setenv MPI_INCLUDE $MPI_LOC/include

setenv LIBMPI "-lmpi_f90 -lmpi_f77 -lmpi -ldl -Wl,--export-dynamic -lnsl -lutil"
"-pthread -L/opt/hwloc/lib -L/opt/ompi-1.6.3/lib -lmpi_f90 -lmpi_f77 -lmpi -ldl -lhwloc -Wl,--export-dynamic -lrt -lnsl -lutil -lm -ldl -Wl,-rpath -Wl,/opt/hwloc/lib -Wl,-rpath -Wl,/opt/hwloc/lib"

setenv LIBMPI "-lmpi -lmpigf -lmpigi -lrt -lpthread"
setenv LARGE_FILES TRUE

setenv LIB_DEFINES -DDFLT_TOT_MEM=16777216

setenv TCGRSH /usr/bin/ssh

blas

setenv MKLROOT /opt/intel/mkl
setenv BLASOPT "-L$MKLROOT/lib/intel64 -lmkl_intel_ilp64 -lmkl_sequential -lmkl_core -lpthread -lm"
setenv BLAS_LIB "$BLASOPT"
setenv BLAS_SIZE 8

setenv NWCHEM_MODULES "all"
cd $NWCHEM_TOP/src

make nwchem_config
make FC=ifort CC=icc >& make.log

Any kind of help is appreciated!

With kind regards,
Yury

Forum Vet

3:22:53 PM PDT - Mon, Jun 2nd 2014
Yury Could you post the output of the following command /home/minenky/programs/openmpi_intel/bin/mpif90 -show

Clicked A Few Times

10:17:06 PM PDT - Mon, Jun 2nd 2014
Edoapra, thank you for reply. The output: ifort -I/home/minenky/programs/openmpi_intel/include -I/home/minenky/programs/openmpi_intel/lib -Wl,-rpath -Wl,/home/minenky/programs/openmpi_intel/lib -Wl,--enable-new-dtags -L/home/minenky/programs/openmpi_intel/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi Quote:Edoapra Jun 2nd 3:22 pm Yury Could you post the output of the following command /home/minenky/programs/openmpi_intel/bin/mpif90 -show

Forum Vet

1:20:37 PM PDT - Tue, Jun 3rd 2014
Yury Your MPI settings are not compatible with your MPI installation. You should use the following (MPI_INCLUDE and MPI_LIB are the same ones use set, the only difference should be in LIBMPI) MPI_INCLUDE=/home/minenky/programs/openmpi_intel/include MPI_LIB=/home/minenky/programs/openmpi_intel/lib LIBMPI=" -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi" If you still get a failure, please give us access to the following files $NWCHEM_TOP/src/tools/build/config.log $NWCHEM_TOP/src/tools/build/armci/config.log

Clicked A Few Times

1:41:16 PM PDT - Thu, Jun 5th 2014
Thanks for helping! Just got another error - gn -w -g -vec-report1 -i8 -align -w -g -vec-report1 -O3 -prefetch -unroll -tpp7 -ip -xP -no-prec-div -xHost -no-prec-div -I. -I/home/minenky/programs/nwchem/nwchem-6.3/src/include -I/home/minenky/programs/nwchem/nwchem-6.3/src/tools/install/include -DEXT_INT -DLINUX -DLINUX64 -DIFCV8 -DIFCLINUX -DPARALLEL_DIAG -DMPI Parallel.F ifort -c -i8 -i8 -align -w -g -vec-report1 -i8 -align -w -g -vec-report1 -i8 -align -w -g -vec-report1 -i8 -align -w -g -vec-report1 -i8 -align -w -g -vec-report1 -O3 -prefetch -unroll -tpp7 -ip -xP -no-prec-div -xHost -no-prec-div -I. -I/home/minenky/programs/nwchem/nwchem-6.3/src/include -I/home/minenky/programs/nwchem/nwchem-6.3/src/tools/install/include -DEXT_INT -DLINUX -DLINUX64 -DIFCV8 -DIFCLINUX -DPARALLEL_DIAG -DMPI Parallel-mpi.F ifort: command line remark #10382: option '-xHOST' setting '-xSSE4.2' ifort: command line remark #10382: option '-xHOST' setting '-xSSE4.2' Parallel-mpi.F(29): #error: can't find include file: mpif.h Parallel-mpi.F(158): #error: can't find include file: mpif.h Parallel-mpi.F(264): #error: can't find include file: mpif.h Parallel-mpi.F(314): #error: can't find include file: mpif.h Parallel-mpi.F(456): #error: can't find include file: mpif.h Parallel-mpi.F(500): #error: can't find include file: mpif.h Parallel-mpi.F(538): #error: can't find include file: mpif.h Parallel-mpi.F(581): #error: can't find include file: mpif.h Parallel-mpi.F(624): #error: can't find include file: mpif.h Parallel-mpi.F(672): #error: can't find include file: mpif.h Parallel-mpi.F(729): #error: can't find include file: mpif.h Parallel-mpi.F(795): #error: can't find include file: mpif.h Parallel-mpi.F(839): #error: can't find include file: mpif.h Parallel-mpi.F(879): #error: can't find include file: mpif.h Parallel-mpi.F(927): #error: can't find include file: mpif.h Parallel-mpi.F(1004): #error: can't find include file: mpif.h Parallel-mpi.F(1093): #error: can't find include file: mpif.h Parallel-mpi.F(1126): #error: can't find include file: mpif.h Parallel-mpi.F(1175): #error: can't find include file: mpif.h Parallel-mpi.F(1223): #error: can't find include file: mpif.h Parallel-mpi.F(1274): #error: can't find include file: mpif.h Parallel-mpi.F(1339): #error: can't find include file: mpif.h Parallel-mpi.F(1412): #error: can't find include file: mpif.h Parallel-mpi.F(1459): #error: can't find include file: mpif.h Parallel-mpi.F(1505): #error: can't find include file: mpif.h Parallel-mpi.F(1558): #error: can't find include file: mpif.h Parallel-mpi.F(1625): #error: can't find include file: mpif.h Parallel-mpi.F(1754): #error: can't find include file: mpif.h make[4]: * [/home/minenky/programs/nwchem/nwchem-6.3/lib/LINUX64/libnwpwlib.a(Parallel-mpi.o)] Error 28 make[4]: * Waiting for unfinished jobs.... make[3]: * [optimized] Error 2 make[2]: * [subdirs] Error 1 make[1]: * [subdirs] Error 1 make[1]: * Waiting for unfinished jobs.... make: *** [libraries] Error 1 The problem is that ifort cannot compile the above mentioned file: arallel-mpi.F If I specify FC=mpifort in the compiling script, it does the job - but the resulting version of nwchem can not run in parallel normally.

Forum Regular

3:29:09 PM PDT - Thu, Jun 5th 2014
Hi Yuri, The problem you are seeing is caused by the fact that the location of the MPI include files is not specified on the compile line for Parallel-mpi.F. I.e. there should have been a flag "-I/home/minenky/programs/openmpi_intel/include" but it is not there. This flag should have been added to LIB_DEFINES in the makefile src/nwpw/nwpwlib/Parallel/GNUmakefile. I wonder why that didn't happen. Anyway as a work around you could add it manually (even though that still feels unsatisfactory). Huub

Clicked A Few Times

2:00:15 AM PDT - Fri, Jun 6th 2014

Hello Huub,
Thanks for helping! I have added the flags the script to include the info about openmpi_intel/include:

and got it compiled.. BUT -
I can not run the code. When I try to run it mpirun nwchem Carbene.nw it says:

 Unable to open nwchem.nw --- appending .nw

------------------------------------------------------------------------

nwchem: failed to open the input file        0

------------------------------------------------------------------------

There is an error in the input file

------------------------------------------------------------------------

For more information see the NWChem manual at 

http://nwchemgit.github.io/index.php/NWChem_Documentation

For further details see manual section:

But when I create an emty file nwchem.nw - it runs, I see it via top i, but nothing happens - no output from the code. Did anyone have such experience?

Clicked A Few Times

3:03:28 AM PDT - Fri, Jun 6th 2014
OK, finally found the mistake and got it compiled. I think I am getting closer to the final normally working version. The main problem now is following. When running it with openmpi (mpirun) it is almost 8 times slower on 8 cores comparing to running it on just one core which I find very strange. Any suggestions?

Forum Vet

10:41:33 AM PDT - Fri, Jun 6th 2014
What input file are you trying to run? It might be too small of a task to see it scaling

Gets Around

5:20:26 PM PDT - Fri, Jun 6th 2014
Yury, I am using GNU compilers and the default openmpi system package that comes with Ubuntu 14.04 rather than Intel tools, but I too had poor performance with MPI-TS. I normally build with ARMCI_NETWORK=SOCKETS but I wanted to try MPI-TS in an attempt to fix problems similar to those reported here: http://nwchemgit.github.io/Special_AWCforum/st/id1298/armci_malloc%3Amalloc_1_fail... I am using the most recent code snapshot in downloads: http://nwchemgit.github.io/download.php?f=Nwchem-6.3.revision25564-src.2014-05-03.tar.gz Running serially, the QA test case tce_polar_ccsd_small took 314 seconds wall clock time. With ARMCI_NETWORK=SOCKETS the same case ran in 139 seconds using 4 cores on a 4 core i7 system. That's OK scaling, not great, but my scaling is limited on most non-direct jobs because of slow disk I/O. I also tried an "mpi serial" job with the same build using mpirun -np 1 and that took 317 seconds wall clock time, basically unchanged from simple serial. With ARMCI_NETWORK=MPI-TS on the same hardware and OS the wall clock time went to 518 seconds on 4 cores for tce_polar_ccsd_small. That's slower than serial, as you experienced.

Forum >> NWChem's corner >> Compiling NWChem