Building NWChem 6.1 with Intel 11.1 and Openmpi 1.6 on a IB cluster running CentOS 6.2

Just Got Here

12:43:52 PM PDT - Tue, May 15th 2012

Dear All,

I have been trying with various degrees of (un)success to build NWChem 6.1 on a IB cluster running CentOS 6.2 (gcc 4.4.4), with distro provided IB libraries, using the intel compiler 11.1 (rev 080), MKL 10.2 (update 7) and the openmpi 1.6 built with the intel compiler and IB.

I gave up (at least for now) using scalapack because I always get errors (of the undefined reference kind) and now I am only using MKL blas and lapack.

Here is the script I use to build:

################################################

#!/bin/bash



. /u/local/Modules/default/init/modules.sh

module load intel openmpi/1.6

module li



export NWCHEM_TOP=/u/local/downloads/nwchem/nwchem-6.1

export NWCHEM_TARGET=LINUX64

echo "NWCHEM_TOP=$NWCHEM_TOP"

echo "NWCHEM_TARGET=$NWCHEM_TARGET"



export ARMCI_NETWORK=OPENIB

export IB_HOME=/usr

export IB_INCLUDE=/usr/include/infiniband/

export IB_LIB=/usr/lib64

export IB_LIB_NAME="-libumad -libverbs -lpthread"

export MSG_COMMS=MPI

export TCGRSH=/usr/bin/ssh

echo "ARMCI_NETWORK=$ARMCI_NETWORK"

echo "IB_HOME=$IB_HOME"

echo "IB_INCLUDE=$IB_INCLUDE"

echo "IB_LIB=$IB_LIB"

echo "IB_LIB_NAME=$IB_LIB_NAME"

echo "MSG_COMMS=$MSG_COMMS"

echo "TCGRSH=$TCGRSH"



export USE_MPI=y

export USE_MPIF=y

export USE_MPIF4=y

export MPI_LOC=/u/local/intel/11.1/openmpi/1.6.0

export MPI_LIB=$MPI_LOC/lib

export MPI_INCLUDE=$MPI_LOC/include



export LIBMPI="-L$MPI_LIB  -lmpi -lrdmacm -libverbs -lrt -lnsl -lutil -ldl -lm -Wl,--export-dynamic -lrt -lnsl -lutil"

echo "USE_MPI=$USE_MPI"

echo "USE_MPIF=$USE_MPIF"

echo "USE_MPIF4=$USE_MPIF4"

echo "MPI_LOC=$MPI_LOC"

echo "MPI_LIB=$MPI_LIB"

echo "export MPI_INCLUDE=$export MPI_INCLUDE"

echo "LIBMPI=$LIBMPI"



export NWCHEM_MODULES="all python"

export LARGE_FILES=TRUE

export USE_NOFSCHECK=TRUE

export LIB_DEFINES=-DDFLT_TOT_MEM=16777216

echo "NWCHEM_MODULES=$NWCHEM_MODULES"

echo "LARGE_FILES=$LARGE_FILES"

echo "LARGE_FILES=$LARGE_FILES"

echo "LIB_DEFINES=$LIB_DEFINES"



export PYTHONHOME=/usr

export PYTHONVERSION=2.6

export USE_PYTHON64=y

export PYTHONLIBTYPE=so

echo "PYTHONHOME=$PYTHONHOME"

echo "PYTHONVERSION=$PYTHONVERSION"

echo "USE_PYTHON64=$USE_PYTHON64"

echo "PYTHONLIBTYPE=$PYTHONLIBTYPE"

 

# NOTE, the definition of PYTHONLIBTYPE and the change in config/makefile.h as shown below 

# is done to prevent the error:

#

#  File not found:  '/usr/lib64/python2.6/config/libpython2.6.a' 

#

# see http://nwchemgit.github.io/Special_AWCforum/st/id349/#post_1188

#

sed -i 's/libpython$(PYTHONVERSION).a/libpython$(PYTHONVERSION).$(PYTHONLIBTYPE)/g' config/makefile.h



export MKLLIB=/u/local/compilers/intel/11.1/080/mkl/lib/em64t



export BLASOPT="$MKLLIB/libmkl_solver_ilp64_sequential.a -Wl,--start-group   $MKLLIB/libmkl_intel_ilp64.a $MKLLIB/libmkl_sequential.a $MKLLIB/libmkl_core.a -Wl,--end-group   -lpthread -lm"

echo "BLASOPT=$BLASOPT"



export FC="ifort -i8"

export CC="icc -DMKL_ILP64"

echo "FC=$FC"

echo "CC=$CC"



echo "### START MAKING####"



#echo "make realclean" 

#make realclean



echo "cd $NWCHEM_TOP/src"

cd $NWCHEM_TOP/src

pwd

echo "make nwchem_config" 

make nwchem_config

echo " "

echo "########"

echo " "

echo "make"

make 

################################################

And here is the error in make log file:

make nwchem.o stubs.o
make[1]: warning: -jN forced in submake: disabling jobserver mode.
ifort -i8 -i8 -g  -I.  -I/u/local/downloads/nwchem/nwchem-6.1/src/include -I/u/local/downloads/nwchem/nwchem-6.1/src/tools/install/include -DEXT_INT -DLINUX -DLINUX64 -DPARAL
LEL_DIAG -DCOMPILATION_DATE="'`date +%a_%b_%d_%H:%M:%S_%Y`'" -DCOMPILATION_DIR="'/u/local/downloads/nwchem/nwchem-6.1'" -DNWCHEM_BRANCH="'6.1'"  -c -o nwchem.o nwchem.F
ifort -i8 -i8 -g  -I.  -I/u/local/downloads/nwchem/nwchem-6.1/src/include -I/u/local/downloads/nwchem/nwchem-6.1/src/tools/install/include -DEXT_INT -DLINUX -DLINUX64 -DPARAL
LEL_DIAG -DCOMPILATION_DATE="'`date +%a_%b_%d_%H:%M:%S_%Y`'" -DCOMPILATION_DIR="'/u/local/downloads/nwchem/nwchem-6.1'" -DNWCHEM_BRANCH="'6.1'"  -c -o stubs.o stubs.F
ifort -i8 -i8 -g  -Wl,--export-dynamic  -L/u/local/downloads/nwchem/nwchem-6.1/lib/LINUX64 -L/u/local/downloads/nwchem/nwchem-6.1/src/tools/install/lib  -o /u/local/downloads
/nwchem/nwchem-6.1/bin/LINUX64/nwchem nwchem.o stubs.o -lnwctask -lccsd -lmcscf -lselci -lmp2 -lmoints -lstepper -ldriver -loptim -lnwdft -lgradients -lcphf -lesp -lddscf -ld
angchang -lguess -lhessian -lvib -lnwcutil -lrimp2 -lproperty -lnwints -lprepar -lnwmd -lnwpw -lofpw -lpaw -lpspw -lband -lnwpwlib -lcafe -lspace -lanalyze -lqhop -lpfft -ldp
lot -lnwpython -ldrdy -lvscf -lqmmm -lqmd -letrans -lpspw -ltce -lbq -lcons -lperfm -ldntmc -lccca -lnwcutil -lga -lpeigs -lperfm -lcons -lbq -lnwcutil /usr/lib64/python2.6/c
onfig/libpython2.6.so /u/local/compilers/intel/11.1/080/mkl/lib/em64t/libmkl_solver_ilp64_sequential.a -Wl,--start-group  /u/local/compilers/intel/11.1/080/mkl/lib/em64t/libm
kl_intel_ilp64.a /u/local/compilers/intel/11.1/080/mkl/lib/em64t/libmkl_sequential.a /u/local/compilers/intel/11.1/080/mkl/lib/em64t/libmkl_core.a -Wl,--end-group -lpthread -
lm  -llapack  -lblas   -L/u/local/intel/11.1/openmpi/1.6.0/lib -L/u/local/intel/11.1/openmpi/1.6.0/lib  -lmpi -lrdmacm -libverbs -lrt -lnsl -lutil -ldl -lm -Wl,--export-dynam
ic -lrt -lnsl -lutil  -L/usr/lib64 -libumad -libverbs -lpthread  -lnwcutil  -lpthread -lutil -ldl  
/u/local/downloads/nwchem/nwchem-6.1/lib/LINUX64/libnwpwlib.a(Parallel-mpi.o): In function `parallel2d_finalize_':
Parallel-mpi.F:(.text+0x37): undefined reference to `mpi_comm_free_'
Parallel-mpi.F:(.text+0x47): undefined reference to `mpi_group_free_'
Parallel-mpi.F:(.text+0x57): undefined reference to `mpi_comm_free_'
Parallel-mpi.F:(.text+0x67): undefined reference to `mpi_group_free_'
/u/local/downloads/nwchem/nwchem-6.1/lib/LINUX64/libnwpwlib.a(Parallel-mpi.o): In function `parallel3d_finalize_':

I would gladly post the entire make.log and the tools/build/config.log and the tools/build/armci/config.log files if you could tell me how to upload files.

Could you please advise?

Thanks,

Raffaella.

Forum Vet

1:36:35 PM PDT - Tue, May 15th 2012
Need to link with library for MPI Fortran bindings
Raffaella, Since you have defined USE_MPIF, you need to include the library for MPI Fortran bindings (e.g. for openmpi "-lmpi_f90 -lmpi_f77" and for mpich2 "-lmpichf90") I have spotted another definition in your environmental that might cause you problems. Please define FC=ifort and not FC="ifort -i8". The -i8 definition (plus more options) is automatically picked up. Ciao, Edo

Just Got Here

4:50:52 PM PDT - Tue, May 15th 2012

ifort: error #10106

Ciao Edo,

Thanks for your reply.

Yes, the MPILIB were off, I changed them to:

export LIBMPI="-L$MPI_LIB -lmpi_f90 -lmpi_f77 -lmpi -ldl -Wl,--export-dynamic -lnsl -lutil"

and place the IB one in:

export IB_LIB_NAME="-lrdmacm -libumad -libverbs -ldl -lpthread -lm -Wl,--export-dynamic -lrt -lnsl -lutil"

(a little redundant).

However now I get a segmentation violation:

 Making all in md5
make[2]: warning: -jN forced in submake: disabling jobserver mode.
icc -DMKL_ILP64 -c -I.  -I/u/local/downloads/nwchem/nwchem-6.1/src/include -I/u/local/downloads/nwchem/nwchem-6.1/src/tools/install/include -DEXT_INT -DLINUX -DLINUX64 -DPARALLEL_DIAG -DCOMPILATION_DATE="'`date +%a_%b_%d_%H:%M:%S_%Y`'" -DCOMPILATION_DIR="'/u/local/downloads/nwchem/nwchem-6.1'" -DNWCHEM_BRANCH="'6.1'" -DJOBTIMEPATH=\"/u/local/downloads/nwchem/nwchem-6.1/bin/LINUX64\" -DBYTE_SHORT_PACKING   -O3 -funroll-loops -ffast-math   -O0 -g -g  -o md5wrap.o md5wrap.c
icc -DMKL_ILP64 -c -I.  -I/u/local/downloads/nwchem/nwchem-6.1/src/include -I/u/local/downloads/nwchem/nwchem-6.1/src/tools/install/include -DEXT_INT -DLINUX -DLINUX64 -DPARALLEL_DIAG -DCOMPILATION_DATE="'`date +%a_%b_%d_%H:%M:%S_%Y`'" -DCOMPILATION_DIR="'/u/local/downloads/nwchem/nwchem-6.1'" -DNWCHEM_BRANCH="'6.1'" -DJOBTIMEPATH=\"/u/local/downloads/nwchem/nwchem-6.1/bin/LINUX64\" -DBYTE_SHORT_PACKING   -O3 -funroll-loops -ffast-math   -O0 -g -g  -o md5.o md5.c
icc: command line warning #10006: ignoring unknown option '-ffast-math'
icc: command line warning #10120: overriding '-O3' with '-O0'
icc: command line warning #10006: ignoring unknown option '-ffast-math'
icc: command line warning #10120: overriding '-O3' with '-O0'
make[3]: warning: -jN forced in submake: disabling jobserver mode.
Got lock on /u/local/downloads/nwchem/nwchem-6.1/lib/LINUX64/libnwcutil.lock
ar r /u/local/downloads/nwchem/nwchem-6.1/lib/LINUX64/libnwcutil.a md5wrap.o md5.o
ranlib /u/local/downloads/nwchem/nwchem-6.1/lib/LINUX64/libnwcutil.a
/u/local/downloads/nwchem/nwchem-6.1/lib/LINUX64/libnwcutil.a
ifort: error #10106: Fatal error in /u/local/compilers/intel/11.1/080/bin/intel64/fortcom, terminated by segmentation violation
compilation aborted for util_version.F (code 1)
make[1]: *** [/u/local/downloads/nwchem/nwchem-6.1/lib/LINUX64/libnwcutil.a(util_version.o)] Error 1
make: *** [libraries] Error 1

I am not sure if the problem above is related to the intel compiler.

I will now remove "-i8" from the FC definition (should I also remove "-DMKL_ILP64" from CC?) and try again. I will post any outcome.

Do you know of anyone who built nwchem successfully with the particular version of the intel compiler I am using (11.1.080)? I could try the 12.1 version fo the compiler.

Thanks,

Raffaella.

Forum Vet

4:16:48 PM PDT - Wed, May 16th 2012
need to shrink util_version.F
Raffaella, ifort 11 is fine. The problem is this the file $NWCHEM_TOP/src/uti/util_version.F. Since it's a very long file (but -- frankly speaking -- not very useful), ifort has some hiccups while processing it. Here is how to overcome the problem cd $NWCHEM_TOP/src/util head -1 util_version.F > new.F tail -1 util_version.F >>new.F mv util_version.F bigv.F mv new.F util_version.F make FC=gfortran cd .. make FC=gfortran link Please let me know if this works for you, too. Edo

Just Got Here

10:38:08 AM PDT - Thu, May 17th 2012

intel 12.1 (update 4) worked like a charm & question about possible alternative to .nwchemrc file

Hello Edo,

Thanks so much for your reply. I have not yet tried what you suggested but yesterday, after installing the latest update of the intel compiler (12.1.4), rebuilding the latest openmpi (1.6) with it, nwchem compiled as is with no problem after just the first attempt (!!!).

For posterity, here is the script I used:

#################################
#!/bin/bash

. /u/local/Modules/default/init/modules.sh
module load intel/12.1 openmpi/1.6
module li

export NWCHEM_TOP=/u/local/downloads/nwchem/nwchem-6.1
export NWCHEM_TARGET=LINUX64
echo "NWCHEM_TOP=$NWCHEM_TOP"
echo "NWCHEM_TARGET=$NWCHEM_TARGET"


export ARMCI_NETWORK=OPENIB
export IB_HOME=/usr
export IB_INCLUDE=/usr/include/infiniband/
export IB_LIB=/usr/lib64
#export IB_LIB_NAME="-lrdmacm -libumad -libverbs -ldl -lpthread -lm -Wl,--export-dynamic -lrt -lnsl -lutil"
export IB_LIB_NAME="-lrdmacm -libverbs -lrt -lnsl -lutil -ldl -lm -Wl,--export-dynamic -lrt -lnsl -lutil"
export MSG_COMMS=MPI
export TCGRSH=/usr/bin/ssh
echo "ARMCI_NETWORK=$ARMCI_NETWORK"
echo "IB_HOME=$IB_HOME"
echo "IB_INCLUDE=$IB_INCLUDE"
echo "IB_LIB=$IB_LIB"
echo "IB_LIB_NAME=$IB_LIB_NAME"
echo "MSG_COMMS=$MSG_COMMS"
echo "TCGRSH=$TCGRSH"

export USE_MPI=y
export USE_MPIF=y
export USE_MPIF4=y
export MPI_LOC=/u/local/intel/12.1/openmpi/1.6.0
export MPI_LIB=$MPI_LOC/lib
export MPI_INCLUDE=$MPI_LOC/include
export LIBMPI="-lpthread -L$MPI_LIB -lmpi_f90 -lmpi_f77 -lmpi"
echo "USE_MPI=$USE_MPI"
echo "USE_MPIF=$USE_MPIF"
echo "USE_MPIF4=$USE_MPIF4"
echo "MPI_LOC=$MPI_LOC"
echo "MPI_LIB=$MPI_LIB"
echo "export MPI_INCLUDE=$export MPI_INCLUDE"
echo "LIBMPI=$LIBMPI"

export NWCHEM_MODULES="all python"
export LARGE_FILES=TRUE
export USE_NOFSCHECK=TRUE
export LIB_DEFINES=-DDFLT_TOT_MEM=16777216
echo "NWCHEM_MODULES=$NWCHEM_MODULES"
echo "LARGE_FILES=$LARGE_FILES"
echo "LARGE_FILES=$LARGE_FILES"
echo "LIB_DEFINES=$LIB_DEFINES"

export PYTHONHOME=/usr
export PYTHONVERSION=2.6
export USE_PYTHON64=y
export PYTHONLIBTYPE=so
echo "PYTHONHOME=$PYTHONHOME"
echo "PYTHONVERSION=$PYTHONVERSION"
echo "USE_PYTHON64=$USE_PYTHON64"
echo "PYTHONLIBTYPE=$PYTHONLIBTYPE"


# NOTE, the definition of PYTHONLIBTYPE and the change in config/makefile.h as shown below
# is done to prevent the error:
#
#  File not found:  '/usr/lib64/python2.6/config/libpython2.6.a' 
#
# see http://nwchemgit.github.io/Special_AWCforum/st/id349/#post_1188
#
#sed -i 's/libpython$(PYTHONVERSION).a/libpython$(PYTHONVERSION).$(PYTHONLIBTYPE)/g' config/makefile.h
export MKLROOT=/u/local/compilers/intel/12.1/4/composer_xe_2011_sp1.10.319/mkl
export BLASOPT="-Wl,--start-group  $MKLROOT/lib/intel64/libmkl_intel_ilp64.a $MKLROOT/lib/intel64/libmkl_sequential.a $MKLROOT/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -lm"
echo "BLASOPT=$BLASOPT"

export FC=/u/local/compilers/intel/12.1/4/composer_xe_2011_sp1.10.319/bin/intel64/ifort
export CC=/u/local/compilers/intel/12.1/4/composer_xe_2011_sp1.10.319/bin/intel64/icc
echo "FC=$FC"
echo "CC=$CC"

echo "### START MAKING####"

#echo "make realclean"
#make realclean

echo "cd $NWCHEM_TOP/src"
cd $NWCHEM_TOP/src
pwd
echo "make nwchem_config"
make nwchem_config
echo " "
echo "########"
echo " "
echo "make FC=$FC CC=$CC"
make FC=$FC CC=$CC
#################################

Can I ask you whether instead of creating a .nwchemrc file in the users's $HOME would be OK to define environmental variables for: nwchem_basis_library, nwchem_nwpw_library, ffield, amber_1, amber_2, amber_3, amber_4, spce, charmm_s, charmm_x?

Also, do you think that not having used scalapack will greatly impact the performance of nwchem?

Raffaella.

Just Got Here

10:35:29 PM PDT - Sun, Oct 28th 2012
Thanks, I had this problem with the Intel Compiler 12.1 and your suggestion to cut down the size of util_version.F worked. I didn't even have to use gfortran. Andy Quote:Edoapra May 16th 4:16 pm Raffaella, ifort 11 is fine. The problem is this the file $NWCHEM_TOP/src/uti/util_version.F. Since it's a very long file (but -- frankly speaking -- not very useful), ifort has some hiccups while processing it. Here is how to overcome the problem cd $NWCHEM_TOP/src/util head -1 util_version.F > new.F tail -1 util_version.F >>new.F mv util_version.F bigv.F mv new.F util_version.F make FC=gfortran cd .. make FC=gfortran link Please let me know if this works for you, too. Edo

Forum >> NWChem's corner >> Compiling NWChem