Linking NWChem against Intel's MKL
Clicked A Few Times
7:39:34 PM PST - Wed, Jan 2nd 2013
i could be wrong on this but it seems that NWChem is not using the MKL it was linked against. I say this for two reasons.

1) The CPU time of all the test jobs seem for the NWChem built with the standard libraries and the one built with MKL are identical (+- 2%)

2) I have noticed that in the final linking step to generate NWChem binary "-lblas and -llapack" are there, which i suspect are the default libraries for NWChem

gfortran -L/home/joe/nwchem-6.1.1-src/lib/LINUX64 -L/home/joe/nwchem-6.1.1-src/src/tools/install/lib -o /home/joe/nwchem-6.1.1-src/bin/LINUX64/nwchem nwchem.o stubs.o -lnwctask -lccsd -lmcscf -lselci -lmp2 -lmoints -lstepper -ldriver -loptim -lnwdft -lgradients -lcphf -lesp -lddscf -ldangchang -lguess -lhessian -lvib -lnwcutil -lrimp2 -lproperty -lnwints -lprepar -lnwmd -lnwpw -lofpw -lpaw -lpspw -lband -lnwpwlib -lcafe -lspace -lanalyze -lqhop -lpfft -ldplot -ldrdy -lvscf -lqmmm -lqmd -letrans -lpspw -ltce -lbq -lcons -lperfm -ldntmc -lccca -lnwcutil -lga -lpeigs -lperfm -lcons -lbq -lnwcutil -L/home/joe/mkl/mkl/lib/intel64 -lmkl_scalapack_ilp64 -lmkl_cdft_core -lmkl_gf_ilp64 -lmkl_sequential -lmkl_core -lmkl_blacs_intelmpi_ilp64 -ldl -lpthread -lm -llapack -lblas -L/home/joe/mpich-install/lib -lmpich -lopa -lmpl -lrt -lpthread

The following is the script I use for environment variables

#!/bin/bash
export NWCHEM_TOP=$HOME/nwchem-6.1.1-src
export NWCHEM_TARGET=LINUX64
export USE_MPI=y
export USE_MPIF=y
export USE_MPIF4=y
export MPI_LOC=$HOME/mpich-install
export LIBMPI="-lmpich -lopa -lmpl -lrt -lpthread"
export MPI_LIB=$MPI_LOC/lib
export MPI_INCLUDE=$MPI_LOC/include
export LARGE_FILES=TRUE
export NWCHEM_MODULES=all
export MKLROOT=$HOME/mkl/mkl/lib/intel64
export BLASOPT="-L$MKLROOT -lmkl_scalapack_ilp64 -lmkl_cdft_core -lmkl_gf_ilp64 -lmkl_sequential -lmkl_core -lmkl_blacs_intelmpi_ilp64 -ldl -lpthread -lm"
#make clean
make FC=gfortran FFLAGS="-fdefault-integer-8 -m64"  nwchem_config
make FC=gfortran FFLAGS="-fdefault-integer-8 -m64"  HAS_BLAS=yes


Any ideas? Thanks in advance

Forum Regular
11:15:09 AM PST - Thu, Jan 3rd 2013
Hi DouDou,

I think that actually MKL is probably being used. However, you might not be able to see that easily for two reasons:
1. In the usual test jobs the BLAS calls take very little time. So using a much more efficient BLAS library will still not change the CPU time much, probably too little to see.
2. The presence of -llapack and -lblas is correct. Those libraries are included as MKL does not necessarily provide all LAPACK and BLAS routines. The routine that are not found in MKL are then picked up from the LAPACK and BLAS implementations in the source code. For this to work properly the LAPACK and BLAS libraries have to be listed after the MKL libraries, which you can easily see is correct on your link line.

My recommendation is to use a tool like ldd on the binary and check that the binary was actually linked against the libraries you want. Alternatively you try and use the debugger on the binary and have it stop the program in for example dgemm (a BLAS routine) and see what the debugger says about where that routine comes from. Alternatively (but more work) you can put some print statements in the source code of the BLAS routines in NWChem. If the MKL versions are used you should not see anything from those print statements.

I am sorry that it is not so easy to see what libraries the code uses, but if you really want to know there are ways to find out but it takes a little work.

Best wishes,

Huub

Forum Vet
11:52:19 AM PST - Thu, Jan 3rd 2013
How to check if mkl was linked
Doudou,
Here is the command for checking if your binary contains any mkl blas. If the command doesnot return anything,
the you do not have mkl blas objects in your nwchem binary.

nm $HOME/nwchem-6.1.1-src/bin/LINUX64/nwchem | egrep mkl_ | egrep dgemm

Clicked A Few Times
2:31:37 PM PST - Thu, Jan 3rd 2013
Thanks Huub and Edopra for your responses,

'nm $HOME/nwchem-6.1.1-src/bin/LINUX64/nwchem | egrep mkl_ | egrep dgemm' did not return, does that mean there are no MKL objects my NWChem binary? that is weird because in the link line the libraries are there.
I don't really have to use MKL, I can use another set of optimized math libraries, which ones are the most popular/best tested for NWChem? Thanks again

Forum Vet
10:20:41 AM PST - Fri, Jan 4th 2013
What about the ldd command, i.e.

ldd $HOME/nwchem-6.1.1-src/bin/LINUX64/nwchem

Does this show any mkl library?

Quote:DouDou Jan 3rd 1:31 pm
Thanks Huub and Edopra for your responses,

'nm $HOME/nwchem-6.1.1-src/bin/LINUX64/nwchem | egrep mkl_ | egrep dgemm' did not return, does that mean there are no MKL objects my NWChem binary? that is weird because in the link line the libraries are there.
I don't really have to use MKL, I can use another set of optimized math libraries, which ones are the most popular/best tested for NWChem? Thanks again

Clicked A Few Times
10:42:47 AM PST - Fri, Jan 4th 2013
Quote:Edoapra Jan 4th 9:20 am
What about the ldd command, i.e.

ldd $HOME/nwchem-6.1.1-src/bin/LINUX64/nwchem

Does this show any mkl library?

Quote:DouDou Jan 3rd 1:31 pm
Thanks Huub and Edopra for your responses,

'nm $HOME/nwchem-6.1.1-src/bin/LINUX64/nwchem | egrep mkl_ | egrep dgemm' did not return, does that mean there are no MKL objects my NWChem binary? that is weird because in the link line the libraries are there.
I don't really have to use MKL, I can use another set of optimized math libraries, which ones are the most popular/best tested for NWChem? Thanks again


Yes, they are there 

        libmkl_scalapack_ilp64.so => /home/joe/mkl/mkl/lib/intel64/libmkl_scalapack_ilp64.so (0x00002aaaaaaad000)
        libmkl_cdft_core.so => /home/joe/mkl/mkl/lib/intel64/libmkl_cdft_core.so (0x00002aaaab2b5000)
        libmkl_gf_ilp64.so => /home/joe/mkl/mkl/lib/intel64/libmkl_gf_ilp64.so (0x00002aaaab4d1000)
        libmkl_sequential.so => /home/joe/mkl/mkl/lib/intel64/libmkl_sequential.so (0x00002aaaabbd3000)
        libmkl_core.so => /home/joe/mkl/mkl/lib/intel64/libmkl_core.so (0x00002aaaac271000)
        libmkl_blacs_intelmpi_ilp64.so => /home/joe/mkl/mkl/lib/intel64/libmkl_blacs_intelmpi_ilp64.so (0x00002aaaad47f000)
        libdl.so.2 => /lib64/libdl.so.2 (0x0000003790800000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003790c00000)
        libgfortran.so.1 => /usr/lib64/libgfortran.so.1 (0x00002aaaad708000)
        libm.so.6 => /lib64/libm.so.6 (0x0000003790400000)
        librt.so.1 => /lib64/librt.so.1 (0x0000003791400000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003669600000)
        libc.so.6 => /lib64/libc.so.6 (0x0000003790000000)
        /lib64/ld-linux-x86-64.so.2 (0x000000378f000000)

Gets Around
9:11:08 AM PST - Wed, Dec 10th 2014
Quote:DouDou Jan 3rd 2:39 am


-L/home/joe/mkl/mkl/lib/intel64 -lmkl_scalapack_ilp64 -lmkl_cdft_core -lmkl_gf_ilp64 -lmkl_sequential -lmkl_core -lmkl_blacs_intelmpi_ilp64 -ldl -lpthread -lm -llapack -lblas -L/home/joe/mpich-install/lib -lmpich -lopa -lmpl -lrt -lpthread

Any ideas? Thanks in advance


You can read up on the left-to-right convention of the GNU linker (and possibly others) to understand why the presence of the built-in Netlib libraries are not used if the symbols are found in MKL first.

Jeff


Forum >> NWChem's corner >> Compiling NWChem