Can't find MKL routines during compilation on CentOS 7 cluster
Clicked A Few Times
2:42:55 PM PST - Tue, Dec 3rd 2019
I've been having some trouble with this for a while. I managed to get NWChem to compile and run using the instructions at http://nwchemgit.github.io/index.php/Compiling_NWChem. I noticed my performance was five to ten times worse than what I had gotten from a compilation on a shared platform elsewhere. All nodes are Intel Xeon processors. I thought I would recompile with BLAS/SCALAPACK/LAPACK taken from Intel's MKL (which I installed in my home directory). I also made some other changes to try and include threading and other tweaks to get better performance. The compilation now fails and can't find the routines from MKL or the ELPA libraries. Any thoughts?

My script is below (thought it appears my sha-bang is having some syntax issues in the preview):

  1. !/bin/bash -l
source /home/epj/intel/parallel_studio_xe_2019.5.075/psxevars.sh intel64
source /home/epj/intel/bin/compilervars.sh intel64
source /home/epj/intel/mkl/bin/mklvars.sh intel64
module purge
module load intel/2019
module load mpi/mpich-3.0-x86_64
export NWCHEM_TOP=/home/epj/thirdcomp/nwchem-6.8.1-release
export USE_MPI=y
export USE_MPIF=y
export USE_MPI4=y
export NWCHEM_TARGET=LINUX64
export NWCHEM_LONG_PATHS=y
export USE_NOFSCHECK=y
export TCGRSH=/usr/bin/ssh
export LARGE_FILES=y
export USE_PYTHONCONFIG=y
export PYTHONVERSION=2.7
export PYTHONHOME=/usr
export USE_PYTHON64=y
export USE_64TO32=y
export HAS_BLAS=y
export USE_SCALAPACK=y
export BLAS_SIZE=8
export LAPACK_SIZE=8
export SCALAPACK_SIZE=8
export USE_OPENMP=1
export USE_F90_ALLOCATABLE=T
export ARMCI_NETWORK=OPENIB
export IB_HOME=/usr
export IB_INCLUDE=/usr/include/infiniband
export IB_LIB=/usr/lib64
export IB_LIB_NAME="-libumad -libverbs -lpthread"
export MPI_LOC="/usr/lib64/mpich"
export MPI_INCLUDE="/usr/include/mpich-x86_64 -I/usr/include/mpich-x86_64"
export MPI_LIB="/usr/lib64/mpich/lib"
export LIBMPI="-lmpichf90 -lmpich -lopa -lmpl -lrt -lpthread"
export MKLLIB="/home/epj/intel/mkl/lib/intel64"
export MKLINC="/home/epj/intel/mkl/include"
export BLASOPT="-L$MKLLIB -lmkl_intel_ilp64 -lmkl_intel_thread -lmkl_sequential -lmkl_core -lmkl_def -lmkl_vml_mc2 -lmkl_scalapack_ilp64 -lmkl_blacs_intelmpi_ilp64 -lmklblas95_ilp64 -lmkl_lapack95_ilp64 -liomp5 -lpthread -lm -ldl"
export LAPACK_LIB="$BLASOPT"
export LAPACK_LIBS="$BLASOPT"
export LAPACKOPT="$BLASOPT"
export SCALAPACK="-L$MKLLIB -lmkl_intel_ilp64 -lmkl_intel_thread -lmkl_sequential -lmkl_core -lmkl_def -lmkl_vml_mc2 -lmkl_scalapack_ilp64 -lmkl_blacs_intelmpi_ilp64 -lmklblas95_ilp64 -lmkl_lapack95_ilp64 -liomp5 -lpthread -lm -ldl"
export SCALAPACK_LIB="$SCALAPACK"
export SCALAPACK_LIBS="$SCALAPACK"
export ELPA="-I/usr/lib64/gfortran/modules/openmpi-x86_64 -L/usr/lib64/openmpi/lib -lelpa"
export LD_LIBRARY_PATH=/usr/lib64/mpich/lib:$LD_LIBRARY_PATH
export PATH=/usr/lib64/mpich/bin:$PATH
cd $NWCHEM_TOP/src
make realclean
make nwchem_config NWCHEM_MODULES="all python" >& make_config1a.log
make 64_to_32 >& make_64_1a.log
make FC=ifort CC=icc >& make1a.log

Clicked A Few Times
10:32:46 AM PST - Thu, Dec 12th 2019
Hi folks,

I still seem to be running into problems, please let me know if anyone has suggestions. At the moment I am getting the error message below. My script for compiling is the error message.



Error message:

Compiling util_mic_support.c...
util_mic_support.c(113): warning #144: a value of type "int" cannot be used to initialize an entity of type "MPI_Comm"
     MPI_Comm ga_comm=GA_MPI_Comm_pgroup_default();

                      ^

icc: warning #10362: Environment configuration problem encountered. Please check for proper MPSS installation and environment setup.
util_mic_support.c(4): catastrophic error: *MIC* cannot open source file "stdio.h"
 #include <stdio.h>

                   ^

compilation aborted for util_mic_support.c (code 4)
make[1]: *** [/home/epj/thirdcomp/nwchem-6.8.1-release/lib/LINUX64/libnwcutil.a(util_mic_support.o)] Error 4
make[1]: Leaving directory `/home/epj/thirdcomp/nwchem-6.8.1-release/src/util'
make: *** [libraries] Error 1



Script (note the sha-bang is still being oddly parsed in this forum and that export ELPA is actually just commented out):

  1. !/bin/bash -l
source /home/epj/intel/parallel_studio_xe_2019.5.075/psxevars.sh intel64
source /home/epj/intel/bin/compilervars.sh intel64
source /home/epj/intel/mkl/bin/mklvars.sh intel64
module purge
module load hpc_defaults/0.1
module load intel/2017
module load openmpi/2.1.0/intel-17
export NWCHEM_TOP=/home/epj/thirdcomp/nwchem-6.8.1-release
export NWCHEM_TARGET=LINUX64
export NWCHEM_LONG_PATHS=y
export USE_NOFSCHECK=y
export LARGE_FILES=y

export ARMCI_NETWORK=OPENIB
export IB_HOME=/usr
export IB_INCLUDE=/usr/include/infiniband
export IB_LIB=/usr/lib64
export IB_LIB_NAME="-libumad -libverbs -lpthread"

export USE_PYTHONCONFIG=y
export PYTHONHOME=/usr
export PYTHONVERSION=2.7
export PYTHONLIBTYPE=so
export USE_PYTHON64=y

export TCGRSH=/usr/bin/ssh
export USE_F90_ALLOCATABLE=y

export USE_MPI=y
export USE_MPIF=y
export USE_MPIF4=y
export MPI_LOC="/usr/lib64/openmpi"
export USE_OPENMP=y
export USE_OFFLOAD=y
export MKLROOT="/home/epj/intel/mkl"
export MKLLIB="/home/epj/intel/mkl/lib/intel64"
export MKLINC="/home/epj/intel/mkl/include"

export HAS_BLAS=y
export BLAS_SIZE=8
export BLASOPT="-L${MKLROOT}/lib/intel64 -mkl -openmp -lmkl_intel_ilp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm -ldl"
export LAPACK_SIZE=8
export LAPACK_LIB="$BLASOPT"
export LAPACK_LIBS="$BLASOPT"
export LAPACKOPT="$BLASOPT"
export USE_SCALAPACK=y
export SCALAPACK_SIZE=8
export SCALAPACK="-L${MKLROOT}/lib/intel64 -mkl -openmp -lmkl_scalapack_ilp64 -lmkl_intel_ilp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_intelmpi_ilp64 -liomp5 -lpthread -lm -ldl"
export SCALAPACK_LIB="$SCALAPACK"
export SCALAPACK_LIBS="$SCALAPACK"

  1. export ELPA="-I/usr/lib64/gfortran/modules/openmpi-x86_64 -L/usr/lib64/openmpi/lib -lelpa"
export USE_64TO32=y
export PATH=/usr/lib64/openmpi/bin/:$PATH
export LD_LIBRARY_PATH=/usr/lib64/openmpi/lib:/home/epj/intel/mkl/lib/intel64:$LD_LIBRARY_PATH
cd $NWCHEM_TOP/src

make realclean
make nwchem_config NWCHEM_MODULES="all python" >& make_config1t.log
make 64_to_32 >& make_64_1t.log
make FC=ifort CC=icc >& make1t.log

Clicked A Few Times
9:07:05 AM PST - Mon, Dec 23rd 2019
Still no success
Hi all,

I'm hoping someone on the forum has some insight to lend on my compilation error. I've been trying for almost two months to get a stable build that scales well when jobs are ran on multiple nodes. It's been a pretty demoralizing task. I thought I was close on Friday when some attempts led to decent performance when ran on three nodes. I tried some additional options and unfortunately, this morning neither the new builds or the ones that were successful are working. Rather, the output file at run time reads " nwchem.F: ma_init failed (ga_uses_ma=F) 911" despite a "successful" compilation. I will list my current compilation and job submission scripts below, as well as my test input file. Note that some lines are commented out, these are options I have tried toggling on and off. Note that this forum seems to interpret the pound/hashtag symbol as the start of a list.

--compilation--
  1. !/bin/bash -l
module purge
module load hpc_defaults/0.1
module load intel/2019
export NWCHEM_TOP=/home/epj/secondcomp/nwchem-6.8.1-release
export NWCHEM_TARGET=LINUX64
export NWCHEM_LONG_PATHS=y
export NWCHEM_MODULES="all python"
export USE_NOFSCHECK=y
export LARGE_FILES=y

export I_MPI_OFI_LIBRARY_INTERNAL=1
export FI_PROVIDER=verbs
export I_MPI_FABRICS=shm:ofi

export ARMCI_NETWORK=OPENIB
  1. export IB_HOME=/usr
  2. export IB_INCLUDE=/usr/include/infiniband
  3. export IB_LIB=/usr/lib64
  4. export IB_LIB_NAME="-libumad -libverbs -lpthread"

export USE_PYTHONCONFIG=y
export PYTHONHOME=/usr
export PYTHONVERSION=2.7
export PYTHONLIBTYPE=so
export USE_PYTHON64=y

export TCGRSH=/usr/bin/ssh
export USE_F90_ALLOCATABLE=y

export USE_MPI=y
export USE_MPIF=y
export USE_MPIF4=y
export LIBMPI="-lmpifort -lmpi -lmpi_ilp64 -ldl -lrt -lpthread"
  1. export USE_OPENMP=y

export HAS_BLAS=y
export BLAS_SIZE=8
  1. export BLAS_SIZE=4
export BLASOPT="-mkl -lpthread -lm"
  1. export BLASOPT="-L$MKLROOT/lib/intel64 -lmkl_rt -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_def -lmkl_blacs_intelmpi_lp64 -lmkl_lapack95_lp64 -liomp5 -lpthread -ldl -lm"
export LAPACK_SIZE=8
  1. export LAPACK_SIZE=4
export LAPACK_LIB="$BLASOPT"
export LAPACK_LIBS="$BLASOPT"
export LAPACKOPT="$BLASOPT"
export USE_SCALAPACK=y
export SCALAPACK_SIZE=8
  1. export SCALAPACK_SIZE=4
export SCALAPACK="-mkl -lpthread -lm"
  1. export SCALAPACK="-L$MKLROOT/lib/intel64 -lmkl_rt -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_def -lmkl_blacs_intelmpi_lp64 -lmkl_lapack95_lp64 -liomp5 -lmkl_scalapack_lp64 -lpthread -ldl -lm"
export SCALAPACK_LIB="$SCALAPACK"
export SCALAPACK_LIBS="$SCALAPACK"

export USE_64TO32=y
export CC=icc
export FC=ifort

cd $NWCHEM_TOP/src

make realclean
make nwchem_config >& make_config1zg.log
make 64_to_32 >& make_64_1zg.log
make >& make1zg.log

--job submission--
  1. !/bin/bash -l
  2. SBATCH -e myjob.err
  3. SBATCH -J water
  4. SBATCH --mail-type=END,FAIL
  5. SBATCH --nodes=3
  6. SBATCH --ntasks=48
  7. SBATCH --ntasks-per-node=16
  8. SBATCH --constraint="[mlx]"
  9. SBATCH -o myjob.out
  10. SBATCH -t 1:00:00
module purge
module load hpc_defaults/0.1
module load intel/2019
  1. export I_MPI_OFI_LIBRARY_INTERNAL=1
export FI_PROVIDER=verbs
  1. export I_MPI_FABRICS=shm:ofi
export ARMCI_NETWORK=OPENIB
export ARMCI_DEFAULT_SHMMAX=32768
export NWCHEM_BASIS_LIBRARY="/home/epj/secondcomp/nwchem-6.8.1-release/src/basis/libraries/"
export NWCHEM_NWPW_LIBRARY="/home/epj/secondcomp/nwchem-6.8.1-release/src/nwpw/libraryps/"
export ARMCI_OPENIB_DEVICE=mlx4_0
export OMP_NUM_THREADS=1
export MKL_NUM_THREADS=1
export NWC_RANKS_PER_DEVICE=0
export OFFLOAD_INIT=on_offload
export MPIRETURN=999
mpirun -np $SLURM_NPROCS /home/epj/secondcomp/nwchem-6.8.1-release/bin/LINUX64/nwchem ./input.nw > run.out
export MPIRETURN=$?


  2. End of the job script


exit $MPIRETURN

--input file--
echo

start freq-trial

scratch_dir /scratch/

memory 4000 mb noverify

geometry units angstrom noautoz nocenter noautosym
 O      0.00000000    0.00000000    0.00000000

 H      0.00000000    1.93042809   -1.10715266

 H      0.00000000   -1.93042809   -1.10715266
end

BASIS "ao basis" spherical PRINT
 H library 6-311g**

 O library 6-311g**
END

dft
 xc b3lyp

 odft

 mult 1

 mulliken

 direct

 maxiter 1000

 convergence energy 1e-6
end

task dft optimize

dft
 convergence energy 1e-8
end

freq
 animate
end

task dft vscf

Clicked A Few Times
8:15:12 AM PST - Mon, Jan 27th 2020
Any thoughts?

Clicked A Few Times
11:23:46 AM PST - Mon, Feb 3rd 2020
Again, if anyone has thoughts on the above issues, they would be very appreciated. My performance on jobs executed across several nodes appears to be very poor and/or prone to crashing depending on the combination of several environmental variables.
Perhaps I am in the wrong place. If achieving a useful executable for parallel job execution is a question better submitted to the "Running NWChem" forum, please let me know. I would be happy to post over there instead.


Forum >> NWChem's corner >> Compiling NWChem