nwchem-6.6 ifort, segmentation fault


Jump to page 12Next 16Last
Clicked A Few Times
10:06:31 AM PST - Fri, Jan 8th 2016
Hi all,

I got a problem that I couldn't figure out. I appreciate very much if you could help me.

I am using Red Hat Enterprise Linux Server release 6.7 (Santiago).
I compiled the nwchem-6.6 with the following script:

export NWCHEM_TOP=~/Softwares/nwchem-6.6
export NWCHEM_TARGET=LINUX64
export NWCHEM_MODULES=all
export USE_INTERNALBLAS=y
export USE_MPI=y
export USE_MPIF=y
export USE_MPIF4=y
export USE_OPENMP=1
export MKLROOT=/opt/software/intel/composerxe2013/composer_xe_2013.0.079/mkl
export BLASOPT="-L/opt/software/intel/composerxe2013/composer_xe_2013.0.079/mkl/lib/intel64/ -mkl -openmp -lpthread -lm"
export LIBRARY_PATH="$LIBRARY_PATH:/opt/software/intel/composerxe2013/composer_xe_2013.0.079/mkl/lib/intel64/"
export SCALAPACK="-mkl -openmp -lmkl_scalapack_ilp64 -lmkl_blacs_intelmpi_ilp64 -lpthread -lm"
export MPI_LOC=/opt/software/openmpi/1.4.3-gcc/
export MPI_LIB=$MPI_LOC/lib
export MPI_INCLUDE=$MPI_LOC/include
export LIBMPI="-lmpi -lmpi_f77"
export NWCHEM_EXECUTABLE=~/Softwares/nwchem-6.6/bin/LINUX64/nwchem
dir=`pwd`
cd ~/Softwares/nwchem-6.6/src
make nwchem_config NWCHEM_MODULES=all > make_config_intel.log
make FC=ifort -j 16 >& make_intel.log

then it generated nwchem under ~/Softwares/nwchem-6.6/bin/LINUX64/, Then I run an example in .../nwchem-6.6/QA/tests/tce_ozone_2eorb:
../../../bin/LINUX64/nwchem  tce_ozone_2eorb.nw
and I got segmentation fault. The last message is following:

Summary of "ao basis" -> "ao basis" (spherical)

------------------------------------------------------------------------------

      Tag                 Description            Shells   Functions and Types

---------------- ------------------------------  ------  ---------------------

O                       user specified             10       24   5s3p2d




     Symmetry analysis of basis

     --------------------------



       a1         29

       a2          9

       b1         12

       b2         22




Forming initial guess at       0.1s

forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
libmpi.so.0 00007FD4B0D172AE Unknown Unknown Unknown
libmkl_blacs_inte 00007FD4B0FAC4D1 Unknown Unknown Unknown
libmkl_blacs_inte 00007FD4B0F9ECBB Unknown Unknown Unknown
libmkl_blacs_inte 00007FD4B0F9E719 Unknown Unknown Unknown
libmkl_blacs_inte 00007FD4B0FC578D Unknown Unknown Unknown
nwchem 00000000031C2819 Unknown Unknown Unknown
nwchem 00000000009F12A3 Unknown Unknown Unknown
nwchem 0000000000A0BE12 Unknown Unknown Unknown
nwchem 00000000009F2D3B Unknown Unknown Unknown
nwchem 0000000001AE21AF Unknown Unknown Unknown
nwchem 0000000001AE1F71 Unknown Unknown Unknown
nwchem 0000000000423ED6 Unknown Unknown Unknown
nwchem 0000000000415D31 Unknown Unknown Unknown
nwchem 000000000040A56B Unknown Unknown Unknown
nwchem 000000000040A01C Unknown Unknown Unknown
libc.so.6 0000003AE061ED5D Unknown Unknown Unknown
nwchem 0000000000409EE9 Unknown Unknown Unknown

I also tried different BLASOPT like:
export BLASOPT="-L/opt/software/intel/composerxe2013/composer_xe_2013.0.079/mkl/lib/intel64/ -lmkl_core -lmkl_sequential -lmkl_intel_ilp64"
&&
export BLASOPT="-L/opt/software/intel/composerxe2013/composer_xe_2013.0.079/mkl/lib/intel64/ -lmkl_core -lmkl_intel_ilp64"

but they don't work and I got same segmentation fault.

It looks like something wrong with openmpi, but I don't know why and how to solve it.

Thank you very much for your help!

Best,
Bobo

Forum Vet
10:19:18 AM PST - Fri, Jan 8th 2016
Please try the following


export BLAS_SIZE=8
export SCALAPACK_SIZE=8
cd $NWCHEM_TOP/src/tools
make clean
make FC=ifort
cd ..
make FC=ifort link

Clicked A Few Times
10:30:05 AM PST - Fri, Jan 8th 2016
Hi Edoapra,

It still doesn't work. I still got the segmentation fault.

Thanks,
Bobo

Forum Vet
10:56:55 AM PST - Fri, Jan 8th 2016
unset SCALAPACK
Let's try to see if SCALAPACK is the culprit
unset SCALAPACK
unset SCALAPACK_SIZE
cd $NWCHEM_TOP/src/tools
make clean
make FC=ifort
cd ..
make FC=ifort

Clicked A Few Times
11:36:28 AM PST - Fri, Jan 8th 2016
Compiling error
If I unset SCALAPACK. I can't compile successfully. I got compiling error as following:
/home/bobo/Softwares/nwchem-6.6/src/tools/install/lib/libga.a(scalapack.o): In function `gai_lu_s
olve_':
../ga-5-4/global/src/scalapack.F:(.text+0x18ad): undefined reference to `numroc_'
../ga-5-4/global/src/scalapack.F:(.text+0x18d4): undefined reference to `numroc_'
../ga-5-4/global/src/scalapack.F:(.text+0x18fb): undefined reference to `numroc_'
../ga-5-4/global/src/scalapack.F:(.text+0x1922): undefined reference to `numroc_'
../ga-5-4/global/src/scalapack.F:(.text+0x201d): undefined reference to `descinit_'
../ga-5-4/global/src/scalapack.F:(.text+0x2070): undefined reference to `descinit_'
../ga-5-4/global/src/scalapack.F:(.text+0x20d2): undefined reference to `pdgetrf_'
../ga-5-4/global/src/scalapack.F:(.text+0x2155): undefined reference to `pdgetrs_'
../ga-5-4/global/src/scalapack.F:(.text+0x2524): undefined reference to `blacs_gridinit_'
../ga-5-4/global/src/scalapack.F:(.text+0x2551): undefined reference to `blacs_gridinfo_'
../ga-5-4/global/src/scalapack.F:(.text+0x262c): undefined reference to `blacs_gridexit_'
../ga-5-4/global/src/scalapack.F:(.text+0x264f): undefined reference to `blacs_gridexit_'
../ga-5-4/global/src/scalapack.F:(.text+0x2810): undefined reference to `blacs_gridinfo_'
/home/bobo/Softwares/nwchem-6.6/src/tools/install/lib/libga.a(scalapack.o): In function `ga_chole
sky_':
../ga-5-4/global/src/scalapack.F:(.text+0x3256): undefined reference to `numroc_'
../ga-5-4/global/src/scalapack.F:(.text+0x327d): undefined reference to `numroc_'
../ga-5-4/global/src/scalapack.F:(.text+0x35f1): undefined reference to `descinit_'
../ga-5-4/global/src/scalapack.F:(.text+0x3644): undefined reference to `pdpotrf_'
../ga-5-4/global/src/scalapack.F:(.text+0x3c15): undefined reference to `blacs_gridinfo_'
../ga-5-4/global/src/scalapack.F:(.text+0x3d0b): undefined reference to `blacs_gridexit_'

Thanks,
Bobo

Forum Vet
2:55:33 PM PST - Fri, Jan 8th 2016
Let's keep it simple
Let's try too keep it simple since you have too many variables to deal with (MKL,Scalapack, MPI, etc ...)
Please set (and unset) the following variables

export NWCHEM_TOP=~/Softwares/nwchem-6.6
export NWCHEM_TARGET=LINUX64
export NWCHEM_MODULES=all
export USE_INTERNALBLAS=y
export USE_MPI=y
export USE_MPIF=y
export USE_MPIF4=y
export PATH=/opt/software/openmpi/1.4.3-gcc/bin:$PATH
unset USE_OPENMP
unset BLASOPT 
unset SCALAPACK
unset BLAS_SIZE
unset SCALAPACK_SIZE
unset MPI_LIB
unset MPI_INCLUDE
unset LIBMPI

Then, recompile from scratch

cd $NWCHEM_TOP/src
make realclean
rm -rf tools/build tools/install
rm -rf ../lib/LINUX64/*
make nwchem_config
make FC=ifort  >& make_intel.log


By the way, pre-compiled NWChem 6.6 is available for RedHat 6.6 from the Epel repository. You can install it by typing the following commands


sudo yum -y install http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
sudo yum update
sudo yum install nwchem-openmpi.x86_64

Clicked A Few Times
3:23:12 PM PST - Fri, Jan 8th 2016
Hi Edoapra,

Actually I need to add some code to nwchem and then test, so I need to compile it on my side.

I tried the simple compiling as you suggested, but I got the following:

checking whether it is safe to define __EXTENSIONS__... yes
checking whether a simple C MPI program compiles... yes
configure: searching for MPI_TS...
checking mpi.h usability... yes
checking mpi.h presence... yes
checking for mpi.h... yes
checking for library containing MPI_Init... no
configure: error: test for COMEX_NETWORK=MPI_TS failed
configure: error: ../../ga-5-4/comex/configure failed for comex
make[1]: *** [build/config.status] Error 1
make[1]: Leaving directory `/home/bobo/Softwares/nwchem-6.6/src/tools'
make: *** [libraries] Error 1

Not sure what does this mean. I have ga-5-4 installed and
[bobo@cerberus src]$ ls ../../ga-5-4b/comex/
aclocal.m4 config.h.in configure.ac libcomex.la Makefile src-common/ src-mpi-pr/ src-template/
build-aux/ config.log libarmcii.la .libs/ Makefile.am src-dmapp/ src-mpi-pt/ stamp-h1
compat/ config.status libarmci.la libtool Makefile.in src-mpi/ src-ofa/ testing/
config.h configure libcomexi.la m4/ src-armci/ src-mpi-mt/ src-portals4/ tools/

Thanks for your help.
Bobo

Forum Vet
4:17:09 PM PST - Fri, Jan 8th 2016
mpif90
I assumed that you had mpif90 installed in /opt/software/openmpi/1.4.3-gcc/bin, that's why I have asked you to add this directory to your path. Please find the correct location of mpif90 and add that directory to your PATH

http://nwchemgit.github.io/index.php/Compiling_NWChem#MPI_variables

Clicked A Few Times
7:06:24 PM PST - Fri, Jan 8th 2016
mpif90 gives me the following:
[bobo@cerberus src]$ mpif90 -show
ifort -I/opt/intel/impi/5.0.1.035/intel64/include -I/opt/intel/impi/5.0.1.035/intel64/lib -L/opt/intel/impi/5.0.1.035/intel64/lib

however, there is no path of /opt/intel/impi/5.0.1.035/intel64 in my machine. but mpif90 is in /opt/software/openmpi/1.4.3-gcc/bin/, so I am not sure why mpif90 -show isn't giving the same path
[bobo@cerberus src]$ ls /opt/software/openmpi/1.4.3-gcc/bin/
mpic++ mpiCC-vt mpiexec mpif90-vt ompi-iof opari orte-ps otfconfig otfmerge vtf90
mpicc mpic++-vt mpif77 mpirun ompi-ps orte-clean orterun otfdecompress vtcc vtfilter
mpiCC mpicxx mpif77-vt ompi-clean ompi-server orted otfaux otfdump vtcxx vtunify
mpicc-vt mpicxx-vt mpif90 ompi_info opal_wrapper orte-iof otfcompress otfinfo vtf77

so I tried:
export NWCHEM_TOP=~/Softwares/nwchem-6.6
export NWCHEM_TARGET=LINUX64
export NWCHEM_MODULES=all
export USE_INTERNALBLAS=y
export USE_MPI=y
export USE_MPIF=y
export USE_MPIF4=y
  1. export PATH=/opt/software/openmpi/1.4.3-gcc/bin:$PATH
  2. export PATH=/opt/intel/impi/5.0.1.035/intel64/bin:$PATH
export MPI_LOC=/opt/software/openmpi/1.4.3-gcc/
export MPI_LIB=$MPI_LOC/lib
export MPI_INCLUDE=$MPI_LOC/include
unset USE_OPENMP
unset BLASOPT
unset SCALAPACK
unset BLAS_SIZE
unset SCALAPACK_SIZE
  1. unset MPI_LIB
  2. unset MPI_INCLUDE
  3. unset LIBMPI
cd $NWCHEM_TOP/src
make realclean
rm -rf tools/build tools/install
rm -rf ../lib/LINUX64/*
make nwchem_config
make FC=ifort >& make_simple.log

and It compiled successfully and I can run it without error!

Then do I need to try adding something like MKL and Scalapack?
I will try some of those and let you know.

Thank you very much.

Bobo

Forum Vet
11:57:56 AM PST - Mon, Jan 11th 2016
MKL
First, let's try MKL.
What version of Intel compiler are you using?
In other words, what's the output of ifort -V?
Based on the version of ifort you are using, we need to choose the correct value of BLASOPT by using the Intel MKL advisor
https://software.intel.com/en-us/articles/intel-mkl-link-line-advisor

Clicked A Few Times
12:05:19 PM PST - Mon, Jan 11th 2016
Hi Edoapra,

[bobo@hydra src]$ ifort -v
ifort version 13.1.3

Forum Vet
12:54:40 PM PST - Mon, Jan 11th 2016
Please try the following


export BLASOPT="-Wl,--start-group \
 $MKLROOT/lib/intel64/libmkl_intel_ilp64.a \
 $MKLROOT/lib/intel64/libmkl_core.a \
 $MKLROOT/lib/intel64/libmkl_sequential.a \
 -Wl,--end-group -lpthread -lm"

export BLAS_SIZE=8

cd $NWCHEM_TOP/src/tools
rm -rf build install
make FC=ifort
cd ..
make FC=ifort link

Clicked A Few Times
2:42:56 PM PST - Tue, Jan 12th 2016
Dear Edoapra

I don't why I got mpi error again, when I tried to recompile it by just using mpi (no mkl, no BLASOPT)

checking for mpi.h... yes
checking for library containing MPI_Init... no
configure: error: test for COMEX_NETWORK=MPI_TS failed
configure: error: ../../ga-5-4/comex/configure failed for comex
make[1]: *** [build/config.status] Error 1
make[1]: Leaving directory `/home/bobo/Softwares/nwchem-6.6/src/tools'
make: *** [libraries] Error 1

Actually I succeeded last time. Don't know why not work this time.
Still mpif90 -show gives
[bobo@cerberus src]$ mpif90 -show
ifort -I/opt/intel/impi/5.0.1.035/intel64/include -I/opt/intel/impi/5.0.1.035/intel64/lib -L/opt/intel/impi/5.0.1.035/intel64/lib

but there is no folder called impi in /opt/intel/
[bobo@cerberus src]$ ls /opt/intel/
ActivationTool ism licenses

If i change MPI_LOC=/opt/intel/impi/5.0.1.035/intel64, I will get
checking whether it is safe to define __EXTENSIONS__... yes
checking for C compiler vendor... gnu
configure: WARNING: /opt/intel/impi/5.0.1.035/intel64 of with_mpi not parsed
checking whether a simple C MPI program compiles... no
configure: error: in `/home/bobo/Softwares/nwchem-6.6/src/tools/build':
configure: error: could not compile simple C MPI program
See `config.log' for more details

Do you have any idea how should I do?
Thank you very much.

Forum Vet
10:48:58 PM PST - Tue, Jan 12th 2016
Please go back to the previous settings, then recompile just the tools directory.
Next, set BLASOPT as shown in my last message and try "make FC=ifort link".
Please send me the error that this command will produce

Clicked A Few Times
6:49:51 AM PST - Wed, Jan 13th 2016
Hi Edoapra
I did
export NWCHEM_TOP=~/Softwares/nwchem-6.6
export NWCHEM_TARGET=LINUX64
export NWCHEM_MODULES=all
export USE_INTERNALBLAS=y
export USE_MPI=y
export USE_MPIF=y
export USE_MPIF4=y
export MPI_LOC=/opt/software/openmpi/1.4.3-gcc/
export MPI_LIB=$MPI_LOC/lib
export MPI_INCLUDE=$MPI_LOC/include
export BLASOPT="-Wl,--start-group \
$MKLROOT/lib/intel64/libmkl_intel_ilp64.a \

$MKLROOT/lib/intel64/libmkl_core.a \

$MKLROOT/lib/intel64/libmkl_sequential.a \

-Wl,--end-group -lpthread -lm"

export BLAS_SIZE=8
cd $NWCHEM_TOP/src/tools
rm -rf build install
make FC=ifort

I got
checking for library containing MPI_Init... no
configure: error: test for COMEX_NETWORK=MPI_TS failed
configure: error: ../../ga-5-4/comex/configure failed for comex
make: *** [build/config.status] Error 1

Forum Vet
10:21:29 AM PST - Wed, Jan 13th 2016
I am not sure if I can help you here.
Are you stating that you can no longer reproduce what you described in the following post
http://nwchemgit.github.io/Special_AWCforum/sp/id6833
?


Forum >> NWChem's corner >> Compiling NWChem
Jump to page 12Next 16Last