NWChem 6.1 scalapack compilation fails


  • Guest -
I have successfully compiled NWChem v. 6.1, but when going forward to switch on support for scalapack, the last linking stage fails with unresolved symbols:
"""
/home/raber/NWChem/gcc/nwchem-6.1/lib/LINUX64/libnwdft.a(dft_diagn.o): In function `dft_scaleig_':
/home/raber/NWChem/gcc/nwchem-6.1/src/nwdft/scf_dft/dft_diagn.F:210: undefined reference to `ga_pdsyev_'
/home/raber/NWChem/gcc/nwchem-6.1/src/nwdft/scf_dft/dft_diagn.F:212: undefined reference to `ga_pdsyevx_'
/home/raber/NWChem/gcc/nwchem-6.1/src/nwdft/scf_dft/dft_diagn.F:214: undefined reference to `ga_pdsyevd_'
collect2: ld returned 1 exit status
"""

The script I use to compile NWChem is:


  1. !/bin/bash

export IB_HOME=/usr
export IB_INCLUDE=/usr/include/infiniband/
export IB_LIB=/usr/lib64
export IB_LIB_NAME="-libumad -libverbs -lpthread"
export ARMCI_NETWORK=OPENIB

export USE_SUBGROUPS=yes
export LARGE_FILES=yes

export USE_MPI=yes
export USE_MPIF=yes
export MSG_COMMS=MPI
export MPI_LOC=/usr/local/openmpi/gcc/1.4.4
export MPI_INCLUDE=$MPI_LOC/include
export MPI_LIB=$MPI_LOC/lib
export LIBMPI="-Wl,-rpath,/usr/local/openmpi/gcc/1.4.4/lib -lmpi -lmpi_f77 -lpthread -lz -lssl -static-libgcc"

export USE_PYTHON=yes
export PYTHONHOME=/usr
export PYTHONVERSION=2.6
export PYTHONPATH=./:$NWCHEM_TOP/contrib/python/

export NWCHEM_TARGET=LINUX64
  1. export NWCHEM_MODULES="all python"
export NWCHEM_MODULES="pnnl"
export NWCHEM_TOP=/home/raber/NWChem/gcc/nwchem-6.1
export NWCHEM_EXECUTABLE=$NWCHEM_TOP/bin/LINUX64/nwchem

export HAS_BLAS=yes
export USE_SCALAPACK=yes
export BLASOPT="-Wl,-rpath,/home/intel/composer_xe_2011_sp1.7.256/composer_xe_2012_sp1.7.256/mkl/lib/intel64 -L/home/intel/composer_xe_2011_sp1.7.256/composer_xe_2011_sp1.7.256/mkl/lib/intel64 -Wl,--start-group -lmkl_gf_ilp64 -lmkl_core -lmkl_sequential -lmkl_blacs_openmpi_ilp64 -lmkl_scalapack_ilp64 -Wl,--end-group"

cd $NWCHEM_TOP/src

make DIAG=PAR FC=gfortran CC=gcc FOPTIMIZE="-O3 -march=native" COPTIMIZE="-O3 -march=native" nwchem_config
make DIAG=PAR FC=gfortran CC=gcc FOPTIMIZE="-O3 -march=native" COPTIMIZE="-O3 -march=native" $1



I think I have followed the install instructions wrt scalapack but maybe I'm missing something. I find the subroutines in the code (i.e. ga_pdsyev{x,d}_ ), for instance in the files ~/nwchem-6.1/src/tools/{,ga-5-1}/global/src/scalapack.F but I can't find them in any of the link libraries or other object files( simple, brute force "grep"). Please advice.

Cheers,
Johan

  • Guest -
s/1./#/ in the parent
This wiki extension apparently change hash characters for '1.', please turn on your internal filters.

/Johan

Forum Vet
I assume you did a make realclean before you build.

Can you check the src/tools/build/config.log to see if it actually picked up the SCALAPACK libraries? If you can figure it out, send me the config.log file at bert.dejong@pnnl.gov.

Bert

[QUOTE= Feb 1st 1:11 pm]I have successfully compiled NWChem v. 6.1, but when going forward to switch on support for scalapack, the last linking stage fails with unresolved symbols:
"""
/home/raber/NWChem/gcc/nwchem-6.1/lib/LINUX64/libnwdft.a(dft_diagn.o): In function `dft_scaleig_':
/home/raber/NWChem/gcc/nwchem-6.1/src/nwdft/scf_dft/dft_diagn.F:210: undefined reference to `ga_pdsyev_'
/home/raber/NWChem/gcc/nwchem-6.1/src/nwdft/scf_dft/dft_diagn.F:212: undefined reference to `ga_pdsyevx_'
/home/raber/NWChem/gcc/nwchem-6.1/src/nwdft/scf_dft/dft_diagn.F:214: undefined reference to `ga_pdsyevd_'
collect2: ld returned 1 exit status
"""

The script I use to compile NWChem is:


  1. !/bin/bash

export IB_HOME=/usr
export IB_INCLUDE=/usr/include/infiniband/
export IB_LIB=/usr/lib64
export IB_LIB_NAME="-libumad -libverbs -lpthread"
export ARMCI_NETWORK=OPENIB

export USE_SUBGROUPS=yes
export LARGE_FILES=yes

export USE_MPI=yes
export USE_MPIF=yes
export MSG_COMMS=MPI
export MPI_LOC=/usr/local/openmpi/gcc/1.4.4
export MPI_INCLUDE=$MPI_LOC/include
export MPI_LIB=$MPI_LOC/lib
export LIBMPI="-Wl,-rpath,/usr/local/openmpi/gcc/1.4.4/lib -lmpi -lmpi_f77 -lpthread -lz -lssl -static-libgcc"

export USE_PYTHON=yes
export PYTHONHOME=/usr
export PYTHONVERSION=2.6
export PYTHONPATH=./:$NWCHEM_TOP/contrib/python/

export NWCHEM_TARGET=LINUX64
  1. export NWCHEM_MODULES="all python"
export NWCHEM_MODULES="pnnl"
export NWCHEM_TOP=/home/raber/NWChem/gcc/nwchem-6.1
export NWCHEM_EXECUTABLE=$NWCHEM_TOP/bin/LINUX64/nwchem

export HAS_BLAS=yes
export USE_SCALAPACK=yes
export BLASOPT="-Wl,-rpath,/home/intel/composer_xe_2011_sp1.7.256/composer_xe_2012_sp1.7.256/mkl/lib/intel64 -L/home/intel/composer_xe_2011_sp1.7.256/composer_xe_2011_sp1.7.256/mkl/lib/intel64 -Wl,--start-group -lmkl_gf_ilp64 -lmkl_core -lmkl_sequential -lmkl_blacs_openmpi_ilp64 -lmkl_scalapack_ilp64 -Wl,--end-group"

cd $NWCHEM_TOP/src

make DIAG=PAR FC=gfortran CC=gcc FOPTIMIZE="-O3 -march=native" COPTIMIZE="-O3 -march=native" nwchem_config
make DIAG=PAR FC=gfortran CC=gcc FOPTIMIZE="-O3 -march=native" COPTIMIZE="-O3 -march=native" $1



I think I have followed the install instructions wrt scalapack but maybe I'm missing something. I find the subroutines in the code (i.e. ga_pdsyev{x,d}_ ), for instance in the files ~/nwchem-6.1/src/tools/{,ga-5-1}/global/src/scalapack.F but I can't find them in any of the link libraries or other object files( simple, brute force "grep"). Please advice.

Cheers,
Johan

  • Guest -
NWChem 6.1 scalapack compilation fails (follow-up)
Thanks a lot for helping me out.

I did a "make realclean" as I recall, and just in case I misremembered, I untarred a fresh copy and tried again. The result was the same.

I will send you the log file out of band. I'm having some trouble interpreting the log but I think the BLASOPT variable is being misinterpreted and the scalapack library isn't found. I'll experiment a bit with the format of the variable I use and see if my luck turns better.

Best regards,
Johan

Gets Around
Yes, i have a similar impression. I had to put a full link line related to scalapack (see point 3) into BLAS_LIB
(i guess BLASOPT would work too) variable
in oder to src/tools/ga-5-1/configure to find scalapack. On Fedora 16 x86_64:
0. yum install scalapack-openmpi-devel blacs-openmpi-devel openmpi-devel atlas-devel
1. cd src/tools/ga-5-1/
2. run:

./configure '--with-mpi=-I/usr/include/openmpi-x86_64 -L/usr/lib64/openmpi/lib' '--with-blas4=-L/usr/lib64/atlas -llapack -lf77blas -lcblas -latlas' '--with-scalapack=-L/usr/lib64/openmpi/lib -lmpiblacsF77init -lmpiblacs -lscalapack' 'CC=gcc' 'F77=gfortran' 'MPIEXEC=/usr/lib64/openmpi/bin/mpiexec'

Error in config.log:

configure:20768: checking for pdgetrs_ in -lscalapack configure:20801: gcc -o conftest -L/usr/lib64/atlas conftest.c -lscalapack -L/usr/lib/gcc/x86_64-redhat-linux/4.6.2 -L/usr/lib/gcc/x86_64-redhat-linux/4.6.2/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.6.2/../../.. -lgfortran -lm -lquadmath -lm >&5 /usr/bin/ld: cannot find -lscalapack collect2: ld returned 1 exit status

3. Adding blas/lapack and openmpi libs into --with-scalapack does not work, and the following configure line works (note the complete absence of --with-scalapack!):

./configure '--with-mpi=-I/usr/include/openmpi-x86_64 -L/usr/lib64/openmpi/lib' '--with-blas4=-L/usr/lib64/atlas -llapack -lf77blas -lcblas -latlas -L/usr/lib64/openmpi/lib -lmpiblacsF77init -lmpiblacs -lscalapack -L/usr/lib64/openmpi/lib -lmpi -lopen-pal -lopen-rte -lmpi_f90 -lmpi_f77' 'CC=gcc' 'F77=gfortran' 'MPIEXEC=/usr/lib64/openmpi/bin/mpiexec'

By the way, are there any scalapack tests in QA directory?

Forum Vet
The only scalapack routines used are those for the eigensolver, which is used in the DFT calculations. So, running DFT tests will do a QA on scalapack.

Bert

Quote:Marcindulak Feb 6th 2:24 pm
Yes, i have a similar impression. I had to put a full link line related to scalapack (see point 3) into BLAS_LIB
(i guess BLASOPT would work too) variable
in oder to src/tools/ga-5-1/configure to find scalapack. On Fedora 16 x86_64:
0. yum install scalapack-openmpi-devel blacs-openmpi-devel openmpi-devel atlas-devel
1. cd src/tools/ga-5-1/
2. run:

./configure '--with-mpi=-I/usr/include/openmpi-x86_64 -L/usr/lib64/openmpi/lib' '--with-blas4=-L/usr/lib64/atlas -llapack -lf77blas -lcblas -latlas' '--with-scalapack=-L/usr/lib64/openmpi/lib -lmpiblacsF77init -lmpiblacs -lscalapack' 'CC=gcc' 'F77=gfortran' 'MPIEXEC=/usr/lib64/openmpi/bin/mpiexec'

Error in config.log:

configure:20768: checking for pdgetrs_ in -lscalapack configure:20801: gcc -o conftest -L/usr/lib64/atlas conftest.c -lscalapack -L/usr/lib/gcc/x86_64-redhat-linux/4.6.2 -L/usr/lib/gcc/x86_64-redhat-linux/4.6.2/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.6.2/../../.. -lgfortran -lm -lquadmath -lm >&5 /usr/bin/ld: cannot find -lscalapack collect2: ld returned 1 exit status

3. Adding blas/lapack and openmpi libs into --with-scalapack does not work, and the following configure line works (note the complete absence of --with-scalapack!):

./configure '--with-mpi=-I/usr/include/openmpi-x86_64 -L/usr/lib64/openmpi/lib' '--with-blas4=-L/usr/lib64/atlas -llapack -lf77blas -lcblas -latlas -L/usr/lib64/openmpi/lib -lmpiblacsF77init -lmpiblacs -lscalapack -L/usr/lib64/openmpi/lib -lmpi -lopen-pal -lopen-rte -lmpi_f90 -lmpi_f77' 'CC=gcc' 'F77=gfortran' 'MPIEXEC=/usr/lib64/openmpi/bin/mpiexec'

By the way, are there any scalapack tests in QA directory?

Forum Vet
new tarball available with fixes for tools compilation
We have just uploaded a new 6.1 source tarball that should address most of the problems you have encountered trying to link Scalapack. Please use just the following settings (and neither BLAS_LIB nor SCALAPACK_LIB)
USE_SCALAPACK=y
SCALAPACK="Scalapack and Blacs library location"
BLASOPT="Blas and Lapack library location"

Gets Around
Could you when making changes to the source release it as 6.1.1 etc?
This will help to avoid confusion about which version people are referring to,
and it's a standard practice for installed software by package managers, like yum , yast, apt-get.
They recognize that a new version is available by comparing versions.

Forum Vet
Marcin,
Thank you for your suggestion.
The sub-sub version numbering you are mentioning refers to RPMs, PKGs, etc ..., while we are simply releasing a source code tarball. We are think of releasing source RPMs and I think your suggestion would be appropriate in that context.
Edo
Quote:Marcindulak Feb 9th 2:54 am
Could you when making changes to the source release it as 6.1.1 etc?
This will help to avoid confusion about which version people are referring to,
and it's a standard practice for installed software by package managers, like yum , yast, apt-get.
They recognize that a new version is available by comparing versions.

Gets Around
Sub versioning would facilitate my work - i'm in fact working on getting RPMS on 6.1
(see the bottom of http://nwchemgit.github.io/Special_AWCforum/st/id331/how_to_use_ASE%27s_nebcode_an...)

Clicked A Few Times
Quote:Edoapra Feb 9th 6:17 pm
Marcin,
Thank you for your suggestion.
The sub-sub version numbering you are mentioning refers to RPMs, PKGs, etc ..., while we are simply releasing a source code tarball. We are think of releasing source RPMs and I think your suggestion would be appropriate in that context.
Edo
Quote:Marcindulak Feb 9th 2:54 am
Could you when making changes to the source release it as 6.1.1 etc?
This will help to avoid confusion about which version people are referring to,
and it's a standard practice for installed software by package managers, like yum , yast, apt-get.
They recognize that a new version is available by comparing versions.


The problem with not changing the tarball is that we will not know which version it is used.
I mean shall we start using something like nwchem-6.1-before-Feb.-8 or nwchem-6.1-post-Feb.-8?
I believe that Marcindulak suggestion is sensible.

Thanks a lot, anyway, for your continuum support.

Best regards,

Rafael R. Pappalardo

Forum Vet
Sure, we will try to follow a similar convention for release tarballs.

Edo

Quote:Rafapa Feb 9th 10:58 am


The problem with not changing the tarball is that we will not know which version it is used.
I mean shall we start using something like nwchem-6.1-before-Feb.-8 or nwchem-6.1-post-Feb.-8?
I believe that Marcindulak suggestion is sensible.

Thanks a lot, anyway, for your continuum support.

Best regards,

Rafael R. Pappalardo

Gets Around
I would like us to consider another change in the naming convention.
Instead of: Nwchem-6.1-2012-Feb-10.tar.gz
could you use the most common way: Nwchem-6.1.0.tar.gz?

The current naming causes really a lot of troubles when packaging an RPM.
Apart from an unnecessary complication in the spec file (i don't think it's the right place to describe all the details,
still I describe some of them below)
i feel Nwchem-6.1.2012.02.10-X.{i686,x86_64}.rpm looks ugly compared to Nwchem-6.1.0-X.{i686,x86_64}.rpm

On a positive side i have (experimental) RPMS of 6.1 version (Feb 8) ready. They are accessible through software package managers,
for example for CentOS5:

$ yum install wget
$ cd /etc/yum.repos.d/
$ wget http://download.opensuse.org/repositories/home:/marcindulak/CentOS_CentOS-5/home:marcindul...

It's good to set enabled=0 in /etc/yum.repos.d/home:marcindulak.repo so you don't get other updates from there,
and then:

$ yum install nwchem --enablerepo=home_marcindulak

or as direct download http://download.opensuse.org/repositories/home:/marcindulak/CentOS_CentOS-5/
Build logs are avaliable (i run a small set of tests during build), for example
https://build.opensuse.org/package/rawlog?arch=i586&package=nwchem-6.1&project=hom...

For yast on openSUSE see https://wiki.fysik.dtu.dk/cmr/install/install.html#installation-with-a-package-manger

Let me explain some of the problems caused by the current name of the source.
RPM assumes naming of name-version-release.{i686,x86_64}.rpm, where release traditionally denotes a change in RPM build process,
not in the source (see http://www.rpm.org/max-rpm/ch-rpm-file-format.html).
Dashes are not allowed in version nor release.
Due to this requirement Nwchem-6.1-2012-Feb-10-X.{i686,x86_64}.rpm would be interpreted as:
name=Nwchem-6.1-2012-Feb
version=10
release=X
so package managers like (yum, yast, ...) would consider different nwchem RPMS as different programs, not different
versions of the same program!
Therefore i would choose to name the RPM Nwchem-6.1.2012.Feb.10-X.{i686,x86_64}.rpm (note the dots instead of dashes).
The problem would be that a future December 2012 Nwchem-6.1.2012.Dec.DD-Y would be considered earlier than the February release.
Therefore I would create an internal (in the RPM spec file) table of numeric values of months so the name would become
Nwchem-6.1.2012.02.10-X.{i686,x86_64}.rpm.

Marcin

Gets Around
I would like to go back to the main subject of the thread.
http://nwchemgit.github.io/images/Nwchem-6.1-2012-Feb-10.tar.gz compiles with scalapack
on Fedora 16 x86_64 with the following commands:

export MPIEXEC=/usr/lib64/openmpi/bin/mpiexec
export LARGE_FILES=TRUE
export USE_NOFSCHECK=TRUE
export TCGRSH=ssh
export NWCHEM_TOP=/rpmbuild/BUILD/nwchem-6.1.test
export NWCHEM_TARGET=LINUX64
export USE_MPI=y
export USE_MPIF=y
export USE_MPIF4=y
export MPI_LOC=/usr/lib64/openmpi/
export MPI_LIB=/usr/lib64/openmpi/lib
export MPI_INCLUDE=/usr/include/openmpi-x86_64
export PYTHONHOME=/usr
export PYTHONVERSION=2.7
export USE_PYTHON64=y
export 'LIBMPI=-L/usr/lib64/openmpi/lib -lmpi -lmpi_f90 -lmpi_f77'
export HAS_BLAS=yes
export 'BLASOPT=-L/usr/lib64/atlas -llapack -lf77blas -lcblas -latlas'
export USE_SCALAPACK=yes
export 'SCALAPACK=$(MPI_LIB)/libscalapack.a $(MPI_LIB)/libmpiblacsCinit.a $(MPI_LIB)/libmpiblacsF77init.a $(MPI_LIB)/libmpiblacs.a'
export CC=gcc
export FC=gfortran
export PYTHONLIBTYPE=so
sed -i 's/libpython$(PYTHONVERSION).a/libpython$(PYTHONVERSION).$(PYTHONLIBTYPE)/g' config/makefile.h
make nwchem_config 'NWCHEM_MODULES=all python' 2>&1 | tee make.nwchem_config.log
make 64_to_32 2>&1 | tee make.64_to_32.log
make USE_64TO32=y 2>&1 | tee make.log

Nevertheless, I have doubts, and would like to suggest two improvements:

4. the symbols are there:
nm -a /rpmbuild/BUILD/nwchem-6.1.test/bin/LINUX64/nwchem | grep pdsye
00000000029542e6 T ga_pdsyev_
...
0000000002a382c0 T pdsyev_

and QA/tests/h2o_opt test runs, but maybe it would be better if nwchem prints in the out file
if is actually uses scalapack or not (especially due to the point 5. below).

5. ga still does not fully finds scalapack, with src/tools/build/config.log:

configure:21122: checking whether SCALAPACK implements pdsyevr
configure:21181: gfortran -o conftest -L/usr/lib64/atlas -L/usr/lib64/openmpi/lib -L/usr/lib64/openmpi/lib -L/usr/lib64/openmpi//lib conftest.f /usr/lib64/openmpi/lib/libscalapack.a /usr/lib64/openmpi/lib/libmpiblacsCinit.a /usr/lib64/openmpi/lib/libmpiblacsF77init.a /usr/lib64/openmpi/lib/libmpiblacs.a -llapack -lf77blas -lcblas -latlas -lmpi -lmpi_f90 -lmpi_f77 -lm >&5
/tmp/ccKwIYKk.o: In function `MAIN__':
conftest.f:(.text+0xa): undefined reference to `pdsyevr_'
collect2: ld returned 1 exit status

pdsyevr (http://icl.cs.utk.edu/lapack-forum/archives/scalapack/msg00159.html) is a relatively new
addition to netlib scalapack, has become a part of scalapack 2.0.0 http://netlib.org/scalapack/scalapack-2.0.0.html,
and i'm not sure if/when intel mkl will have it (10.3.4.191 does not have it).
A warning in nwchem runs, when pdsyevr is supposed to be used and not available, would be nice in stderr.

Forum Vet
pdsyevr never used by default
Marcin,
pdsyevr is never used by default.
You have to explicitly set a flag in the input file to have NWChem use pdsyevr (instead of the default pdsyevd).
What I just said deals with the runtime behavior.
As far as installation is concerned, NWChem is not automatically linking pdsyevr.
The tools configure does check for the existence of pdsyevr (as you have noticed yourself). If pdsyer is available in the supplied
Scalapack library, the tools build the GA interface to it (ga_pdsyevr).
Then, another bit of automation is needed to get nwchem to insert a call ga_pdsyevr (this is not there yet).
Anyhow, even when the call to ga_pdsyevr is inserted in NWChem, pdsyevd remains the default.

I hope I made clear enough that the support for pdsyevr in NWChem 6.1 is completely experimental, unsupported and not advertised (yet).
Edo


Forum >> NWChem's corner >> Compiling NWChem