NWChem 6.6 for BlueGene/Q


Click here for full thread
Clicked A Few Times
Second attempt, fully documented
Just to make sure the error is reproducible, I have carried out the following steps.

1) Download NWChem 6.6 sources from the downloads page, and apply all available patches.

2) Edit src/tools/GNUmakefile. To avoid an error in a ga configure script related to the C preprocessor, add to the end of CONFIGURE_ARGS (line 532):
 CPP=powerpc64-bgq-linux-cpp


3) Edit src/config/makefile.h. Related to the above C preprocessor error, add a new line near line 2106 after RANLIB = ...:
 CPP = powerpc64-bgq-linux-cpp -P -C -traditional


4) Edit src/util/util_getppn.c. This file will not compile as-is on BG/Q; the following changes are required. Near the beginning, change the BG/Q-specific headers to:
 #if defined(__bgq__)
 #include <kernel/process.h>
 #include <kernel/location.h>
 #include <sys/personality.h>
 #elif ...

Near line 37, fix the unmatched brace by rewriting the code as
 if (1) {
   *ppn_out = Kernel_ProcessCount();


5) Set the environment variables as follows:
 export NWCHEM_TOP=/gpfs/u/scratch/QSFC/QSFCbcjm/nwchem-6.6
 export NWCHEM_MODULES=qm
 export NWCHEM_TARGET=BGQ
 export ARMCI_NETWORK=MPI-TS
 export USE_MPI=y
 export USE_MPIF=y
 export USE_MPIF4=y
 export MPI_INCLUDE=/bgsys/drivers/ppcfloor/comm/xl/include
 export LIBMPI=" "
 ESSL="-L/bgsys/ibm_essl/prod/opt/ibmmath/lib64 -lesslbg "
 LAPACK="-L/gpfs/u/barn/QSFC/shared/lapack-3.6.0/lib -llapack "                                      
 export BLASOPT="-Wl,-zmuldefs $ESSL $LAPACK $ESSL -lxlsmp"                                                                                                                                                       
 export BLAS_LIB="$ESSL "
 export LAPACK_LIB="$ESSL $LAPACK "                                                                                                                                    
 export BLAS_SIZE=4
 export USE_64TO32=y
 export PATH=/bgsys/drivers/ppcfloor/gnu-linux/bin:$PATH
 export DISABLE_GAMIRROR=y


6) cd src; make nwchem_config; make 64_to_32

7) make

This produces a nwchem binary with the same problem as described above. Using a simple test case:
 start nw-test-bgq
 scratch_dir /gpfs/u/scratch/QSFC/QSFCbcjm
 
 geometry noautosym
   H   0.0000  0.0000  -0.5000
   H   0.0000  0.0000   0.5000
 end
 
 basis spherical
   H  library  "aug-cc-pvtz"
 end
 
 dft
   xc m11
   grid ultrafine
   maxiter 100
 end
 title "H2 geometry optimization, aug-cc-pvtz"
 task dft optimize

the calculation reaches the initial guess step and then crashes with a ga_orthog error:
      Screening Tolerance Information
      -------------------------------
          Density screening/tol_rho: 1.00D-10
          AO Gaussian exp screening on grid/accAOfunc:  14
          CD Gaussian exp screening on grid/accCDfunc:  20
          XC Gaussian exp screening on grid/accXCfunc:  20
          Schwarz screening/accCoul: 1.00D-08

 
      Superposition of Atomic Density Guess
      -------------------------------------
 
 Sum of atomic energies:          -0.99964326
 Negative value orthog:  -0.585575734166851874E+258
 Negative value orthog:  -0.585575734166851874E+258
 Negative value orthog:  -0.585575734166851874E+258
 Negative value orthog:  -0.585575734166851874E+258
 Negative value orthog:  -0.585575734166851874E+258
 Negative value orthog:  -0.585575734166851874E+258
 ------------------------------------------------------------------------
 ga_orthog: negative                   1

I am unsure of why this error is occurring, or how to fix it (though I will keep trying).