NWChem 6.6 for BlueGene/Q
Clicked A Few Times
1:11:55 AM PST - Wed, Mar 9th 2016
Hi,

I recently attempted to upgrade to version 6.6 on BG/Q system, and to compare the performance using various implementation of AMRCI, MPICH and BLAS.

It has no major problem if 64-bit BLAS is used (either using standard MPI-TS with MPICH2, non-supported MPI-TS with MPICH3, and even AMRCI-MPI3 with MPICH3)

But I got some problem when I used based using ESSL (64_to_32 is required) which haven't been seen for previous versions on the same machine.

(1) some header files in src/util/util_getppn.c is not included in the /bgsys/drivers/ppcfloor/comm/xl/include, e.g.
  1. if defined(__bgq__)
  2. include <process.h>
  3. include <location.h>
  4. include <personality.h>
...

so the MPI_INCLUDE has to be modify as following to include these headers files, e.g.

Contents

================================================
export MPI_INCLUDE="-I/bgsys/drivers/ppcfloor \
                   -I/bgsys/drivers/ppcfloor/firmware/include \

                   -I/bgsys/drivers/ppcfloor/spi/include \

                   -I/bgsys/drivers/ppcfloor/spi/include/kernel \

                   -I/bgsys/drivers/ppcfloor/spi/include/kernel/cnk  \

                   -I/bgsys/drivers/ppcfloor/comm/xl/include "
================================================

(2) in src/util/util_getppn.c, from line 35 assign *ppn_out on bgq, but it will get an extra "}" after preprocessing
================================================
#if defined(__bgq__)

   *ppn_out = Kernel_ProcessCount();
  1. elif MPI_VERSION >= 3
...
  1. endif
 errlab:

   GA_Error(" util_getppn failure", 0);

   return;

 }
}
================================================

(3) based on the standard MPI-TS procedure, after make 64_to_32 and defined USE_64TO32=yes, but I got a MA error:

================================================
MA fatal error: MA_sizeof: invalid datatype: 4350801871857
================================================

and I noticed that the error should be due to the some of fortran codes in GA (ver 5.4) were built using -qintsize=8 directive. So I edited src/tools/GNUmakefile to force the --enable-i4 is explicitly included for for configure GA, the program starts but crash after detecting the symmetry directive
================================================
                               NWChem Input Module

                               -------------------





Scaling coordinates for geometry "geometry" by  1.889725989

(inverse scale =  0.529177249)



Turning off AUTOSYM since

SYMMETRY directive was detected!

2016-03-09 15:51:34.170 (WARN ) [0xfff78848b10] 80299:ibm.runjob.client.Job: terminated by signal 11
2016-03-09 15:51:34.171 (WARN ) [0xfff78848b10] 80299:ibm.runjob.client.Job: abnormal termination by signal 11 from rank 0
================================================
I have not clue for the cause of this error, and can anyone suggest how to fix this problem? Thanks !


This is the setting I used to build MPI-TS
================================================
export NWCHEM_TOP=/scratch/home/chiensh/nwchem/nwchem-6.6
export NWCHEM_TARGET=BGQ
export ARMCI_NETWORK=MPI-TS
export USE_MPI=y
export USE_MPIF=y
export USE_MPIF4=y
export USE_XLF=y
export USE_I4FLAGS=y
export USE_64TO32=y
export FFLAG_INT="-qintsize=4"
export DISABLE_GAMIRROR=y
export NWCHEM_MODULES="all "
  1. export MRCC_METHODS=TRUE
  2. export NWCHEM_MODULES=smallqm

export MPI_INCLUDE="-I/bgsys/drivers/ppcfloor \
                   -I/bgsys/drivers/ppcfloor/firmware/include \

                   -I/bgsys/drivers/ppcfloor/spi/include \

                   -I/bgsys/drivers/ppcfloor/spi/include/kernel \

                   -I/bgsys/drivers/ppcfloor/spi/include/kernel/cnk  \

                   -I/bgsys/drivers/ppcfloor/comm/xl/include "

export BLAS_SIZE=4
export BLASOPT=" /bgsys/ibm_essl/prod/opt/ibmmath/essl/5.1/lib64/libesslbg.a \
                /apps/libs/lapack/essl/lapack_BGQ_XL_ESSL_INT4_.a \

                /apps/libs/blas/xl/blas_BGQ_XL_INT4_.a   -Wl,-zmuldefs"
export BLAS_LIB=" /bgsys/ibm_essl/prod/opt/ibmmath/essl/5.1/lib64/libesslbg.a \
               /apps/libs/lapack/essl/lapack_BGQ_XL_ESSL_INT4_.a \

                /apps/libs/blas/xl/blas_BGQ_XL_INT4_.a  -zmuldefs  "
================================================

Regards,
Dominic Chien

Clicked A Few Times
4:12:33 PM PDT - Mon, Jul 25th 2016
Hi Dominic,

I've been trying to compile NWChem 6.6 for the BG/Q recently as well, and I run into the same issues with the symmetry keyword as you describe above. Additionally, if I remove the symmetry keyword and use either autosym or noautosym (to force C1 symmetry), I encounter the following error:
                                 NWChem Input Module
                                 -------------------
 
 
 
  Scaling coordinates for geometry "geometry" by  1.889725989
  (inverse scale =  0.529177249)
 
  ------------------------------------------------------------------------
  dosymops: failed matching uniq                   0
  ------------------------------------------------------------------------
  ------------------------------------------------------------------------
   current input line :
      7: end
  ------------------------------------------------------------------------
  ------------------------------------------------------------------------
  There is an error in the input file
  ------------------------------------------------------------------------
  For more information see the NWChem manual at http://nwchemgit.github.io/index.php/NWChem_Documentation
 
 
  For further details see manual section:                                                                                                                                                                                                     
 0:dosymops: failed matching uniq:Received an Error in Communication 
 Abort(-1) on node 0 (rank 0 in comm -2080374784): application called MPI_Abort(comm=0x84000000, -1) - process 0
 2016-07-25 19:04:43.673 (WARN ) [0x40001069550] 784705:ibm.runjob.client.Job: normal termination with status 1 from rank 0

I'm continuing to investigate this issue, but I thought I would post this here in case an NWChem developer has any insights.

Best,
James Buchwald

Forum Vet
9:25:28 AM PDT - Tue, Jul 26th 2016
I would unset the following env. variables and recompile from scratch

export USE_I4FLAGS=y
export FFLAG_INT="-qintsize=4"

Clicked A Few Times
2:22:17 PM PDT - Tue, Jul 26th 2016
Hi Edoapra,

Thanks for the fast reply. I did not have FFLAG_INT set, but I removed USE_I4FLAGS and recompiled. Now I get an error when I run NWChem:
 MA fatal error: MA_sizeof: invalid datatype: 0

My environment settings are as follows:
export NWCHEM_TOP=/gpfs/u/scratch/QSFC/QSFCbcjm/nwchem-6.6
export NWCHEM_MODULES=qm
export NWCHEM_TARGET=BGQ
export ARMCI_NETWORK=MPI-TS
export USE_MPI=y
export USE_MPIF=y
export USE_MPIF4=y
export MPI_INCLUDE=/bgsys/drivers/ppcfloor/comm/xl/include
export LIBMPI=" "
ESSL="-L/bgsys/ibm_essl/prod/opt/ibmmath/lib64 -lesslbg -Wl,-zmuldefs "
LAPACK="-L/gpfs/u/barn/QSFC/shared/blis-07-22-2016/lib -llapack "
export BLASOPT="$ESSL $LAPACK $ESSL "
export BLAS_LIB="$ESSL "
export LAPACK_LIB="$ESSL $LAPACK $ESSL "
export BLAS_SIZE=4
export USE_64TO32=y
export PATH=/bgsys/drivers/ppcfloor/gnu-linux/bin:$PATH
export DISABLE_GAMIRROR=y

Thanks again for the help!

Forum Vet
5:14:24 PM PDT - Tue, Jul 26th 2016
Could you post the output of the commands

1) head -25 $NWCHEM_TOP/src/tools/build.config.log


2) cd $NWCHEM_TOP/src; make link

Clicked A Few Times
6:51:45 PM PDT - Tue, Jul 26th 2016
1) I don't seem to have a build.config.log, do you mean build/config.log? I have that one. Here's the first 25 lines:
 This file contains any messages produced by compilers while
 running configure, to aid debugging if configure makes a mistake.
 
 It was created by Global Arrays (GA) configure 5.4, which was
 generated by GNU Autoconf 2.69.  Invocation command line was
 
   $ ../ga-5-4/configure --prefix=/gpfs/u/scratch/QSFC/QSFCbcjm/nwchem-6.6/src/tools/install --with-tcgmsg --with-mpi=-I/bgsys/drivers/ppcfloor/comm/xl/include -L/bgsys/drivers/V1R2M4/ppc64/comm/lib -L/bgsys/drivers/V1R2M4/ppc64/comm/lib -L/bgsys/drivers/V1R2M4/ppc64/comm/lib64 -L/bgsys/drivers/V1R2M4/ppc64/comm/lib -L/bgsys/drivers/V1R2M4/ppc64/spi/lib -L/bgsys/drivers/V1R2M4/ppc64/comm/sys/lib -L/bgsys/drivers/V1R2M4/ppc64/spi/lib -L/bgsys/drivers/V1R2M4/ppc64/comm/sys/lib -L/bgsys/drivers/V1R2M4/ppc64/comm/lib64 -L/bgsys/drivers/V1R2M4/ppc64/comm/lib -L/bgsys/drivers/V1R2M4/ppc64/spi/lib -L/bgsys/drivers/V1R2M4/ppc64/comm/lib --enable-peigs --enable-underscoring --disable-mpi-tests --without-scalapack --with-lapack=-L/bgsys/ibm_essl/prod/opt/ibmmath/lib64 -lesslbg -Wl,-zmuldefs -L/gpfs/u/barn/QSFC/shared/blis-07-22-2016/lib -llapack -L/bgsys/ibm_essl/prod/opt/ibmmath/lib64 -lesslbg -Wl,-zmuldefs --with-blas4=-L/bgsys/ibm_essl/prod/opt/ibmmath/lib64 -lesslbg -Wl,-zmuldefs --with-mpi-ts CC=mpixlc_r CXX=/opt/ibmcmp/vacpp/bg/12.1/bin/bgxlc++ F77=mpixlf77_r ARMCI_DEFAULT_SHMMAX_UBOUND=131072 CPP=powerpc64-bgq-linux-cpp --enable-i4
 
 ## --------- ##
 ## Platform. ##
 ## --------- ##
 
 hostname = q.ccni.rpi.edu
 uname -m = ppc64
 uname -r = 2.6.32-642.3.1.el6.ppc64
 uname -s = Linux
 uname -v = #1 SMP Sun Jun 26 18:14:05 EDT 2016
 
 /usr/bin/uname -p = unknown
 /bin/uname -X     = unknown
 
 /bin/arch              = ppc64
 /usr/bin/arch -k       = unknown
 /usr/convex/getsysinfo = unknown
 /usr/bin/hostinfo      = unknown

2) Output of make link:
 make nwchem.o stubs.o
 make[1]: Entering directory `/gpfs/u/scratch/QSFC/QSFCbcjm/nwchem-6.6/src'
 Converting nwchem.F -> nwchem.f
 mpixlf77_r -c -qintsize=8  -g -qEXTNAME -qxlf77=leadzero -qthreaded -qnosave  -g -qstrict -O2  nwchem.f
 ** nwchem   === End of Compilation 1 ===
 ** nwchem_banner   === End of Compilation 2 ===
 ** nw_print_restart_info   === End of Compilation 3 ===
 1501-510  Compilation successful for file nwchem.f.
 Converting stubs.F -> stubs.f
 mpixlf77_r -c -qintsize=8  -g -qEXTNAME -qxlf77=leadzero -qthreaded -qnosave  -g -qstrict -O2  stubs.f
 ** module_census   === End of Compilation 1 ===
 ** ana_input   === End of Compilation 2 ===
 ** analyz   === End of Compilation 3 ===
 ** argos   === End of Compilation 4 ===
 ** argos_input   === End of Compilation 5 ===
 ** band_energy   === End of Compilation 6 ===
 ** band_gradient   === End of Compilation 7 ===
 ** cafe   === End of Compilation 8 ===
 ** ccca_input   === End of Compilation 9 ===
 ** crossing   === End of Compilation 10 ===
 ** dangchang_input   === End of Compilation 11 ===
 ** dc_energy   === End of Compilation 12 ===
 ** debtest   === End of Compilation 13 ===
 ** dia_input   === End of Compilation 14 ===
 ** diana   === End of Compilation 15 ===
 ** dntmc_input   === End of Compilation 16 ===
 ** drdy_input   === End of Compilation 17 ===
 ** fft   === End of Compilation 18 ===
 ** gasdev   === End of Compilation 19 ===
 ** geninterface   === End of Compilation 20 ===
 ** jantest   === End of Compilation 21 ===
 ** jvltest   === End of Compilation 22 ===
 ** jvltest2   === End of Compilation 23 ===
 ** kgdtest   === End of Compilation 24 ===
 ** leps_energy   === End of Compilation 25 ===
 ** leps_gradient   === End of Compilation 26 ===
 ** lucia   === End of Compilation 27 ===
 ** mc_data   === End of Compilation 28 ===
 ** mc_driver   === End of Compilation 29 ===
 ** mc_init   === End of Compilation 30 ===
 ** mc_main   === End of Compilation 31 ===
 ** md_data   === End of Compilation 32 ===
 ** md_driver   === End of Compilation 33 ===
 ** md_input   === End of Compilation 34 ===
 ** md_set   === End of Compilation 35 ===
 ** mm_add_egrad   === End of Compilation 36 ===
 ** mm_add_energy   === End of Compilation 37 ===
 ** mm_input   === End of Compilation 38 ===
 ** mmi_end   === End of Compilation 39 ===
 ** mmi_init   === End of Compilation 40 ===
 ** moints_screen   === End of Compilation 41 ===
 ** mymc_input   === End of Compilation 42 ===
 ** mymd_input   === End of Compilation 43 ===
 ** nbo_input   === End of Compilation 44 ===
 ** nwmd   === End of Compilation 45 ===
 ** nwpw   === End of Compilation 46 ===
 ** nwpw_input   === End of Compilation 47 ===
 ** nwxc_eval_df   === End of Compilation 48 ===
 ** nwxc_eval_df2   === End of Compilation 49 ===
 ** nwxc_eval_df3   === End of Compilation 50 ===
 ** nwxc_get_cam   === End of Compilation 51 ===
 ** nwxc_get_disp   === End of Compilation 52 ===
 ** nwxc_getvals   === End of Compilation 53 ===
 ** nwxc_getwght   === End of Compilation 54 ===
 ** nwxc_has_2nd   === End of Compilation 55 ===
 ** nwxc_has_cam   === End of Compilation 56 ===
 ** nwxc_has_disp   === End of Compilation 57 ===
 ** nwxc_has_hfx   === End of Compilation 58 ===
 ** nwxc_has_mp2c   === End of Compilation 59 ===
 ** nwxc_has_xc   === End of Compilation 60 ===
 ** nwxc_init   === End of Compilation 61 ===
 ** nwxc_input   === End of Compilation 62 ===
 ** nwxc_is_gga   === End of Compilation 63 ===
 ** nwxc_is_lda   === End of Compilation 64 ===
 ** nwxc_is_mgga   === End of Compilation 65 ===
 ** nwxc_is_on   === End of Compilation 66 ===
 ** nwxc_print   === End of Compilation 67 ===
 ** nwxc_print_nwpw   === End of Compilation 68 ===
 ** nwxc_reset_rho_tolerance   === End of Compilation 69 ===
 ** nwxc_rtdb_delete   === End of Compilation 70 ===
 ** nwxc_rtdb_load   === End of Compilation 71 ===
 ** nwxc_rtdb_store   === End of Compilation 72 ===
 ** nwxc_vdw3_dftd3   === End of Compilation 73 ===
 ** nwxc_wght_hfx   === End of Compilation 74 ===
 ** nwxc_wght_mp2c   === End of Compilation 75 ===
 ** ofpw_energy   === End of Compilation 76 ===
 ** paw_energy   === End of Compilation 77 ===
 ** paw_gradient   === End of Compilation 78 ===
 ** perfm   === End of Compilation 79 ===
 ** pre_input   === End of Compilation 80 ===
 ** prepar   === End of Compilation 81 ===
 ** pspw_bq_gradient   === End of Compilation 82 ===
 ** pspw_energy   === End of Compilation 83 ===
 ** pspw_gradient   === End of Compilation 84 ===
 ** pspw_input   === End of Compilation 85 ===
 ** python_input   === End of Compilation 86 ===
 ** qhop   === End of Compilation 87 ===
 ** qmd_driver   === End of Compilation 88 ===
 ** qmd_forces   === End of Compilation 89 ===
 ** qmd_input   === End of Compilation 90 ===
 ** qmd_main   === End of Compilation 91 ===
 ** qmd_setup   === End of Compilation 92 ===
 ** qmd_start   === End of Compilation 93 ===
 ** qmd_wrttrj   === End of Compilation 94 ===
 ** qmmm_bq_forces   === End of Compilation 95 ===
 ** qmmm_end   === End of Compilation 96 ===
 ** qmmm_forces   === End of Compilation 97 ===
 ** qmmm_init   === End of Compilation 98 ===
 ** qmmm_input   === End of Compilation 99 ===
 ** qmmm_print_energy1   === End of Compilation 100 ===
 ** raktest   === End of Compilation 101 ===
 ** rimp2g   === End of Compilation 102 ===
 ** rism_input   === End of Compilation 103 ===
 ** rjhtest   === End of Compilation 104 ===
 ** smd_input   === End of Compilation 105 ===
 ** space   === End of Compilation 106 ===
 ** task_band   === End of Compilation 107 ===
 ** task_ccca   === End of Compilation 108 ===
 ** task_ccca_energy   === End of Compilation 109 ===
 ** task_dntmc   === End of Compilation 110 ===
 ** task_drdy   === End of Compilation 111 ===
 ** task_fragment_energy   === End of Compilation 112 ===
 ** task_nbo   === End of Compilation 113 ===
 ** task_ofpw   === End of Compilation 114 ===
 ** task_paw   === End of Compilation 115 ===
 ** task_pspw   === End of Compilation 116 ===
 ** task_python   === End of Compilation 117 ===
 ** task_qmd   === End of Compilation 118 ===
 ** task_qmmm_abc   === End of Compilation 119 ===
 ** task_qmmm_dynamics   === End of Compilation 120 ===
 ** task_qmmm_energy   === End of Compilation 121 ===
 ** task_qmmm_esp   === End of Compilation 122 ===
 ** task_qmmm_fep   === End of Compilation 123 ===
 ** task_qmmm_gradient   === End of Compilation 124 ===
 ** task_qmmm_optimize   === End of Compilation 125 ===
 ** task_qmmm_pes   === End of Compilation 126 ===
 ** task_qmmm_thermo   === End of Compilation 127 ===
 ** task_rism   === End of Compilation 128 ===
 ** task_smd   === End of Compilation 129 ===
 ** task_smd_dynamics   === End of Compilation 130 ===
 ** task_smd_energy   === End of Compilation 131 ===
 ** task_smd_fragment   === End of Compilation 132 ===
 ** uccsdt   === End of Compilation 133 ===
 ** uccsdtest   === End of Compilation 134 ===
 ** urand   === End of Compilation 135 ===
 1501-510  Compilation successful for file stubs.f.
 make[1]: Leaving directory `/gpfs/u/scratch/QSFC/QSFCbcjm/nwchem-6.6/src'
 mpixlf77_r -qintsize=8  -g -qEXTNAME -qxlf77=leadzero -qthreaded -qnosave  -g -qstrict -O2  -Wl,--relax -L/gpfs/u/scratch/QSFC/QSFCbcjm/nwchem-6.6/lib/BGQ -L/gpfs/u/scratch/QSFC/QSFCbcjm/nwchem-6.6/src/tools/install/lib   -o /gpfs/u/scratch/QSFC/QSFCbcjm/nwchem-6.6/bin/BGQ/nwchem nwchem.o stubs.o -lnwctask -lccsd -lmcscf -lselci -lmp2 -lmoints -lstepper -ldriver -loptim -lnwdft -lgradients -lcphf -lesp -lddscf -lguess -lhessian -lvib -lnwcutil -lrimp2 -lproperty -lsolvation -lnwints -ldplot -lvscf -letrans -ltce -lbq -lcons -lnwcutil -lga -larmci -lpeigs -lperfm -lcons -lbq -lnwcutil -lnwclapack -L/bgsys/ibm_essl/prod/opt/ibmmath/lib64 -lesslbg -Wl,-zmuldefs  -L/gpfs/u/barn/QSFC/shared/blis-07-22-2016/lib -llapack  -L/bgsys/ibm_essl/prod/opt/ibmmath/lib64 -lesslbg -Wl,-zmuldefs   -lnwcblas   -l64to32 -L/bgsys/ibm_essl/prod/opt/ibmmath/lib64 -lesslbg -Wl,-zmuldefs  -L/gpfs/u/barn/QSFC/shared/blis-07-22-2016/lib -llapack  -L/bgsys/ibm_essl/prod/opt/ibmmath/lib64 -lesslbg -Wl,-zmuldefs     -L/bgsys/drivers/V1R2M4/ppc64/comm/lib -L/bgsys/drivers/V1R2M4/ppc64/comm/lib -L/bgsys/drivers/V1R2M4/ppc64/comm/lib64 -L/bgsys/drivers/V1R2M4/ppc64/comm/lib -L/bgsys/drivers/V1R2M4/ppc64/spi/lib -L/bgsys/drivers/V1R2M4/ppc64/comm/sys/lib -L/bgsys/drivers/V1R2M4/ppc64/spi/lib -L/bgsys/drivers/V1R2M4/ppc64/comm/sys/lib -L/bgsys/drivers/V1R2M4/ppc64/comm/lib64 -L/bgsys/drivers/V1R2M4/ppc64/comm/lib -L/bgsys/drivers/V1R2M4/ppc64/spi/lib -L/bgsys/drivers/V1R2M4/ppc64/comm/lib        -lpthread
 link is built

Forum Vet
10:31:07 AM PDT - Wed, Jul 27th 2016
--enable-i4
I am not quite sure why the "--enable-i4" argument is used in your configuration of the tools, do you?

Clicked A Few Times
7:36:44 PM PDT - Wed, Jul 27th 2016
It was manually added to src/tools/GNUmakefile, as in Chiensh's original post above. Without that line, the code will compile but NWChem will immediately crash with
 MA fatal error: MA_sizeof: invalid datatype: 4350801871857

I will try compiling without --enable-i4 and without the environment variables you suggested to unset, and see if that helps.

As always, thanks for the help!

Clicked A Few Times
7:52:49 AM PDT - Thu, Jul 28th 2016
Okay, removing --enable-i4 and unsetting USE_I4FLAGS appears to fix the previous error. However, with a test input file:
 start nw-test-bgq
 scratch_dir /gpfs/u/scratch/QSFC/QSFCbcjm
 
 geometry noautosym
   H   0.0000  0.0000  -0.5000
   H   0.0000  0.0000   0.5000
 end
 
 basis spherical
   H  library  "aug-cc-pvtz"
 end
 
 dft
   xc m11
   grid xfine
   maxiter 100
 end
 title "H2 geometry optimization, aug-cc-pvtz"
 task dft optimize

I get the following error:
       Screening Tolerance Information
       -------------------------------
           Density screening/tol_rho: 1.00D-10
           AO Gaussian exp screening on grid/accAOfunc:  14
           CD Gaussian exp screening on grid/accCDfunc:  20
           XC Gaussian exp screening on grid/accXCfunc:  20
           Schwarz screening/accCoul: 1.00D-08 

  
       Superposition of Atomic Density Guess
       -------------------------------------
  
  Sum of atomic energies:          -0.99964326
  Negative value orthog:  -0.134681141935900293E+273
  Negative value orthog:  -0.134681141935900293E+273
  Negative value orthog:  -0.134681141935900293E+273
  Negative value orthog:  -0.134681141935900293E+273
  Negative value orthog:  -0.134681141935900293E+273
  Negative value orthog:  -0.134681141935900293E+273
  ------------------------------------------------------------------------
  ga_orthog: negative                   1

Do you have any insight regarding this error?

Forum Vet
10:03:04 AM PDT - Thu, Jul 28th 2016
64_to_32
did you execute

rm -f 64_to_32; make 64_to_32

before typing

make

?

Clicked A Few Times
1:33:41 PM PDT - Thu, Jul 28th 2016
I had originally run make 64_to_32 prior to compiling the code, but hadn't after the latest make realclean. I executed these commands today, successfully built the nwchem executable, but continue to obtain the error messages related to ga_orthog from my last post.

Clicked A Few Times
7:06:59 PM PDT - Mon, Aug 8th 2016
Second attempt, fully documented
Just to make sure the error is reproducible, I have carried out the following steps.

1) Download NWChem 6.6 sources from the downloads page, and apply all available patches.

2) Edit src/tools/GNUmakefile. To avoid an error in a ga configure script related to the C preprocessor, add to the end of CONFIGURE_ARGS (line 532):
 CPP=powerpc64-bgq-linux-cpp


3) Edit src/config/makefile.h. Related to the above C preprocessor error, add a new line near line 2106 after RANLIB = ...:
 CPP = powerpc64-bgq-linux-cpp -P -C -traditional


4) Edit src/util/util_getppn.c. This file will not compile as-is on BG/Q; the following changes are required. Near the beginning, change the BG/Q-specific headers to:
 #if defined(__bgq__)
 #include <kernel/process.h>
 #include <kernel/location.h>
 #include <sys/personality.h>
 #elif ...

Near line 37, fix the unmatched brace by rewriting the code as
 if (1) {
   *ppn_out = Kernel_ProcessCount();


5) Set the environment variables as follows:
 export NWCHEM_TOP=/gpfs/u/scratch/QSFC/QSFCbcjm/nwchem-6.6
 export NWCHEM_MODULES=qm
 export NWCHEM_TARGET=BGQ
 export ARMCI_NETWORK=MPI-TS
 export USE_MPI=y
 export USE_MPIF=y
 export USE_MPIF4=y
 export MPI_INCLUDE=/bgsys/drivers/ppcfloor/comm/xl/include
 export LIBMPI=" "
 ESSL="-L/bgsys/ibm_essl/prod/opt/ibmmath/lib64 -lesslbg "
 LAPACK="-L/gpfs/u/barn/QSFC/shared/lapack-3.6.0/lib -llapack "                                      
 export BLASOPT="-Wl,-zmuldefs $ESSL $LAPACK $ESSL -lxlsmp"                                                                                                                                                       
 export BLAS_LIB="$ESSL "
 export LAPACK_LIB="$ESSL $LAPACK "                                                                                                                                    
 export BLAS_SIZE=4
 export USE_64TO32=y
 export PATH=/bgsys/drivers/ppcfloor/gnu-linux/bin:$PATH
 export DISABLE_GAMIRROR=y


6) cd src; make nwchem_config; make 64_to_32

7) make

This produces a nwchem binary with the same problem as described above. Using a simple test case:
 start nw-test-bgq
 scratch_dir /gpfs/u/scratch/QSFC/QSFCbcjm
 
 geometry noautosym
   H   0.0000  0.0000  -0.5000
   H   0.0000  0.0000   0.5000
 end
 
 basis spherical
   H  library  "aug-cc-pvtz"
 end
 
 dft
   xc m11
   grid ultrafine
   maxiter 100
 end
 title "H2 geometry optimization, aug-cc-pvtz"
 task dft optimize

the calculation reaches the initial guess step and then crashes with a ga_orthog error:
      Screening Tolerance Information
      -------------------------------
          Density screening/tol_rho: 1.00D-10
          AO Gaussian exp screening on grid/accAOfunc:  14
          CD Gaussian exp screening on grid/accCDfunc:  20
          XC Gaussian exp screening on grid/accXCfunc:  20
          Schwarz screening/accCoul: 1.00D-08

 
      Superposition of Atomic Density Guess
      -------------------------------------
 
 Sum of atomic energies:          -0.99964326
 Negative value orthog:  -0.585575734166851874E+258
 Negative value orthog:  -0.585575734166851874E+258
 Negative value orthog:  -0.585575734166851874E+258
 Negative value orthog:  -0.585575734166851874E+258
 Negative value orthog:  -0.585575734166851874E+258
 Negative value orthog:  -0.585575734166851874E+258
 ------------------------------------------------------------------------
 ga_orthog: negative                   1

I am unsure of why this error is occurring, or how to fix it (though I will keep trying).

Clicked A Few Times
7:31:51 AM PDT - Wed, Aug 10th 2016
Problem is independent of BLAS implementation
Since the bug I'm seeing is in ga_orthog, I thought that there might be something funny going on with ESSL. I have built a new copy of NWChem 6.6 with BLASOPT unset and USE_INTERNALBLAS=y. The same problem as above continues to appear.

Forum Vet
1:53:31 PM PDT - Wed, Aug 10th 2016
Buchwj
I have been able to reproduce your failure but I am still working at figuring out the most reliable fix for it

Forum Vet
3:55:08 PM PDT - Wed, Aug 10th 2016
patches available
The following patch should fix your ga_orthog failure

http://nwchemgit.github.io/download.php?f=Util_gnumakefile.patch.gz

To apply
cd $NWCHEM_TOP

wget -O Util_gnumakefile.patch.gz http://nwchemgit.github.io/download.php?f=Util_gnumakefile.patch.gz

gzip -d Util_gnumakefile.patch.gz

patch -p0 < Util_gnumakefile.patch

cd src/util 

make 64_to_32

make

cd ..

make link

Please execute all the steps in order listed and do not skip any of them.
Let me how it goes

Clicked A Few Times
10:49:54 PM PDT - Wed, Aug 10th 2016
Dr. Apra,

The new patch works perfectly! I am able to run NWChem 6.6 on BG/Q with no issues!
Thanks again for all the help. Much appreciated!

James


Forum >> NWChem's corner >> Compiling NWChem