ARMCI Errors with MKL ScaLAPACK

Just Got Here

11:51:47 AM PDT - Wed, Mar 13th 2013

Hi All. Although this is the "running NWChem" section, it may be that my problem lies in compilations. I've been trying to compile and run NWChem with Intel MKL BLAS and ScaLAPACK. Both compile successfully, and with BLAS alone the code runs successfully also. For ScaLAPACK however, I get a number of error messages. I've searched for these messages in other threads, but haven't found solutions. Here is the out file from the point that I start to receive warnings and errors (as compared to the non-ScaLAPACK run that works):

WARNING : Found   702 linear dependencies 
 S eigenvalue threshold:     1.00000E-05
 Smallest S eigenvalue :     0.00000E+00
 Largest  S eigenvalue :     0.00000E+00
 

 !! The overlap matrix has 702 vectors deemed linearly dependent with
    eigenvalues:
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00 0.00D+00
 
0:0:pnga_create_config:ga_set_data:wrong dimension specified:: 0
(rank:0 hostname:n017 pid:15607):ARMCI DASSERT fail. ../../ga-5-1/armci/src/common/armci.c:ARMCI_Error():208 cond:0
2:2:pnga_create_config:ga_set_data:wrong dimension specified:: 0
(rank:2 hostname:n017 pid:15609):ARMCI DASSERT fail. ../../ga-5-1/armci/src/common/armci.c:ARMCI_Error():208 cond:0
5:5:pnga_create_config:ga_set_data:wrong dimension specified:: 0
(rank:5 hostname:n018 pid:15898):ARMCI DASSERT fail. ../../ga-5-1/armci/src/common/armci.c:ARMCI_Error():208 cond:0
12:12:pnga_create_config:ga_set_data:wrong dimension specified:: 0
(rank:12 hostname:n008 pid:30328):ARMCI DASSERT fail. ../../ga-5-1/armci/src/common/armci.c:ARMCI_Error():208 cond:0
11:11:pnga_create_config:ga_set_data:wrong dimension specified:: 0
(rank:11 hostname:n019 pid:13369):ARMCI DASSERT fail. ../../ga-5-1/armci/src/common/armci.c:ARMCI_Error():208 cond:0
23:23:pnga_create_config:ga_set_data:wrong dimension specified:: 0
(rank:23 hostname:n016 pid:18750):ARMCI DASSERT fail. ../../ga-5-1/armci/src/common/armci.c:ARMCI_Error():208 cond:0
29:29:pnga_create_config:ga_set_data:wrong dimension specified:: 0
(rank:29 hostname:n009 pid:124701):ARMCI DASSERT fail. ../../ga-5-1/armci/src/common/armci.c:ARMCI_Error():208 cond:0
13:13:pnga_create_config:ga_set_data:wrong dimension specified:: 0
(rank:13 hostname:n008 pid:30330):ARMCI DASSERT fail. ../../ga-5-1/armci/src/common/armci.c:ARMCI_Error():208 cond:0
1:1:pnga_create_config:ga_set_data:wrong dimension specified:: 0
(rank:1 hostname:n017 pid:15606):ARMCI DASSERT fail. ../../ga-5-1/armci/src/common/armci.c:ARMCI_Error():208 cond:0
6:6:pnga_create_config:ga_set_data:wrong dimension specified:: 0
(rank:6 hostname:n018 pid:15896):ARMCI DASSERT fail. ../../ga-5-1/armci/src/common/armci.c:ARMCI_Error():208 cond:0
24:24:pnga_create_config:ga_set_data:wrong dimension specified:: 0
(rank:24 hostname:n015 pid:23657):ARMCI DASSERT fail. ../../ga-5-1/armci/src/common/armci.c:ARMCI_Error():208 cond:0
28:28:pnga_create_config:ga_set_data:wrong dimension specified:: 0
(rank:28 hostname:n009 pid:124700):ARMCI Drank 21 in job 1  n017_37473   caused collective abort of all ranks
  exit status of rank 21: killed by signal 9 
ASSERT fail. ../../ga-5-1/armci/src/common/armci.c:ARMCI_Error():208 cond:0
14:14:pnga_create_config:ga_set_data:wrong dimension specified:: 0
(rank:14 hostname:n008 pid:30329):ARMCI DASSERT fail. ../../ga-5-1/armci/src/common/armci.c:ARMCI_Error():208 cond:0
25:25:pnga_create_config:ga_set_data:wrong dimension specified:: 0
(rank:25 hostname:n015 pid:23659):ARMCI DASSERT fail. ../../ga-5-1/armci/src/common/armci.c:ARMCI_Error():208 cond:0
26:26:pnga_create_config:ga_set_data:wrong dimension specified:: 0
(rank:26 hostname:n015 pid:23658):ARMCI DASSERT fail. ../../ga-5-1/armci/src/common/armci.c:ARMCI_Error():208 cond:0
3:3:pnga_create_config:ga_set_data:wrong dimension specified:: 0
(rank:3 hostname:n017 pid:15608):ARMCI DASSERT fail. ../../ga-5-1/armci/src/common/armci.c:ARMCI_Error():208 cond:0
8:8:pnga_create_config:ga_set_data:wrong dimension specified:: 0
(rank:8 hostname:n019 pid:13371):ARMCI DASSERT fail. ../../ga-5-1/armci/src/common/armci.c:ARMCI_Error():208 cond:0
17:17:pnga_create_config:ga_set_data:wrong dimension specified:: 0
(rank:17 hostname:n014 pid:24858):ARMCI DASSERT fail. ../../ga-5-1/armci/src/common/armci.c:ARMCI_Error():208 cond:0
7:7:pnga_create_config:ga_set_data:wrong dimension specified:: 0
(rank:7 hostname:n018 pid:15899):ARMCI DASSERT fail. ../../ga-5-1/armci/src/common/armci.c:ARMCI_Error():208 cond:0
16:16:pnga_create_config:ga_set_data:wrong dimension specified:: 0
(rank:16 hostname:n014 pid:24857):ARMCI DASSERT fail. ../../ga-5-1/armci/src/common/armci.c:ARMCI_Error():208 cond:0
15:15:pnga_create_config:ga_set_data:wrong dimension specified:: 0
(rank:15 hostname:n008 pid:30331):ARMCI DASSERT fail. ../../ga-5-1/armci/src/common/armci.c:ARMCI_Error():208 cond:0
31:31:pnga_create_config:ga_set_data:wrong dimension specified:: 0
(rank:31 hostname:n009 pid:124703):ARMCI DASSERT fail. ../../ga-5-1/armci/src/common/armci.c:ARMCI_Error():208 cond:0
18:18:pnga_create_config:ga_set_data:wrong dimension specified:: 0
(rank:18 hostname:n014 pid:24860):ARMCI DASSERT fail. ../../ga-5-1/armci/src/common/armci.c:ARMCI_Error():208 cond:0
4:4:pnga_create_config:ga_set_data:wrong dimension specified:: 0
(rank:4 hostname:n018 pid:15897):ARMCI DASSERT fail. ../../ga-5-1/armci/src/common/armci.c:ARMCI_Error():208 cond:0
10:10:pnga_create_config:ga_set_data:wrong dimension specified:: 0
(rank:10 hostname:n019 pid:13372):ARMCI DASSERT fail. ../../ga-5-1/armci/src/common/armci.c:ARMCI_Error():208 cond:0
21:21:pnga_create_config:ga_set_data:wrong dimension specified:: 0
(rank:21 hostname:n016 pid:18751):ARMCI DASSERT fail. ../../ga-5-1/armci/src/common/armci.c:ARMCI_Error():208 cond:0
22:22:pnga_create_config:ga_set_data:wrong dimension specified:: 0
(rank:22 hostname:n016 pid:18752):ARMCI DASSERT fail. ../../ga-5-1/armci/src/common/armci.c:ARMCI_Error():208 cond:0
9:9:pnga_create_config:ga_set_data:wrong dimension specified:: 0
(rank:9 hostname:n019 pid:13370):ARMCI DASSERT fail. ../../ga-5-1/armci/src/common/armci.c:ARMCI_Error():208 cond:0
30:30:pnga_create_config:ga_set_data:wrong dimension specified:: 0
(rank:30 hostname:n009 pid:124702):ARMCI DASSERT fail. ../../ga-5-1/armci/src/common/armci.c:ARMCI_Error():208 cond:0
27:27:pnga_create_config:ga_set_data:wrong dimension specified:: 0
(rank:27 hostname:n015 pid:23656):ARMCI DASSERT fail. ../../ga-5-1/armci/src/common/armci.c:ARMCI_Error():208 cond:0
19:19:pnga_create_config:ga_set_data:wrong dimension specified:: 0
(rank:19 hostname:n014 pid:24859):ARMCI DASSERT fail. ../../ga-5-1/armci/src/common/armci.c:ARMCI_Error():208 cond:0
20:20:pnga_create_config:ga_set_data:wrong dimension specified:: 0
(rank:20 hostname:n016 pid:18749):ARMCI DASSERT fail. ../../ga-5-1/armci/src/common/armci.c:ARMCI_Error():208 cond:0
rank 27 in job 1  n017_37473   caused collective abort of all ranks
  exit status of rank 27: killed by signal 9 
rank 31 in job 1  n017_37473   caused collective abort of all ranks
  exit status of rank 31: killed by signal 9 
rank 29 in job 1  n017_37473   caused collective abort of all ranks
  exit status of rank 29: killed by signal 9 
rank 11 in job 1  n017_37473   caused collective abort of all ranks
  exit status of rank 11: killed by signal 9 
rank 19 in job 1  n017_37473   caused collective abort of all ranks
  exit status of rank 19: killed by signal 9 
rank 7 in job 1  n017_37473   caused collective abort of all ranks
  exit status of rank 7: killed by signal 9 
rank 6 in job 1  n017_37473   caused collective abort of all ranks
  exit status of rank 6: killed by signal 9 
rank 15 in job 1  n017_37473   caused collective abort of all ranks
  exit status of rank 15: killed by signal 9 
rank 13 in job 1  n017_37473   caused collective abort of all ranks
  exit status of rank 13: killed by signal 9 
rank 3 in job 1  n017_37473   caused collective abort of all ranks
  exit status of rank 3: killed by signal 9

Any ideas? Here is my submission file:

RESTART comp1-arm
SCRATCH_DIR   ./tmp
PERMANENT_DIR ./perm
MEMORY total 4 gb
ECCE_PRINT ./ecce/react
ECHO
#
geometry noautosym nocenter
 o                   -10.28637712     4.61966189    -5.39287590
 c                   -11.11768366     5.42635357    -4.95534853
 n                   -12.34472699     5.02944581    -4.54153201
 c                   -12.79877547     3.64703492    -4.58111847
 c                   -10.80560699     6.90779147    -4.84320506
 h                   -12.98964026     5.73432269    -4.21361480
 h                   -13.67842060     3.54590380    -5.22564896
 h                   -13.04967199     3.29232606    -3.57572536
 h                   -11.98682453     3.04302516    -4.98545846
 h                   -11.65514108     7.51228854    -4.51277437
 h                   -10.46778430     7.27176047    -5.81797717
 h                    -9.98227841     7.04496553    -4.13418092
 C                    -4.12932144    -1.32356068    -0.51535190
 C                    -3.11469791    -0.28811770    -0.49542853
 N                    -1.87026718    -0.87349486    -0.39190481
 C                    -2.06822120    -2.23092771    -0.34033812
 C                    -3.48615111    -2.51869935    -0.41109388
 C                    -3.39702040     1.08771162    -0.54440414
 C                    -2.39781435     2.07556797    -0.45055101
 N                    -1.02576984     1.85740329    -0.41186836
 C                    -0.41919738     3.09892718    -0.31296902
 C                    -1.43386732     4.12245305    -0.29056459
 C                    -2.64492263     3.49390384    -0.38012744
 C                     0.96612659     3.33031608    -0.28899847
 C                     1.96136004     2.35654679    -0.29525918
 N                     1.76104914     1.00039198    -0.31206960
 C                     3.01421343     0.42929712    -0.23521765
 C                     4.02563850     1.46345288    -0.19640032
 C                     3.37605648     2.65695569    -0.23517723
 Fe                   -0.06215314     0.06114262    -0.34777075
 S                    -0.08783199    -0.07132045     1.94350118
 C                     0.03880520     1.60548511     2.66619446
 C                     3.29661517    -0.93523022    -0.16698766
 C                     2.27956424    -1.91587252    -0.18863847
 N                     0.91501850    -1.70975770    -0.29360814
 C                     0.31239074    -2.94511637    -0.22200728
 C                     1.32835229    -3.96998988    -0.09661871
 C                     2.53390641    -3.33803953    -0.07738927
 C                    -1.06943901    -3.20815061    -0.24027885
 H                     4.32763725    -1.26001754    -0.08622430
 H                     1.12776273    -5.03219751    -0.02706819
 H                     5.09124752     1.27771733    -0.14045103
 H                     3.79516289     3.65547020    -0.21660655
 H                     1.28680157     4.36707453    -0.24805465
 H                    -1.23748736     5.18528484    -0.21728631
 H                    -3.62905082     3.94727847    -0.38989322
 H                    -4.43251381     1.40415345    -0.60321816
 H                    -5.19459805    -1.13963899    -0.58757060
 H                    -3.91602916    -3.51268692    -0.38476153
 H                    -1.38090029    -4.24615634    -0.18399558
 H                     3.51906181    -3.77996159     0.01034955
 H                     0.03011143     1.49274866     3.75568550
 H                     0.97150940     2.09821362     2.37530399
 H                    -0.80232522     2.24157464     2.37456831
 O                    -0.13370537    -0.07547497    -2.29722943
 C                    -2.62791541    -0.75800577    -4.55346316
 C                    -3.27850339     0.57341869    -4.32153024
 O                    -2.57553401     1.55451272    -3.88680579
 O                    -4.50001955     0.68508625    -4.61712346
 H                    -2.11322403    -0.72883836    -5.52262805
 H                    -3.38079880    -1.54673871    -4.58476177
 H                    -1.87517665    -0.96646837    -3.78492054
 N                    -5.82960322     2.76932443    -4.77491187
 C                    -5.39516066     4.01123413    -5.17312018
 C                    -6.48868822     4.79512904    -5.44191116
 N                    -7.59023544     3.99606843    -5.19137545
 C                    -7.15587179     2.78306122    -4.79876330
 C                    -6.55138622     6.21336318    -5.90979152
 H                    -7.05292302     6.86430989    -5.18302896
 H                    -5.53789700     6.59508129    -6.05593078
 H                    -7.08689056     6.30304463    -6.86254586
 H                    -8.57714003     4.27840823    -5.29310307
 H                    -7.79727665     1.95152453    -4.55076624
 H                    -5.14630186     1.84290928    -4.60452876
 H                    -4.34254451     4.24233399    -5.23347655
 O                    -0.33248569     1.22934597    -2.99542758
 H                    -0.41818518     1.81839853    -2.21030302
 H                    -1.50545023     1.29045642    -3.48991824
 bqH                  -0.11986114     6.35784586    -3.69020299
 bqH                  -1.22429083     5.82823172    -4.79548340
 bqH                  -0.21443627     5.72879747    -4.46131480
end
#
geometry adjust noautosym nocenter
ZCOORD
# O-O scission
bond 55 76  
# His:N-H
bond 63 74   
# Glu:O2-H
bond 58 78 
# distal O-H
bond 76 78 
# Fe-O
bond 30 55
#System Orientation Constraints
bond 18 79 constant
bond 24 79 constant
bond 33 79 constant
bond 39 79 constant
bond 56 79 constant
bond 68 79 constant
angle 18 79 80 constant
angle 24 79 80 constant
angle 33 79 80 constant
angle 39 79 80 constant
angle 56 79 80 constant
angle 68 79 80 constant
torsion 18 79 80 81 constant
torsion 24 79 80 81 constant
torsion 33 79 80 81 constant
torsion 39 79 80 81 constant
torsion 56 79 80 81 constant
torsion 68 79 80 81 constant
angle 68 65 80 constant
#fix ghost atoms as well
bond 56 80 constant
bond 56 81 constant
bond 68 80 constant
bond 68 81 constant
bond 80 18 constant
bond 81 18 constant
angle 81 18 24 constant
angle 80 18 24 constant
angle 81 18 24 constant
torsion 79 18 24 33 constant
torsion 80 18 24 33 constant
torsion 81 18 24 33 constant
#
bond 80 79 constant
bond 81 79 constant
bond 80 81 constant
#
#fix orientation of ASP
angle 2 66 67 constant
angle 3 66 67 constant
angle 5 66 67 constant
torsion 5 2 66 67 constant
torsion 3 2 66 67 constant
end
end
#
BASIS spherical
* library 6-31G* except Fe
bqH library H 6-31G
Fe library LANL2DZ_ecp
END
ECP
Fe library LANL2DZ_ecp
END
########################################
DFT
CONVERGENCE energy   1e-7
CONVERGENCE density  1e-6
CONVERGENCE gradient 5e-6
vectors output 11.mos
# direct
semidirect memsize 100000000 filesize 0
iterations 500
xc b3lyp
odft
mult 2
#  grid lebedev 160 18 H 90 14 C 100 17 N 100 17 O 100 17 S 125 18 Fe 160 18
 grid lebedev 125 18 H 75 11 C 90 14 N 90 14 O 90 14 S 90 14 Fe 125 18
# grid lebedev 90 14 H 75 11 C 75 11
grid  ssf  euler
END
driver
#GMAX   0.0015
#GRMS   0.0010
#XMAX   0.0024
#XRMS   0.0018
 trust 0.015
 default; maxiter 1500; eprec 1e-7; xyz ./XYZ/react
END
title "react search"
task dft optimize
task dft property

In case it's relevant, here is the script I used to compile NWChem:

#!/bin/bash

#---For details, visit the following pages:
#---http://nwchemgit.github.io/index.php/Compiling_NWChem
#---http://www.hpcadvisorycouncil.com/pdf/NWChem_Best_Practice.pdf

. $MODULESHOME/../global/profile.modules
module load make
module load gcc/4.6.2
module load intel/13.0.1

export NWCHEM_TOP=`pwd`
export LARGE_FILES=TRUE
export NWCHEM_TARGET=LINUX64
export NWCHEM_MODULES=all
export ENABLE_COMPONENT=yes
export TCGRSH=/usr/bin/ssh

export USE_MPI=y
export USE_MPIF=y
export USE_MPIF4=y
export MPI_HOME=$I_MPI_ROOT
export MPI_LOC=$I_MPI_ROOT
export LIBMPI="-lmpigf -lmpigi -lmpi_ilp64 -lmpi"
export MPI_LIB=$I_MPI_ROOT/lib64
export MPI_INCLUDE=$I_MPI_ROOT/include64

export MKL_INCLUDE=$MKLROOT/include/intel64/ilp64

#---If the following is removed, the code compiles and runs. Otherwise,
#---the code compiles but fails to run with ARMCI errors
export USE_SCALAPACK=y
export SCALAPACK="-L$MKLROOT/lib/intel64 -lmkl_scalapack_ilp64 -lmkl_intel_ilp64 -lmkl_sequential -lmkl_core -lmkl_blacs_intelmpi_ilp64 -lpthread
 -lm"
#--------------------------------------------------------------------------

export BLASOPT="-L$MKLROOT/lib/intel64 -L$MKLROOT/lib/intel64/libmkl_blas95_ilp64.a -lmkl_intel_ilp64 -lmkl_sequential -lmkl_core -lpthread -lm"
export FOPTIMIZE="-O3 -xSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 -no-prec-div -funroll-loops -unroll-aggressive"

cd $NWCHEM_TOP/src
make realclean

cp util/*.fh include/

make nwchem_config
make FC=ifort

Any ideas? Sorry for the long post.

Just Got Here

2:09:33 PM PDT - Wed, Mar 13th 2013

Update. OK, now I'm confused. The build of NWChem compiled with MKL-BLAS only now doesn't work, with a similar (albeit shorter) error. Nothing has changed! It worked a few hours ago! Here is the error:

 ------------------------------------------------------------------------
 ssread: end of file reading from unit        67
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
  current input line :
   185: task dft optimize
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
 An error occured while trying to read or write to disk space
 ------------------------------------------------------------------------
 For more information see the NWChem manual at
 http://nwchemgit.github.io/index.php/NWChem_Documentation


 For further details see manual section:

0:0:ssread: end of file reading from unit:: 67
(rank:0 hostname:n010 pid:36555):ARMCI DASSERT fail. ../../ga-5-1/armci/src/common/armci.c:ARMCI_Error():208 cond:0
rank 0 in job 1  n010_41300   caused collective abort of all ranks
  exit status of rank 0: killed by signal 9
Job  /home/share/platform/lsf/8.3/linux2.6-glibc2.3-x86_64/bin/intelmpi_wrapper nwchem react2.nw

Further info: I'm using Intel compilers 2013. I've already applied a couple of patches that I found at this forum.

Forum Vet

9:02:41 AM PDT - Thu, Mar 14th 2013
Your zeros feel like an integer-32 vs integer-64 issue, but I'll have one of the other developers comment on that more. On the disk, wonder if your scratch area is full and has issues reading and writing. Bert

Just Got Here

10:45:12 AM PDT - Thu, Mar 14th 2013
Thanks for the response Bert. I noticed that all of the nwchem-related subdirectories in my working directory were at size 8912, which looks like there's some kind of limit in place. I'm not sure where this limit is set, but the solution has been to change RESTART to START in my *.nw file. This has solved the problem with MKL - BLAS only, but the problem with MKL - BLAS + ScaLAPACK remains. As for the 32 vs. 64 bit, issue, I've been careful to use the ILP versions of the libraries, using the MKL link line advisor to guide me. My understanding is that if you use ILP64, you don't need to worry about the "make 64 to 32" thing. I was a little confused about USE_MPIF4: My code won't kompile without it, but it seems to be related to the use of 32-bit integers.

Forum Vet

6:04:10 PM PDT - Thu, Mar 14th 2013
11bolts, 1) USE_MPIF4 has no bearing to your DFT run (it affects only the planewave code) 2) You are right when say that the Intel ILP64 libraries don't need the 64to32 trick. I have just compiled NWChem with the Intel 2013 composer and I have used settings pretty similar to yours (the only differences are that I am using MPICH2 instead of Intel MPI and that I have not defined BLAS95 as you did). The binary I generated had no problem running the input file you provided. The relevant env. variables are the following (using NWChem 6.1 or later versions). `export BLAS_LIB="-L$MKLROOT/lib/intel64 -lmkl_intel_ilp64 -lmkl_sequential -lmkl_core -lpthread -lm" export BLASOPT="$BLAS_LIB" export BLAS_SIZE=8 export SCALAPACK_SIZE=8 export SCALAPACK="-L$MKLROOT/lib/intel64 -lmkl_scalapack_ilp64 -lmkl_intel_ilp64 -lmkl_sequential -lmkl_core\ -lmkl_blacs_intelmpi_ilp64 -lpthread -lm" export SCALAPACK_LIB="$SCALAPACK" export USE_SCALAPACK=y` 3) Big fat warning for compiling NWChem with the Intel Composer XE 2013: We have already found a few parts of NWChem that required modifications to get correct behaviors when Intel 2013 was used. We suggest you to either download the development version of NWChem (with the related caveats) or apply the following patches to 6.1.1 http://nwchemgit.github.io/images/Texasmem2.patch.gz http://nwchemgit.github.io/images/Giaxyz.patch.gz As a result of these experiences with Intel 2013, I strongly discourage from redefining the value of FOPTIMIZE as you did, and use the default value instead. Cheers, Edo

Just Got Here

10:25:25 AM PDT - Fri, Mar 15th 2013
Thanks Edo, it's working now. It turns out I didn't have the Texasmem2 patch. I had Giaxyz and another patch called DStebz3. I started with a fresh download, applied only the patches you suggested, and modified my build script to use your MKL variable lines. I also removed the fortran optimization line, although I note that in my previous compilation attempts, the presence of the optimization flags didn't affect the problem. In fact, I got these flags from the Intel Cluster Ready guide to building NWChem. Thanks again for your help.

Forum Vet

4:15:44 PM PDT - Fri, Mar 15th 2013
Quote:11bolts Mar 15th 9:25 am Thanks Edo, it's working now. It turns out I didn't have the Texasmem2 patch. I had Giaxyz and another patch called DStebz3. I started with a fresh download, applied only the patches you suggested, and modified my build script to use your MKL variable lines. I also removed the fortran optimization line, although I note that in my previous compilation attempts, the presence of the optimization flags didn't affect the problem. In fact, I got these flags from the Intel Cluster Ready guide to building NWChem. Thanks again for your help. Thanks for the feedback. By the way, the guide you mention was redacted in 2010, a) well before Intel 2013 go released and b) I am not quite sure if those compiler options were ever validated by a rigorous QA testing process. Thanks for reminding me of the dstebz3 patch that I forgot to mention in the earlier posting. To summarize, the patches needed for NWChem 6.1.1 to work nicely with Intel Fortran Composer XE 2013 are http://nwchemgit.github.io/images/Dstebz3.patch.gz http://nwchemgit.github.io/images/Texasmem2.patch.gz http://nwchemgit.github.io/images/Giaxyz.patch.gz

Forum >> NWChem's corner >> Running NWChem