Segmentation Violation error / ARMCI DASSERT fail

Just Got Here

8:56:59 AM PST - Fri, Feb 5th 2016

I am performing DFT calculations on clusters of organic molecules, in particular, I consider a dimer of two C60 molecules.
The goemetry optimization with the (small) 3-21G basis and xc B3LYP went smooth.

Now, I am running into trouble when I go for larger basis sets and/or other functionals.

This is what I do:

geometry

C      -2.90509337     4.77370520     1.84015723

C      -3.26656366     4.15340832     0.64946491

  ...

C      2.54161624    -6.53394777    -1.60786644

end

basis spherical

* library "cc-pVTZ"

end

dft

xc m06-2x

convergence energy 1e-08

end

task dft energy

The computation crashes after iteration 10. Unfortunately, the meaning of the error messages is not at all clear to me:

7:Segmentation Violation error, status=: 11

(rank:7 hostname:taurusi5428 pid:2262):ARMCI DASSERT fail. ../../ga-5-3/armci/src/common/signaltrap.c:SigSegvHandler():310 cond:0

I have no idea if there is something wrong with my input (memory specifications...) or if the problem is somewhere in the NWChem installation on my system.

I would be very thankful for any help!

Full NWChem job

Submission script

Output

Error log

Forum Vet

11:40:42 AM PST - Fri, Feb 5th 2016
please switch to 6.6 and apply Xccvs98.patch.gz
Your output shows that you have been using NWChem 6.5 We have been fixing bugs in the MO6 functional since the 6.5 release. Please install NWChem 6.6 and apply the recent Xccvs98 patch http://nwchemgit.github.io/download.php?f=Xccvs98.patch.gz http://nwchemgit.github.io/index.php/Download#NWChem_6.6_Source_Code

Just Got Here

11:06:34 AM PST - Tue, Feb 9th 2016

Thank you very much for your answer, Edoapra!

I have now run the calculation with the patched NWChem 6.6.

Unfortunately, I still have the same problems.

Along the calculation (and already before the first iteration) there are warnings of the type

74: WARNING:armci_set_mem_offset: offset changed 553546424320 to 553570721792

but the computation continues with meaningful output.
In run 2, I also got the error

slurmstepd: IPMI-RAW: ipmi_cmd_rawcommand unsuccessful => ipmi_cmd_raw BMC returned an error 255

at an early stage, which didn't kill the computation either.

After the 10th iteration, I get

9:Segmentation Violation error, status=: 11

(rank:9 hostname:taurusi6487 pid:22923):ARMCI DASSERT fail. ../../ga-5-4/armci/src/common/signaltrap.c:SigSegvHandler():315 cond:0

in my output (9 was 33 in the other run), and a bunch of errors in the error log. The first one is either

Last System Error Message from Task 9:: Numerical result out of range

Last System Error Message from Task 33:: Resource temporarily unavailable

Followed by an MPI_ABORT.

The full output and error logs are here:

Error log run 1
Output run 1
Error log run 2
Output run 2

Any idea what I should try next? These problems do not occur when I take only one C60 molecule instead of two. Should I try to specify memory limits manually?

Your help is much appreciated.

Forum Vet

4:31:51 PM PST - Tue, Feb 9th 2016
ARMCI_DEFAULT_SHMMAX
Have you set the value of ARMCI_DEFAULT_SHMMAX for your runs? More details at http://nwchemgit.github.io/index.php/FAQ#How_do_I_set_the_correct_value_for_ARMCI_DEFAULT_SH...

Just Got Here

2:27:21 AM PST - Mon, Feb 15th 2016
Thank you, Edoapra, including export ARMCI_DEFAULT_SHMMAX=2048 in the submission script did do the job!

Clicked A Few Times

6:40:05 AM PST - Mon, Feb 15th 2016
Edo, Could you specify what you mean by bugs in m06 in 6.5? I've been running 6.5 for a while now, using almost exclusively m06-2x. Should I worry about the results of those calculations?

Forum >> NWChem's corner >> Running NWChem