CDFT Jobs Failing After Running for Several Days


Click here for full thread
Just Got Here
Hi, I am relatively new to NWCHEM and I'm having trouble getting any of my CDFT geometry optimizations to converge. I have had no problems with regular DFT optimizations, only CDFT.

I have a large molecule (128 atoms, 639 electrons). I optimized the geometry with regular DFT with BP86/LANL2DZ/3-21G, and used this output as the input to my CDFT job (below) also using BP86/LANL2DZ/3-21G. Nothing I have tried seems to work, I always get errors in my output. This job ran for around 9 days before failing. I also included the end of the output file below. If anyone has any suggestions, I would appreciate it.

Thanks,
Matt


Input:
Title "203"

Start  203

memory total 45056 mb

echo

charge -1

geometry noautosym units angstrom
.
.
.
.
.
end

basis "ao basis" spherical print
  H library "3-21G"
  O library "3-21G"
  C library "3-21G"
  N library "3-21G"
  Ru library "LANL2DZ ECP"
END
ECP
  Ru library "LANL2DZ ECP"
END

dft
  iterations 10000
  mult 2
  XC slater 1.0 becke88 nonlocal 1.0 perdew86 1.0 vwn_1_rpa 1.0
  mulliken
  convergence nolevelshifting
  cdft 1 59 charge -1
end

set dft:cdft_maxiter 10000

driver
  maxiter 10000
  TRUST 0.05
end

task dft optimize







End of output:

 WARNING: error on integrated density =  0.21D-05
 greater than required accuracy of  0.10D-05
 CDFT multipliers:
      1       -0.0176462392
      iter =    9
 d= 0,ls=0.0,diis     3  -4436.1372839970  1.72D-02  4.03D-04  3.14D-02645674.0
                                                     2.39D-04  3.49D-02
 WARNING: error on integrated density =  0.21D-05
 greater than required accuracy of  0.10D-05
 CDFT multipliers:
      1       -0.0165035332
      iter =   10
 d= 0,ls=0.0,diis     4  -4436.1512091993 -1.39D-02  1.09D-04  7.59D-03645757.9
                                                     9.84D-05  8.09D-03
 WARNING: error on integrated density =  0.21D-05
 greater than required accuracy of  0.10D-05
 CDFT multipliers:
      1       -0.0133805791
      iter =    8
 d= 0,ls=0.0,diis     5  -4436.1545721177 -3.36D-03  6.44D-05  1.18D-03645837.7
                                                     3.83D-05  1.11D-03
 WARNING: error on integrated density =  0.21D-05
 greater than required accuracy of  0.10D-05
 CDFT multipliers:
      1       -0.0151945292
      iter =    6
 d= 0,ls=0.0,diis     6  -4436.1551364744 -5.64D-04  2.75D-05  1.12D-04645917.8
                                                     1.37D-05  1.11D-04
datum_fread: failed to read data 36
 ------------------------------------------------------------------------
 grid_acc_def: rtdb_put failed      911
 ------------------------------------------------------------------------
 grid_acc_def: rtdb_put failed      911
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
  current input line : 
 ------------------------------------------------------------------------
  current input line : 
   169: task dft optimize
 ------------------------------------------------------------------------
 grid_acc_def: rtdb_put failed      911
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
  current input line : 
 grid_acc_def: rtdb_put failed      911
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
 grid_acc_def: rtdb_put failed      911
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
  current input line : 
 grid_acc_def: rtdb_put failed      911
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
 grid_acc_def: rtdb_put failed      911
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
  current input line : 
 grid_acc_def: rtdb_put failed      911
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
 An error occured in the Runtime Database
 ------------------------------------------------------------------------
 grid_acc_def: rtdb_put failed      911
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
  current input line : 
  current input line : 
 grid_acc_def: rtdb_put failed      911
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
  current input line : 
  current input line : 
 grid_acc_def: rtdb_put failed      911
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
  current input line : 
  current input line : 
 For more information see the NWChem manual at 
 http://nwchemgit.github.io/index.php/NWChem_Documentation


 For further details see manual section: 
                                                                                
                                                                                
                                                                                
                   
 grid_acc_def: rtdb_put failed      911
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
  current input line : 
     0: 
     0: 
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
 An error occured in the Runtime Database
 ------------------------------------------------------------------------
 For more information see the NWChem manual at 
     0: 
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
 An error occured in the Runtime Database
 ------------------------------------------------------------------------
 For more information see the NWChem manual at 
 http://nwchemgit.github.io/index.php/NWChem_Documentation


0:0:grid_acc_def: rtdb_put failed:: 911
(rank:0 hostname:compute-0-0.local pid:25908):ARMCI DASSERT fail. armci.c:ARMCI_Error():260 cond:0
     0: 
 ------------------------------------------------------------------------
     0: 
 ------------------------------------------------------------------------
     0: 
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
 An error occured in the Runtime Database
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
 An error occured in the Runtime Database
 ------------------------------------------------------------------------
 For more information see the NWChem manual at 
     0: 
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
 An error occured in the Runtime Database
 ------------------------------------------------------------------------
 For more information see the NWChem manual at 
 http://nwchemgit.github.io/index.php/NWChem_Documentation


 For further details see manual section: 
                                                                                
                                                                                
                                                                                
                   
     0: 
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
 An error occured in the Runtime Database
 ------------------------------------------------------------------------
 For more information see the NWChem manual at 
     0: 
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
 An error occured in the Runtime Database
 ------------------------------------------------------------------------
 For more information see the NWChem manual at 
 http://nwchemgit.github.io/index.php/NWChem_Documentation

     0: 
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
 An error occured in the Runtime Database
 ------------------------------------------------------------------------
 For more information see the NWChem manual at 
 http://nwchemgit.github.io/index.php/NWChem_Documentation


 For further details see manual section: 
                                                                                
                                                                                
                                                                                
                   
Last System Error Message from Task 0:: Stale NFS file handle
Last System Error Message from Task 1:: Numerical result out of range
Last System Error Message from Task 3:: Numerical result out of range
Last System Error Message from Task 4:: Numerical result out of range
Last System Error Message from Task 7:: Numerical result out of range
Last System Error Message from Task 8:: Numerical result out of range
Last System Error Message from Task 9:: Numerical result out of range
Last System Error Message from Task 10:: Numerical result out of range
Last System Error Message from Task 11:: Numerical result out of range
 http://nwchemgit.github.io/index.php/NWChem_Documentation


 For further details see manual section: 
                                                                                
                                                                                
                                                                                
                   
4:4:grid_acc_def: rtdb_put failed:: 911
(rank:4 hostname:compute-0-0.local pid:25912):ARMCI DASSERT fail. armci.c:ARMCI_Error():260 cond:0
 For further details see manual section: 
                                                                                
                                                                                
                                                                                
                   
9:9:grid_acc_def: rtdb_put failed:: 911
(rank:9 hostname:compute-0-0.local pid:25917):ARMCI DASSERT fail. armci.c:ARMCI_Error():260 cond:0
 ------------------------------------------------------------------------
 An error occured in the Runtime Database
 ------------------------------------------------------------------------
 For more information see the NWChem manual at 
 http://nwchemgit.github.io/index.php/NWChem_Documentation


 For further details see manual section: 
                                                                                
                                                                                
                                                                                
                   
10:10:grid_acc_def: rtdb_put failed:: 911
(rank:10 hostname:compute-0-0.local pid:25918):ARMCI DASSERT fail. armci.c:ARMCI_Error():260 cond:0
 ------------------------------------------------------------------------
 An error occured in the Runtime Database
 ------------------------------------------------------------------------
 For more information see the NWChem manual at 
 http://nwchemgit.github.io/index.php/NWChem_Documentation


 For further details see manual section: 
                                                                                
                                                                                
                                                                                
                   
1:1:grid_acc_def: rtdb_put failed:: 911
(rank:1 hostname:compute-0-0.local pid:25909):ARMCI DASSERT fail. armci.c:ARMCI_Error():260 cond:0
     0: 
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
 For more information see the NWChem manual at 
 http://nwchemgit.github.io/index.php/NWChem_Documentation


 For further details see manual section: 
                                                                                
                                                                                
                                                                                
                   
3:3:grid_acc_def: rtdb_put failed:: 911
(rank:3 hostname:compute-0-0.local pid:25911):ARMCI DASSERT fail. armci.c:ARMCI_Error():260 cond:0
 http://nwchemgit.github.io/index.php/NWChem_Documentation


 For further details see manual section: 
                                                                                
                                                                                
                                                                                
                   
5:5:grid_acc_def: rtdb_put failed:: 911
(rank:5 hostname:compute-0-0.local pid:25913):ARMCI DASSERT fail. armci.c:ARMCI_Error():260 cond:0
7:7:grid_acc_def: rtdb_put failed:: 911
(rank:7 hostname:compute-0-0.local pid:25915):ARMCI DASSERT fail. armci.c:ARMCI_Error():260 cond:0
 http://nwchemgit.github.io/index.php/NWChem_Documentation


 For further details see manual section: 
                                                                                
                                                                                
                                                                                
                   
6:6:grid_acc_def: rtdb_put failed:: 911
(rank:6 hostname:compute-0-0.local pid:25914):ARMCI DASSERT fail. armci.c:ARMCI_Error():260 cond:0

 For further details see manual section: 
                                                                                
                                                                                
                                                                                
                   
8:8:grid_acc_def: rtdb_put failed:: 911
(rank:8 hostname:compute-0-0.local pid:25916):ARMCI DASSERT fail. armci.c:ARMCI_Error():260 cond:0
11:11:grid_acc_def: rtdb_put failed:: 911
(rank:11 hostname:compute-0-0.local pid:25919):ARMCI DASSERT fail. armci.c:ARMCI_Error():260 cond:0
Last System Error Message from Task 5:: Numerical result out of range
Last System Error Message from Task 6:: Numerical result out of range
 An error occured in the Runtime Database
 ------------------------------------------------------------------------
 For more information see the NWChem manual at 
 http://nwchemgit.github.io/index.php/NWChem_Documentation


 For further details see manual section: 
                                                                                
                                                                                
                                                                                
                   
2:2:grid_acc_def: rtdb_put failed:: 911
(rank:2 hostname:compute-0-0.local pid:25910):ARMCI DASSERT fail. armci.c:ARMCI_Error():260 cond:0
Last System Error Message from Task 2:: Numerical result out of range
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 2 in communicator MPI_COMM_WORLD 
with errorcode 0.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun has exited due to process rank 3 with PID 25911 on
node compute-0-0.local exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------
[compute-0-0.local:25907] 11 more processes have sent help message help-mpi-api.txt / mpi-abort
[compute-0-0.local:25907] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages