| 1:53:36 AM PDT - Mon, Oct 27th 2014 |
|
I'm running NWChem 6.5 on a Ubuntu workstation with two 12-core processors with hyperthreading, effectively giving me the possibility to run 48 processes in parallel. I have 128 GB of memory, and a dedicated scratch disk of 360 GB. The scratch disk is a PCIe SSD, so it's fast, but obviously not as fast as RAM.
The default memory settings are as follows:
heap: 663.8 MB, stack: 663.8 MB, global: 1327.5 MB, total: 2655.1 MB
This was set using the getmem.nwchem script, and makes sense, because multiplied by 48 processes gives the total amount of memory of the system.
Now when I run a DFT calculation, I typically use 44 mpi processes. Most of the time all of them are running at 95+% CPU, so that gives me confidence that I'm using my processor power well. On the other hand, each process uses only 0.2% of memory, so that in total maybe only 10% of the memory is used. The scratch disk is used quite heavily however, mainly for the integral files. Process 0 writes a 50-100 GB .aoints file, while the other processes write 2.5-5.0 GB .aoints files. While these files are written, the processes slow down to about 30-60% CPU, showing that I/O is still a bottleneck at this time.
Now, after my somewhat lengthy introduction, to my questions: Why does NWChem decide that it is better to use scratch than to use RAM? Is it possible to make more use of RAM in these calculations, and would that actually help?
I am aware of the "semidirect" keyword for the HF and DFT modules, where an amount of scratch space or memory can be declared, but not sure how to use this. If I limit scratch space with this keyword, will more memory be used? Or will it move more towards "direct" calculation of the integrals? Is the memory declared by the "semidirect" keyword included in the memory declared by the "memory" keyword, or is it additional? If it is included, is it part of heap, stack, or global?
I am aware that NWChem was not really designed for workstations, so maybe I cannot expect optimal performance. However, I don't think that I am the only one running on a workstation, so hopefully this question is interesting for some other people as well.
|