Jonas
This can be probably solved by setting an appropriate value of the ARMCI_DEFAULT_SHMMAX env. variable.
If you could post your input file and more details about your failed run (e.g. total number of processes used, characteristics of the HW used, etc ... ), we would be in a better position to help you.
Cheers, Edo