Lukasz,
Since when you compiled with NWCHEM_TARGET=IBM, you have generated a 32-bit executable that will not be able to address more than 2GB of memory. In order to overcome this limit, you would have to generate a 64-bit binary using NWCHEM_TARGET=IBM64