The error message is not an indication of lack of memory, but it might be due either to an incorrect NWChem installation or to a bug in the code.
Have you tried a smaller molecule with a similar input file? If this case works, this would imply that installation problems are not likely.