Running with OpenMPI on multiple nodes failing


Click here for full thread
Clicked A Few Times
Sure, thank you. This copied directly from the result out/err files:

1
mpirun (Open MPI) 1.6.5

Report bugs to http://www.open-mpi.org/community/help/

2
linux-vdso.so.1 => (0x00007fff2ebff000)
libmca_common_sm.so.3 => /usr/local/openmpi/openmpi-1.6.5/intel-14.0.1/lib/libmca_common_sm.so.3 (0x00007f9ad8085000)
libmpi_f77.so.1 => /usr/local/openmpi/openmpi-1.6.5/intel-14.0.1/lib/libmpi_f77.so.1 (0x00007f9ad7e4e000)
libmpi.so.1 => /usr/local/openmpi/openmpi-1.6.5/intel-14.0.1/lib/libmpi.so.1 (0x00007f9ad7a4e000)
libmpi_cxx.so.1 => /usr/local/openmpi/openmpi-1.6.5/intel-14.0.1/lib/libmpi_cxx.so.1 (0x00007f9ad7832000)
libmpi_f90.so.1 => /usr/local/openmpi/openmpi-1.6.5/intel-14.0.1/lib/libmpi_f90.so.1 (0x00007f9ad762f000)
libompitrace.so.0 => /usr/local/openmpi/openmpi-1.6.5/intel-14.0.1/lib/libompitrace.so.0 (0x00007f9ad742b000)
libopen-rte.so.4 => /usr/local/openmpi/openmpi-1.6.5/intel-14.0.1/lib/libopen-rte.so.4 (0x00007f9ad70f1000)
libotfaux.so.0 => /usr/local/openmpi/openmpi-1.6.5/intel-14.0.1/lib/libotfaux.so.0 (0x00007f9ad6ee4000)
libdl.so.2 => /lib64/libdl.so.2 (0x0000003fc6c00000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x0000003fd6c00000)
libutil.so.1 => /lib64/libutil.so.1 (0x0000003fd7c00000)
libibumad.so.3 => /usr/lib64/libibumad.so.3 (0x0000003976200000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003fc6800000)
libibverbs.so.1 => /usr/lib64/libibverbs.so.1 (0x0000003975e00000)
libm.so.6 => /lib64/libm.so.6 (0x0000003fc7000000)
libc.so.6 => /lib64/libc.so.6 (0x0000003fc6400000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003fcd000000)
librt.so.1 => /lib64/librt.so.1 (0x0000003fc7400000)
libimf.so => /zhome/Apps/intel/composer_xe_2013_sp1.1.106/compiler/lib/intel64/libimf.so (0x00007f9ad69fc000)
libsvml.so => /zhome/Apps/intel/composer_xe_2013_sp1.1.106/compiler/lib/intel64/libsvml.so (0x00007f9ad5e05000)
libirng.so => /zhome/Apps/intel/composer_xe_2013_sp1.1.106/compiler/lib/intel64/libirng.so (0x00007f9ad5bfe000)
libintlc.so.5 => /zhome/Apps/intel/composer_xe_2013_sp1.1.106/compiler/lib/intel64/libintlc.so.5 (0x00007f9ad59a7000)
libcilkrts.so.5 => /zhome/Apps/intel/composer_xe_2013_sp1.1.106/compiler/lib/intel64/libcilkrts.so.5 (0x00007f9ad5769000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x0000003fcd800000)
libifport.so.5 => /zhome/Apps/intel/composer_xe_2013_sp1.1.106/compiler/lib/intel64/libifport.so.5 (0x00007f9ad5539000)
libifcore.so.5 => /zhome/Apps/intel/composer_xe_2013_sp1.1.106/compiler/lib/intel64/libifcore.so.5 (0x00007f9ad51f8000)
libifcoremt.so.5 => /zhome/Apps/intel/composer_xe_2013_sp1.1.106/compiler/lib/intel64/libifcoremt.so.5 (0x00007f9ad4e8a000)
/lib64/ld-linux-x86-64.so.2 (0x0000003fc6000000)

3
hbar5
hbar5
hbar5
hbar5
hbar5
hbar5
hbar5
hbar5
hbar5
hbar5
hbar5
hbar5
hbar5
hbar5
hbar5
hbar5
hbar4
hbar4
hbar4
hbar4
hbar4
hbar4
hbar4
hbar4
hbar4
hbar4
hbar4
hbar4
hbar4
hbar4
hbar4
hbar4

4) Unfortunately I broke the previous install with a subsequent build with a new value of LIBMPI. So I am starting a new .... so I have no output.dat fie anymore.