p4_error: semget failed for setnum
The cause of the p4_error: semget failed for setnum error that causes compute jobs to die.
This error means that there is not enough shared memory available to allocate a new memory segment for interprocess communication. Often what happens is there are some extra memory segments left over from a crash or programming error of a previous job that needs to be cleaned up. There is a script called cleanipcs that will remove all of your left over ipcs. Users are responsible for cleaning up extra shared memory segments after a crash or when their job is complete.
Use the following command to clean up all memory segments owned by your user id on a cluster: $ cluster-fork /opt/mpich/gnu/sbin/cleanipcs
If you have problems with segments owned by another user even though they have no jobs running on that node please contact technical support to have the node cleaned up.