Installing NWChem on an infinipath rocks cluster
How to install and use NWChem on an infinipath rocks cluster.
Introduction
NWchem is a computational chemistry package that is very communications intensive when used in parallel mode. Infinipath is a very low latency and high bandwidth interconnect that allows nwchem to scale well.To download NWChem:
Go to http://www.emsl.pnl.gov/docs/nwchem/download.html
To get the patch:
Get the patch from pathscale or CSE users can likely find it in /opt/src.
To compile:
# cd /share/apps/nwchem-4.7-mpi-path-2.5/
# export NWCHEM_TOP=`pwd`
# export NWCHEM_TARGET=LINUX64
# export NWCHEM_MODULES=all
# unzip PATHNAME_TO_PATCH/nwchem-4.7-infinipath-patch-1.gz
# patch -p 1 < PATHNAME_TO_PATCH/nwchem-4.7-infinipath-patch-1
# export USE_MPI=y
# export MPI_LIB=/usr/lib64
# export MPI_INCLUDE=/usr/include
# export LIBMPI="-lmpich"
# cd src
# make nwchem_config NWCHEM_MODULES=all
# make FC=pathf90 _FC=pathf90 CC=pathcc
To Run:
Rocks-4.2 defaults to Sun Grid Engine which is what we use on our Infinipath clusters. Here's an example SGE script
to launch a nwchem job on rocks:
#!/bin/bash
#
#$ -cwd
#$ -j y
#$ -S /bin/bash
/usr/bin/mpirun -np $NSLOTS -m $TMPDIR/machines /share/apps/nwchem-4.7-mpi-path-2.5/bin/LINUX64/nwchem ./sassolite.nw
Note that NWChem used in this way uses 1 CPU per node for ARMCI communications, this works out reasonably well on dual core dual socket compute nodes. The 3:1 ratio provides a good balance of server processing overhead and NWchem computations.