USQCD: US Lattice Quantum Chromodynamics

The tables below list all the compilers installed on the various Fermilab LQCD clusters. Unix man page documentation is available for individual compilers. The Portland Group compilers are not binary compatible with GNU meaning that they will not accept object files or libraries created by the GNU compilers. Compiler drivers, such as mpicc, are provided for use when compiling programs containing MPI message passing directives.

2. Compilers Available on the D/s cluster (cluster head-node: ds1.fnal.gov)

(*) If you get the "error while loading shared libraries" then use the following:

Since the GCC 4.5.1 libraries conflict with the deafult GCC 4.1.2, users are required to do the following to point to the correct shared libraries when invoking the 4.5.1 GCC compiler command:

$> export PATH=/usr/local/gcc-4.5.1/bin:$PATH
$> export LD_LIBRARY_PATH=/usr/local/gcc-4.5.1/lib:/usr/local/gcc-4.5.1/lib64:$LD_LIBRARY_PATH

You may also add these lines to your .bashrc file for permanence.

3. Launching jobs on the D/s cluster

The following job launch procedure applies when using MVAPICH. D/s nodes are eight-core, quad processor, so each node will run up to 32 processes at one process per core.

Use /usr/local/mvapich/bin/mpirun as follows to launch jobs:

mpirun - will automatically launch 32 processes on each D/s node.

On D/s, the nodes listed in $PBS_NODEFILE will each be used 32 times, and will be assigned 32 consecutive MPI rank numbers. For example:

qsub -q ds -l nodes=4 my.script # request four nodes
# then, in the run script:
mpirun_rsh -np 128 milc.binary input.file > output.file

Here, $PBS_NODEFILE will have 4 D/s nodes, eg, ds0101 - ds0104, and on each of these nodes 32 instances of "milc.binary" will be run.

mpirun is actually mpiexec from OSC (see http://www.osc.edu/~djohnson/mpiexec/index.php) or use mpirun --help to see additional information.

The processors on D/s have non-uniform paths to system memory. Your jobs may run with higher performance if you explicitly bind processes to
NUMA nodes, and set a local memory allocation policy. To do this, at the beginning of your job script, do

      source /usr/local/mvapich/etc/mvapich.conf.sh

      source /usr/local/mvapich/etc/mvapich.conf.csh

and on your mpirun commands, invoke your binary via the wrapper
numa_32_mv:

      mpirun -np xxx /usr/local/mvapich/bin/numa_32_mv your.binary

4. Compilers Available on the jpsi cluster (cluster head-node: jpsi1.fnal.gov)

to add the Portland Group compiler and libraries to your environment and to access the license manager. If you'd prefer not to use "setup", be sure to set LM_LICENSE_FILE=/usr/local/pgi-904/linux86-64/9.0-4/license.dat

The binaries that are generated using the PGI compilers are not GNU compatible. Applications compiled with the Portland Group compiler must be statically linked with their respective object libraries since the dynamically linkable libraries are not available from cluster worker nodes.

5. Launching jobs on the jpsi cluster

The following job launch procedure applies when using MVAPICH. Jspi nodes are quad-core, dual processor, so each node will run up to eight processes.

Use /usr/local/mvapich/bin/mpirun_rsh* as follows to launch jobs:

mpirun_rsh - will automatically launch 8 processes on each jpsi node.

On Jpsi, the nodes listed in $PBS_NODEFILE will each be used 8 times, and will be assigned 8 consecutive MPI rank numbers. For example:

qsub -q jpsi -l nodes=4 my.script # request four nodes
# then, in the run script:
mpirun_rsh -np 32 milc.binary input.file > output.file

Here, $PBS_NODEFILE will have 4 jpsi nodes, eg, jpsi0101-jpsi0104, and on each of these nodes eight instances of "milc.binary" will be run. There are other versions of mpirun_rsh that can launch 1, 2 or 4 processes as follows:

6. Compilers Available on the kaon cluster (cluster head-node: kaon1.fnal.gov)

to add the Portland Group compiler and libraries to your environment and to access the license manager. If you'd prefer not to use "setup", be sure to set LM_LICENSE_FILE=/fnal/ups/prd/pgi/v7.1-2/license.dat.

The binaries that are generated using the PGI compilers are not GNU compatible. Applications compiled with the Portland Group and Intel compilers must be statically linked with their respective object libraries since the dynamically linkable libraries are not available from cluster worker nodes.

7. Launching jobs on the kaon cluster

The following job launch procedure applies when using MVAPICH. Kaon nodes are dual-core, dual processor, so each node will run up to four processes.

Use /usr/local/mvapich/bin/mpirun_rsh* as follows to launch jobs:

mpirun_rsh - will detect will automatically launch 4 processes on each Kaon node

On Kaon, the nodes listed in $PBS_NODEFILE will each be used 4 times, and will be assigned 4 consecutive MPI rank numbers. For example:

      qsub -q kaon -l nodes=4 my.script     # request four nodes
         # then, in the run script:
         mpirun_rsh -np 16 milc.binary input.file > output.file

   Here, $PBS_NODEFILE will have 4 kaon nodes, eg, kaon0101-kaon0104, and on each of these nodes four instances of "milc.binary" will be run.

8. Compiling an MPI application

Two MPI versions are available on the Infiniband clusters - MVAPICH, which uses the Ohio State University MPI and Open MPI. We recommend using mvapich.

To use a compiler, first check that its executable path is in your shell's PATH environment variable. Use the UNIX env command to see how PATH is currently set. If the desired compiler is not in your PATH, edit your .bashrc (for bash or sh users) or .cshrc (for tcsh or csh users) to add the compiler's path. For example, to add mpiCC, mpicc and mpif77 to your path, add the following line to your shell's start-up script:

.bashrc	.cshrc
export PATH=$PATH:/usr/local/mvapich/bin	setenv PATH $PATH:/usr/local/mvapich/bin

Here is how to compile example program cpi.c which comes with the mvapich software distribution.

FNAL - LQCD Documentation

Quick Links