Quick Links

1. Compilers and Communication Libraries
2. Dsg cluster: Compiling code & launching jobs
3. Ds cluster: Available Compilers
4. Ds cluster: Launching jobs
5. Jpsi cluster: Available Compilers
6. Jpsi cluster: Launching jobs
7. kaon cluster: Available Compilers
8. kaon Cluster: Launching jobs
9. Compiling an MPI application

1. Compilers and Communication Libraries

The tables below list all the compilers installed on the various Fermilab LQCD clusters. Unix man page documentation is available for individual compilers. The Portland Group compilers are not binary compatible with GNU meaning that they will not accept object files or libraries created by the GNU compilers. Compiler drivers, such as mpicc, are provided for use when compiling programs containing MPI message passing directives.

2. Compiling code & launching jobs on the Dsg cluster (cluster head-node: ds1.fnal.gov)

Each dsg cluster worker node has a pair of quad-core Intel processors (8 cores total) and a pair of Nvidia M2050 GPUs. Each GPU has 3GB on-board memory but since ECC is enabled on all GPUs, only 2.6GB is available.

As with all of the Fermilab clusters your jobs on the Dsg cluster will get exclusive access to the entire dsg worker node and so your jobs will be billed two GPU hours for each dsg node hour.

CUDA 4.0.17 is installed at /usr/local/cuda-4.0.17 and /usr/local/cuda is a symbolic link to this directory. At runtime, you will need to set the environment variable

LD_LIBRARY_PATH=/usr/local/cuda/lib64

Our standard MPI's are MVAPICH, MVAPICH2 and OPENMPI, available at /usr/local/{mvapich,mvapich2,openmpi} respectively. These are all symbolic links; use ls -l to see the full path, which will include the version number. OPENMPI currently is 1.4.2 and if you have difficulties on the GPU nodes please try /usr/local/openmpi-1.4.4. .

The MPI libraries listed above were built for the Ds cluster worker nodes (32 cores per node) and so they will automatically launch 32 MPI ranks per GPU node when run on the dsg cluster. This is almost certainly not what you want and you can control MPI process launch as explained below.

For MVAPICH and MVAPICH2 for 2 ranks per GPU node, use

mpiexec -np xx -npernode 2 ... 

For OPENMPI, use

mpiexec -np xx -npernode 2 --byslot 

Note: MVAPICH and MVAPICH2 use the OSU mpiexec. mpirun is a symbolic link to mpiexec, so you can use either mpiexec or mpirun.

For example if you want to launch two MPI processes per dsg node which each process using half of the cores (4 out of 8) or one CPU socket and 1 GPU, you will execute mpirun as follows:

   source /usr/local/mvapich/etc/mvapich.conf.sh
   mpirun -np 4 /usr/local/mvapich/bin/numa_2_gpu binary args &
   mpirun -np 4 /usr/local/mvapich/bin/numa_2_gpu binary args &

The numa_2_gpu wrapper script will distribute the MPI ranks evenly across the two sockets such that all MPI processes that communicate with a particular GPU reside on the same CPU socket. The numa_2_gpu wrapper script can be omitted if you wish to control MPI process affinity (process to cpu core bindings) yourself.

Note that all of these different MPI versions obtain hostnames from Torque (PBS) directly. You do no need to worry about $PBS_NODEFILE contents. Torque is configured without a ppn setting for our nodes, so ppn switches to qsub won't function.

3. Compilers Available on the Ds cluster (cluster head-node: ds1.fnal.gov)

4. Launching jobs on the Ds cluster

      source /usr/local/mvapich/etc/mvapich.conf.sh

      source /usr/local/mvapich/etc/mvapich.conf.csh

      mpirun -np xxx /usr/local/mvapich/bin/numa_32_mv your.binary 

5. Compilers Available on the jpsi cluster (cluster head-node: jpsi1.fnal.gov)

[user@]$> . /usr/local/etc/setups.sh       or       source /usr/local/etc/setups.csh
[user@]$> setup pgi

6. Launching jobs on the jpsi cluster

•   mpirun_rsh_4 - This will launch 4 processes on each node,  with two processes assigned to each processor socket.
•   mpirun_rsh_2 - This will launch 2 processes on each node,  with each process assigned to a different processor socket.
•   mpirun_rsh_1 - This will launch 1 process on each node.

7. Compilers Available on the kaon cluster (cluster head-node: kaon1.fnal.gov)

(**) Use the following:

[user@]$> . /usr/local/etc/setups.sh       or       source /usr/local/etc/setups.csh
[user@]$> setup intel v10_1_008

to add the Intel compiler and libraries to your environment.

(+) Use the following:

[user@]$> . /usr/local/etc/setups.sh       or       source /usr/local/etc/setups.csh
[user@]$> setup pgi

to add the Portland Group compiler and libraries to your environment and to access the license manager. If you'd prefer not to use "setup", be sure to set LM_LICENSE_FILE=/fnal/ups/prd/pgi/v7.1-2/license.dat.

The binaries that are generated using the PGI compilers are not GNU compatible. Applications compiled with the Portland Group and Intel compilers must be statically linked with their respective object libraries since the dynamically linkable libraries are not available from cluster worker nodes.

8. Launching jobs on the kaon cluster

•   mpirun_rsh_2 - This will launch 2 processes on each node, with each process assigned to a different processor socket.


•   mpirun_rsh_1 - This will launch 1 process on each node.

9. Compiling an MPI application

Two MPI versions are available on the Infiniband clusters - MVAPICH, which uses the Ohio State University MPI and Open MPI. We recommend using mvapich. 

To build codes, use

$ /usr/local/mvapich/bin/mpicc     # or mpiCC, mpif77, etc...  (Infiniband)
  or
$ /usr/local/openmpi/bin/mpicc     # or mpiCC, mpif77, etc...  (Infiniband)

One of the MPICH versions as compiled with the Portland Group or Intel compiler.

.bashrc	.cshrc
export PATH=$PATH:/usr/local/mvapich/bin	setenv PATH $PATH:/usr/local/mvapich/bin

Here is how to compile example program cpi.c which comes with the mvapich software distribution.

$ cp /usr/local/mvapich/example/cpi.c .      # copy source to your directory
$ mpicc cpi.c -o cpi                         # compile