Quick Links
- Compilers and Communication
Libraries
- Dsg cluster: Compiling code & launching jobs
- Ds cluster: Available Compilers
- Ds cluster: Launching jobs
- Jpsi cluster: Available Compilers
- Jpsi cluster: Launching jobs
- kaon cluster: Available Compilers
- kaon Cluster: Launching jobs
- Compiling an MPI application
1. Compilers and Communication
Libraries
The tables below list all the compilers installed on the
various Fermilab LQCD clusters. Unix man page documentation is
available for individual compilers. The Portland Group compilers are
not binary compatible with
GNU meaning that they will not accept object files or libraries created
by the GNU compilers. Compiler drivers, such as mpicc, are provided for
use when compiling programs containing MPI message passing
directives.
2. Compiling code & launching jobs on the
Dsg cluster (cluster
head-node:
ds1.fnal.gov)
Each dsg cluster worker node has a pair of quad-core Intel processors (8 cores total) and a pair of Nvidia
M2050 GPUs. Each GPU has 3GB on-board memory but since ECC is enabled on all GPUs, only 2.6GB is available.
As with all of the Fermilab clusters your jobs on the Dsg cluster will get exclusive
access to the entire dsg worker node and so your jobs will be billed two GPU hours
for each dsg node hour.
CUDA 4.0.17 is installed at /usr/local/cuda-4.0.17 and /usr/local/cuda is a
symbolic link to this directory. At runtime, you will need to set the environment variable
LD_LIBRARY_PATH=/usr/local/cuda/lib64
Our standard MPI's are MVAPICH, MVAPICH2 and OPENMPI, available at
/usr/local/{mvapich,mvapich2,openmpi} respectively.
These are all symbolic links; use ls -l to see the full path, which will include the version number.
OPENMPI currently is 1.4.2 and if you have difficulties on the GPU nodes please try
/usr/local/openmpi-1.4.4. .
The MPI libraries listed above were built for the Ds cluster worker nodes (32 cores per node)
and so they will automatically launch 32 MPI ranks per GPU node when run on the dsg cluster. This is
almost certainly not what you want and you can control MPI process launch as explained below.
For MVAPICH and MVAPICH2 for 2 ranks per GPU node, use
mpiexec -np xx -npernode 2 ...
For OPENMPI, use
mpiexec -np xx -npernode 2 --byslot
Note: MVAPICH and MVAPICH2 use the OSU mpiexec. mpirun
is a symbolic link to mpiexec, so you can use either mpiexec or mpirun.
For example if you want to launch two MPI processes per dsg node which each process using half of the cores (4 out of 8) or one CPU socket and
1 GPU, you will execute mpirun as follows:
source /usr/local/mvapich/etc/mvapich.conf.sh
mpirun -np 4 /usr/local/mvapich/bin/numa_2_gpu binary args &
mpirun -np 4 /usr/local/mvapich/bin/numa_2_gpu binary args &
The numa_2_gpu wrapper script will distribute the MPI ranks evenly across the two sockets such that all MPI processes that communicate
with a particular GPU reside on the same CPU socket. The numa_2_gpu wrapper script can be omitted if you wish to control MPI process
affinity (process to cpu core bindings) yourself.
Note that all of these different MPI versions obtain hostnames from Torque (PBS) directly.
You do no need to worry about $PBS_NODEFILE contents.
Torque is
configured without a ppn setting for our nodes, so ppn switches to qsub won't
function.
3. Compilers Available on the Ds cluster (cluster
head-node:
ds1.fnal.gov)
| Compiler
Name
and
Version |
Binary Location |
Binary Name(s) |
|
Gnu
Compilers Version 3.4.6 |
/usr/local/gcc-3.4.6/bin |
g++, gcc, g77
|
| Gnu Compilers Version 4.1.2 |
/usr/bin |
g++, gcc, gfortran |
Gnu
Compilers Version 4.5.1 (*)
|
/usr/local/gcc-4.5.1/bin |
g++, gcc, gfortran |
Gnu
Compilers Version 4.6.2 (*)
|
/usr/local/gcc-4.6.2/bin |
g++, gcc, gfortran |
MVAPICH Version
1.2
|
/usr/local/mvapich-1.2rc1/bin
|
mpicc,
mpif77, mpif90 |
MVAPICH2
Version 1.5
|
/usr/local/mvapich2-1.5.1p1/bin
|
mpicc,
mpif77, mpif90 |
Open MPI Version 1.4.2
|
/usr/local/openmpi-1.4.2/bin
|
mpicc, mpif77, mpif90 |
(*) If you get the "error while
loading shared libraries" then use the following:
Since the GCC 4.5.1 libraries conflict with the deafult GCC 4.1.2,
users are required to do the following to point to the correct shared
libraries when invoking the 4.5.1 GCC compiler command:
$>
export
PATH=/usr/local/gcc-4.5.1/bin:$PATH
$>
export
LD_LIBRARY_PATH=/usr/local/gcc-4.5.1/lib:/usr/local/gcc-4.5.1/lib64:$LD_LIBRARY_PATH
You may also add these lines to your .bashrc
file for permanence. For gcc-4.6.2 substitute 4.6.2 for 4.5.1 in the above export commands.
4. Launching jobs on the Ds cluster
The following job launch
procedure applies when using MVAPICH.
Ds
nodes
are
eight-core, quad processor, so each node will run up to 32
processes at one process per core.
Use /usr/local/mvapich/bin/mpirun
as follows to launch jobs:
mpirun - will
automatically launch 32 processes on each Ds node.
On Ds, the nodes listed in $PBS_NODEFILE will each be used 32 times,
and will be assigned 32 consecutive MPI rank numbers. For
example:
qsub -q ds -l nodes=4 -A myproject
my.script # request four nodes
# then, in the run script:
mpirun_rsh -np 128
milc.binary input.file > output.file
Here, $PBS_NODEFILE will have 4 Ds nodes, eg, ds0101 - ds0104, and
on each of these nodes 32 instances of " milc.binary" will be run.
mpirun
is actually mpiexec
from OSC (see http://www.osc.edu/~djohnson/mpiexec/index.php)
or
use
mpirun --help to see additional information.
The processors on Ds have non-uniform paths to system memory.
Your jobs may run with higher performance if you explicitly bind
processes to
NUMA nodes, and set a local memory allocation policy. To do this,
at the beginning of your job script, do
source /usr/local/mvapich/etc/mvapich.conf.sh
or
source /usr/local/mvapich/etc/mvapich.conf.csh
and on your mpirun commands, invoke your binary via the wrapper
numa_32_mv:
mpirun -np xxx /usr/local/mvapich/bin/numa_32_mv your.binary
5. Compilers Available on the jpsi cluster (cluster
head-node:
jpsi1.fnal.gov)
| Compiler
Name
and
Version |
Binary Location |
Binary Name(s) |
| Gnu Compilers Version 3.4.6 |
/usr/local/gcc-3.4.6/bin |
g++, gcc, g77 |
| Gnu Compilers Version 4.1.2 |
/usr/bin |
g++, gcc, g77 |
|
Gnu
Compilers Version 4.5.1(*) |
/usr/local/gcc-4.5.1/bin |
g++, gcc, g77 |
|
Gnu
Compilers Version 4.6.2(*) |
/usr/local/gcc-4.6.2/bin |
g++, gcc, g77 |
MVAPICH Version
1.1.0
|
/usr/local/mvapich-1.1.0-ofed-1.4.2/bin
|
mpicc,
mpif77, mpif90 |
MVAPICH2
Version 1.2
|
/usr/local/mvapich2-1.2-ofed-1.4.2/bin
|
mpicc,
mpif77, mpif90 |
Open MPI Version 1.3.4
|
/usr/local/openmpi-1.3.4/bin
|
mpicc, mpif77, mpif90 |
(*) If you get the "error while
loading shared libraries" then use the following:
Since the GCC 4.5.1 libraries conflict with the default GCC 4.1.2,
users are required to do the following to point to the correct shared
libraries when invoking the 4.5.1 GCC compiler command:
$>
export
PATH=/usr/local/gcc-4.5.1/bin:$PATH
$>
export
LD_LIBRARY_PATH=/usr/local/gcc-4.5.1/lib:/usr/local/gcc-4.5.1/lib64:$LD_LIBRARY_PATH
You may also add these lines to your .bashrc
file for permanence. For gcc-4.6.2 substitute 4.6.2 for 4.5.1 in the above export commands.
( **) Use the following:
[user@]$> . /usr/local/etc/setups.sh or source /usr/local/etc/setups.csh [user@]$> setup pgi
to add the Portland Group compiler and libraries to your
environment and to access the license manager. If you'd prefer not to
use "setup", be sure to set LM_LICENSE_FILE=/usr/local/pgi-904/linux86-64/9.0-4/license.dat
The binaries that are
generated using the PGI compilers
are not GNU compatible. Applications compiled with the Portland
Group compiler
must be statically
linked with their respective object libraries since the dynamically
linkable libraries are not available from cluster worker nodes.
6. Launching jobs on the jpsi cluster
The following job launch
procedure applies when using MVAPICH.
Jspi
nodes
are
quad-core, dual processor, so each node will run up to
eight processes.
Use /usr/local/mvapich/bin/mpirun_rsh*
as follows to launch jobs:
mpirun_rsh - will
automatically launch 8 processes on each jpsi node.
On Jpsi, the nodes listed in $PBS_NODEFILE will each be used 8 times,
and will be assigned 8 consecutive MPI rank numbers. For
example:
qsub -q jpsi -l nodes=4
my.script # request four nodes
# then, in the run script:
mpirun_rsh -np 32
milc.binary input.file > output.file
Here, $PBS_NODEFILE will have 4 jpsi nodes, eg, jpsi0101-jpsi0104, and
on each of these nodes eight instances of "milc.binary" will be run. There
are other versions of mpirun_rsh that can launch 1, 2 or 4 processes as
follows:
- mpirun_rsh_4
-
This will launch 4 processes on each node, with two processes
assigned to each processor socket.
- mpirun_rsh_2
-
This will launch 2 processes on each node, with each process
assigned to a different processor socket.
- mpirun_rsh_1
-
This will launch 1 process on each node.
7. Compilers Available on the
kaon cluster
(cluster head-node: kaon1.fnal.gov)
| Compiler
Name
and
Version |
Binary Location |
Binary Name(s) |
|
Gnu
Compilers Version 3.3.4 |
/usr/local/gcc-3.3.4/bin |
gcc
|
|
Gnu
Compilers Version 3.4.6 |
/usr/bin |
g++, gcc, g77 |
| Gnu Compilers Version 4.1.0 |
/usr/bin
|
g++4, gcc4
|
| Gnu Compilers Version 4.3.2 |
/usr/local/gcc-4.3.2/bin |
g++, gcc, gfortran |
| Intel
C/Fortran Compiler (**) |
/fnal/ups/prd/intel/v10_1_008/bin/ |
icc, ifort |
| Portland Group (+) |
/fnal/ups/prd/pgi/v7.1-2/linux86-64/7.1-2/bin
|
pgCC, pgcc,
pgf77, pgf90 |
MVAPICH Version
1.1.0
|
/usr/local/mvapich-1.1.0-ofed-1.4.2/bin
|
mpicc,
mpif77, mpif90 |
MVAPICH2
Version 1.2
|
/usr/local/mvapich2-1.2-ofed-1.4.2/bin
|
mpicc,
mpif77, mpif90 |
Open MPI Version 1.3.4
|
/usr/local/openmpi-1.3.4/bin
|
mpicc, mpif77, mpif90 |
(**) Use the following:
[user@]$> . /usr/local/etc/setups.sh or source /usr/local/etc/setups.csh [user@]$> setup intel v10_1_008
to add the Intel compiler and
libraries to your environment.
(+) Use the following:
[user@]$> . /usr/local/etc/setups.sh or source /usr/local/etc/setups.csh [user@]$> setup pgi
to add the Portland Group
compiler and libraries to your environment and to access the license
manager. If you'd prefer not to use "setup", be sure to set LM_LICENSE_FILE=/fnal/ups/prd/pgi/v7.1-2/license.dat.
The binaries that are
generated using the PGI compilers are not GNU compatible. Applications
compiled with
the Portland Group and Intel
compilers must be statically
linked with their respective object libraries since the dynamically
linkable libraries are not available from cluster worker nodes.
8. Launching jobs on the kaon cluster
The following job launch
procedure applies when using MVAPICH.
Kaon nodes are dual-core, dual processor, so each node will run up to
four processes.
Use /usr/local/mvapich/bin/mpirun_rsh*
as follows to launch jobs:
mpirun_rsh - will
detect
will automatically launch 4 processes on each Kaon node
On Kaon, the nodes listed in $PBS_NODEFILE will each be used 4 times,
and will be assigned 4 consecutive MPI rank numbers. For example:
qsub
-q
kaon
-l nodes=4 my.script #
request four nodes
# then, in the run
script:
mpirun_rsh -np 16 milc.binary
input.file > output.file
Here, $PBS_NODEFILE will have 4 kaon nodes, eg,
kaon0101-kaon0104, and on each of these nodes four instances of "milc.binary" will be run.
- mpirun_rsh_2
- This will launch 2 processes on each node, with
each process assigned to a different processor socket.
- mpirun_rsh_1
- This will launch 1 process on each node.
The following jobs launch procedure applies when using Open MPI.
Like
MVAPICH
above, Open MPI's mpirun has been patched to
automatically use 4 processes on each kaon worker node. The
syntax is:
mpirun -np xxx binary < input >
output
9. Compiling an MPI application
Two MPI versions are
available on the Infiniband
clusters - MVAPICH, which uses
the Ohio State University
MPI and Open MPI. We
recommend using mvapich.
To build codes, use
$ /usr/local/mvapich/bin/mpicc # or mpiCC, mpif77, etc... (Infiniband) or $ /usr/local/openmpi/bin/mpicc # or mpiCC, mpif77, etc... (Infiniband)
One of the MPICH versions as compiled with the Portland Group or Intel compiler.
To use a compiler, first
check that its executable path is in your shell's
PATH environment variable. Use the UNIX env command
to see how PATH is
currently set. If the desired compiler is not in your PATH, edit
your .bashrc
(for bash or sh users) or .cshrc (for
tcsh or csh users)
to add the compiler's path. For example, to add mpiCC, mpicc and mpif77
to your path, add the following line to your shell's start-up script:
| .bashrc |
.cshrc |
| export PATH=$PATH:/usr/local/mvapich/bin |
setenv PATH $PATH:/usr/local/mvapich/bin |
Here is how to compile
example program cpi.c which comes with the
mvapich
software distribution.
$ cp /usr/local/mvapich/example/cpi.c . # copy source to your directory $ mpicc cpi.c -o cpi # compile
|