Quick Links
- Compilers and
Communication Libraries
- Compilers
available for the jpsi cluster
- Launching jobs
on
the jpsi cluster
- Compilers
available for the kaon & pion cluster
- Launching jobs
on
the kaon & pion cluster
- Compilers
available for the qcd cluster
- Launching jobs
on the qcd cluster
- Compiling an
MPI
application
1. Compilers and Communication
Libraries
The tables below list all
the
compilers installed on the
various Fermilab LQCD clusters. Unix man page documentation is
available for individual
compilers. The Portland Group compilers are not binary compatible with
GNU meaning that they will not accept object files or libraries created
by the GNU compilers. Compiler drivers, such as mpicc, are provided for
use when compiling programs containing MPI message passing
directives. The implementation of MPI is MPICH using an
abstract communication device implemented in GM, VMI and VAPI.
2. Compilers Available for the jpsi cluster (cluster
head-node: jpsi1.fnal.gov)
| Compiler
Name and Version |
Binary Location |
Binary Name(s) |
|
Gnu
Compilers Version 3.4.6 |
/usr/bin |
g++, gcc, g77 |
|
Gnu
Compilers Version 4.3.2 |
/usr/local/gcc-4.3.2/bin |
g++, gcc, g77 |
|
Gnu
Compilers Version 3.2.3 |
/opt/gcc-3.2.3/bin |
g++, gcc, g77 |
| Portland Group (**) |
/usr/local/pgi-904/linux86-64/9.0-4/bin
|
pgCC, pgcc,
pgf77, pgf90 |
MVAPICH Version
1.1.0
|
/usr/local/mvapich-1.1.0-ofed-1.4.2/bin
|
mpicc,
mpif77, mpif90 |
MVAPICH2
Version 1.2
|
/usr/local/mvapich2-1.2-ofed-1.4.2/bin
|
mpicc,
mpif77, mpif90 |
Open MPI Version 1.3.4
|
/usr/local/openmpi-1.3.4/bin
|
mpicc, mpif77, mpif90 |
(**) Use the following:
. /usr/local/etc/setups.sh or source /usr/local/etc/setups.csh setup pgi
to add the Portland Group
compiler and libraries to your
environment and to access the license manager. If you'd prefer not to
use "setup", be sure to set LM_LICENSE_FILE=/usr/local/pgi-904/linux86-64/9.0-4/license.dat
The binaries that are
generated using the PGI compilers
are not GNU compatible. Applications compiled with the Portland
Group compiler
must be statically
linked with their respective object libraries since the dynamically
linkable libraries are not available from cluster worker nodes.
3. Launching jobs on the jpsi cluster
The following job launch
procedure applies when using MVAPICH.
Jspi nodes are quad-core, dual processor, so each node will run up to
eight processes.
Use /usr/local/mvapich/bin/mpirun_rsh*
as follows to launch jobs:
mpirun_rsh - will
automatically launch 8 processes on each jpsi node.
On Jpsi, the nodes listed in $PBS_NODEFILE will each be used 8 times,
and will be assigned 8 consecutive MPI rank numbers. For
example:
qsub -q jpsi -l nodes=4
my.script # request four nodes
# then, in the run script:
mpirun_rsh -np 32
milc.binary input.file > output.file
Here, $PBS_NODEFILE will have 4 jpsi nodes, eg, jpsi0101-jpsi0104, and
on each of these nodes eight instances of "milc.binary" will be run. There
are other versions of mpirun_rsh that can launch 1, 2 or 4 processes as
follows:
- mpirun_rsh_4
-
This will launch 4 processes on each node, with two processes
assigned to each processor socket.
- mpirun_rsh_2
-
This will launch 2 processes on each node, with each process
assigned to a different processor socket.
- mpirun_rsh_1
-
This will launch 1 process on each node.
4. Compilers Available for the
kaon
& pion cluster
(cluster head-node: kaon1.fnal.gov)
(*) Use the following:
. /usr/local/etc/setups.sh or source /usr/local/etc/setups.csh setup intel v9_1
to add the Intel compiler and
libraries to your environment.
(**) Use the following:
. /usr/local/etc/setups.sh or source /usr/local/etc/setups.csh setup pgi
to add the Portland Group
compiler and libraries to your environment and to access the license
manager. If you'd prefer not to use "setup", be sure to set LM_LICENSE_FILE=/fnal/ups/prd/pgi/v7.1-2/license.dat.
The binaries that are
generated using the PGI compilers are not GNU compatible. Applications
compiled with
the Portland Group, Pathscale and Intel
compilers must be statically
linked with their respective object libraries since the dynamically
linkable libraries are not available from cluster worker nodes.
5. Launching jobs on the kaon &
pion cluster
The following job launch
procedure applies when using MVAPICH.
Kaon nodes are dual-core, dual processor, so each node will run up to
four processes. Pion nodes are single-core, single processor, so
each node will run only one process.
Use /usr/local/mvapich/bin/mpirun_rsh*
as follows to launch jobs:
mpirun_rsh - will
detect whether the job is running on Kaon or Pion, and
will automatically launch 4 processes on each Kaon node, and 1 process
on each Pion node.
On Kaon, the nodes listed in $PBS_NODEFILE will each be used 4 times,
and will be assigned 4 consecutive MPI rank numbers. For example:
qsub -q kaon -l nodes=4 my.script #
request four nodes
# then, in the run
script:
mpirun_rsh -np 16 milc.binary
input.file > output.file
Here, $PBS_NODEFILE will have 4 kaon nodes, eg,
kaon0101-kaon0104, and on each of these nodes four instances of "milc.binary" will be run.
On Pion, the nodes listed in $PBS_NODEFILE will each be used 1 times.
For example:
qsub -q pion -l nodes=16 my.script
# request sixteen nodes
# then, in the run
script:
mpirun_rsh -np 16 milc.binary
input.file > output.file
Here, $PBS_NODEFILE will have 16
Pion nodes, eg, pion0101-pion0116, and on each of these nodes one
instance of "milc.binary" will be run. There are other versions of
mpirun_rsh that can launch 1 or 2 processes as follows:
- mpirun_rsh_2
- Use
on Kaon only. This will launch 2 processes on each node, with
each process assigned to a different processor socket.
- mpirun_rsh_1
- Une
on either Kaon or Pion. This will launch 1 process on each node
node.
The following jobs launch procedure applies when using Open MPI
(Kaon only). Like MVAPICH above, Open MPI's mpirun has been patched to
automatically use 4 processes on each kaon worker node. The
syntax is:
mpirun -np xxx binary < input >
output
Open MPI has
not yet been configured for Pion.
6. Compilers Available for the
qcd cluster
(cluster head-node: lqcd.fnal.gov)
| Compiler Name and Version |
Binary Location |
Binary Name(s) |
|
Gnu
Compilers Version 3.4.6 |
/usr/bin |
g++, gcc, g77 |
| Gnu Compilers Version 3.2.3 |
/usr/local/gcc-3.2.3/bin
|
g++, gcc, g77 |
| Gnu Compilers Version 3.4.5 |
/usr/local/gcc-3.4.5/bin
|
g++, gcc, g77 |
| Intel
Compiler Version 8.1 (*) |
/fnal/ups/prd/intel/v8_1/bin
|
icc,
ifc
(C and C++, F77 and F90) |
| Intel
Compiler Version 9.1 (*) |
/fnal/ups/prd/intel/v9_1/bin |
icc,
ifc
(C and C++, F77 and F90) |
MPICH-GM with Intel Compiler
|
/usr/local/mpich-gm-intel/bin
|
mpicc,
mpif77, mpif90 |
MPICH-VMI Version 3b
|
/usr/local/mpich-vmi/bin |
mpicc,
mpif77, mpif90 |
| MPICH-VMI with Intel Compiler |
/usr/local/mpich-vmi-intel/bin |
mpicc,
mpif77, mpif90 |
| Portland Group (**) |
/fnal/ups/prd/pgi/v5_0/linux86/5.0/bin
|
pgCC, pgcc,
pgf77, pgf90 |
MPICH-GM with Portland Group Compiler
|
/usr/local/mpich-gm-pgi/bin
|
mpicc,
mpiCC, mpif77, mpif90 |
| MPICH-VMI with Portland Group Compiler |
/usr/local/mpich-vmi-pgi/bin |
mpicc,
mpiCC, mpif77, mpif90 |
(*) Use the
following:
. /usr/local/etc/setups.sh or source /usr/local/etc/setups.csh setup intel v8_1
to add the Intel compiler version 8.1 and libraries to
your environment.
(**) Use the
following:
. /usr/local/etc/setups.sh or source /usr/local/etc/setups.csh setup pgi
to add the Portland Group compiler and libraries to your
environment and to access the license manager. If you'd prefer not to
use "setup", be sure to set
LM_LICENSE_FILE=/fnal/ups/prd/pgi/v5_0/license.dat.
The binaries that are generated using the PGI compilers
are not GNU compatible. Applications compiled with the Portland
Group and Intel compilers must be statically
linked with their respective object libraries since the dynamically
linkable libraries are not available from cluster worker nodes.
7. Launching jobs on the qcd cluster
Two MPI versions are
available on "qcd" - mpich-vmi
from NCSA and
mpich-gm from Myrinet.
To build codes, use:
/usr/local/mpich-vmi/bin/mpicc
# or mpiCC, mpif77, etc...
or
/usr/local/mpich-gm/bin/mpicc # or mpiCC,
mpif77, etc...
To run codes, use:
mpich-vmi:
mpirun -np N your.binary args
mpich-gm:
mpirun
-np N your.binary args
To redirect stdin on VMI jobs, use
mpirun -np N -stdin stdin.file your.binary args
8. Compiling an MPI application
Two MPI versions are
available on the Infiniband
clusters - MVAPICH, which uses
the Ohio
State University MPI and Open MPI. We
recommend using mvapich.
Two MPI versions are available on the Myrinet (qcd) cluster - mpich-gm,
which uses the
Myrinet MPI and mpich-vmi.
To build codes, use
$ /usr/local/mvapich/bin/mpicc # or mpiCC, mpif77, etc... (Infiniband) or $ /usr/local/openmpi/bin/mpicc # or mpiCC, mpif77, etc... (Infiniband) or $ /usr/local/mpich-vmi/bin/mpicc # or mpiCC, mpif77, etc... (Myrinet) or $ /usr/local/mpich-gm/bin/mpicc # or mpiCC, mpif77, etc... (Myrinet) or One of the MPICH versions as compiled with the Portland Group or Intel compiler.
To use a compiler, first
check that its executable path is in your shell's
PATH environment variable. Use the unix env command to see how PATH is
currently set. If the desired compiler is not in your PATH, edit
your .bashrc (for bash or sh users) or .cshrc (for tcsh or csh users)
to add the compiler's path. For example, to add mpiCC, mpicc and mpif77
to your path, add the following line to your shell's start-up script:
| .bashrc |
.cshrc |
| export PATH=$PATH:/usr/local/mvapich/bin |
setenv PATH $PATH:/usr/local/mvapich/bin |
Here is how to compile
example program cpi.c which comes with the
mvapich
software distribution.
$ cp /usr/local/mvapich/example/cpi.c . # copy source to your directory $ mpicc cpi.c -o cpi # compile
|