FNAL - LQCD Documentation

New Users

User Authentication

Hardware Details

Software Details

Filesystem Details

Mass/Tape Storage Details

Transferring Files

Compilers

Runtime Environment

TORQUE Batch System
  – Allocation

FAQs

Quick Links

  1. Compilers and Communication Libraries
  2. Compilers available for the jpsi cluster
  3. Launching jobs on the jpsi cluster
  4. Compilers available for the kaon & pion cluster
  5. Launching jobs on the kaon & pion cluster
  6. Compilers available for the qcd cluster
  7. Launching jobs on the qcd cluster
  8. Compiling an MPI application

1. Compilers and Communication Libraries

The tables below list all the compilers installed on the various Fermilab LQCD clusters. Unix man page documentation is available for individual compilers. The Portland Group compilers are not binary compatible with GNU meaning that they will not accept object files or libraries created by the GNU compilers. Compiler drivers, such as mpicc, are provided for use when compiling programs containing MPI message passing directives. The implementation of MPI is MPICH using an abstract communication device implemented in GM, VMI and VAPI.

2. Compilers Available for the jpsi cluster (cluster head-node: jpsi1.fnal.gov)

Compiler Name and Version Binary Location Binary Name(s)
Gnu Compilers Version 3.4.6 /usr/bin g++, gcc, g77
Gnu Compilers Version 4.3.2 /usr/local/gcc-4.3.2/bin g++, gcc, g77
Gnu Compilers Version 3.2.3 /opt/gcc-3.2.3/bin g++, gcc, g77
Portland Group (**) /usr/local/pgi-904/linux86-64/9.0-4/bin
 		pgCC, pgcc, pgf77, pgf90
MVAPICH Version 1.1.0
 		/usr/local/mvapich-1.1.0-ofed-1.4.2/bin
 		mpicc, mpif77, mpif90
MVAPICH2 Version 1.2
 		/usr/local/mvapich2-1.2-ofed-1.4.2/bin
 		mpicc, mpif77, mpif90
Open MPI Version 1.3.4
 		/usr/local/openmpi-1.3.4/bin
 		mpicc, mpif77, mpif90

(**) Use the following: 

. /usr/local/etc/setups.sh       or       source /usr/local/etc/setups.csh
setup pgi

to add the Portland Group compiler and libraries to your environment and to access the license manager. If you'd prefer not to use "setup", be sure to set LM_LICENSE_FILE=/usr/local/pgi-904/linux86-64/9.0-4/license.dat

The binaries that are generated using the PGI compilers are not GNU compatible. Applications compiled with the Portland Group compiler must be statically linked with their respective object libraries since the dynamically linkable libraries are not available from cluster worker nodes.

3. Launching jobs on the jpsi cluster

The following job launch procedure applies when using MVAPICH.  Jspi nodes are quad-core, dual processor, so each node will run up to eight processes.

Use /usr/local/mvapich/bin/mpirun_rsh* as follows to launch jobs:

  mpirun_rsh - will automatically launch 8 processes on each jpsi node.

On Jpsi, the nodes listed in $PBS_NODEFILE will each be used 8 times, and  will be assigned 8 consecutive MPI rank numbers.  For example:

  qsub -q jpsi -l nodes=4 my.script     # request four nodes
  # then, in the run script:
  mpirun_rsh -np 32 milc.binary input.file > output.file

Here, $PBS_NODEFILE will have 4 jpsi nodes, eg, jpsi0101-jpsi0104, and on each of these nodes eight instances of "milc.binary" will be run. There are other versions of mpirun_rsh that can launch 1, 2 or 4 processes as follows:
  •   mpirun_rsh_4 - This will launch 4 processes on each node,  with two processes assigned to each processor socket.
  •   mpirun_rsh_2 - This will launch 2 processes on each node,  with each process assigned to a different processor socket.
  •   mpirun_rsh_1 - This will launch 1 process on each node.

4. Compilers Available for the kaon & pion cluster (cluster head-node: kaon1.fnal.gov)

(*) Use the following: 

. /usr/local/etc/setups.sh       or       source /usr/local/etc/setups.csh
setup intel v9_1

to add the Intel compiler and libraries to your environment.

(**) Use the following: 

. /usr/local/etc/setups.sh       or       source /usr/local/etc/setups.csh
setup pgi

to add the Portland Group compiler and libraries to your environment and to access the license manager. If you'd prefer not to use "setup", be sure to set LM_LICENSE_FILE=/fnal/ups/prd/pgi/v7.1-2/license.dat.

The binaries that are generated using the PGI compilers are not GNU compatible. Applications compiled with the Portland Group, Pathscale and Intel compilers must be statically linked with their respective object libraries since the dynamically linkable libraries are not available from cluster worker nodes.

5. Launching jobs on the kaon & pion cluster

The following job launch procedure applies when using MVAPICH. Kaon nodes are dual-core, dual processor, so each node will run up to four processes.  Pion nodes are single-core, single processor, so each node will run only one process.

  Use /usr/local/mvapich/bin/mpirun_rsh* as follows to launch jobs:

  mpirun_rsh - will detect whether the job is running on Kaon or Pion, and
will automatically launch 4 processes on each Kaon node, and 1 process on each Pion node.  

On Kaon, the nodes listed in $PBS_NODEFILE will each be used 4 times, and will be assigned 4 consecutive MPI rank numbers.  For example:

      qsub -q kaon -l nodes=4 my.script     # request four nodes
         # then, in the run script:
         mpirun_rsh -np 16 milc.binary input.file > output.file

   Here, $PBS_NODEFILE will have 4 kaon nodes, eg, kaon0101-kaon0104, and on each of these nodes four instances of "milc.binary" will be run.

On Pion, the nodes listed in $PBS_NODEFILE will each be used 1 times. For example:

         qsub -q pion -l nodes=16 my.script     # request sixteen nodes
         # then, in the run script:
         mpirun_rsh -np 16 milc.binary input.file > output.file

    Here, $PBS_NODEFILE will have 16 Pion nodes, eg, pion0101-pion0116, and on each of these nodes one instance of "milc.binary" will be run. There are other versions of mpirun_rsh that can launch 1 or 2 processes as follows:
  •   mpirun_rsh_2 - Use on Kaon only.  This will launch 2 processes on each node, with each process assigned to a different processor socket.
  •   mpirun_rsh_1 - Une on either Kaon or Pion.  This will launch 1 process on each node node.
The following jobs launch procedure applies when using Open MPI (Kaon only). Like MVAPICH above, Open MPI's mpirun has been patched to automatically use 4 processes on each kaon worker node.  The syntax is:

       mpirun -np xxx binary < input > output

Open MPI has not yet been configured for Pion.

6. Compilers Available for the qcd cluster (cluster head-node: lqcd.fnal.gov)

Compiler Name and Version Binary Location Binary Name(s)
Gnu Compilers Version 3.4.6 /usr/bin g++, gcc, g77
Gnu Compilers Version 3.2.3 /usr/local/gcc-3.2.3/bin
 g++, gcc, g77
Gnu Compilers Version 3.4.5 /usr/local/gcc-3.4.5/bin
 g++, gcc, g77
Intel Compiler Version 8.1 (*) /fnal/ups/prd/intel/v8_1/bin
 icc, ifc (C and C++, F77 and F90)
Intel Compiler Version 9.1 (*) /fnal/ups/prd/intel/v9_1/bin icc, ifc (C and C++, F77 and F90)
MPICH-GM with Intel Compiler
 /usr/local/mpich-gm-intel/bin
 mpicc, mpif77, mpif90
MPICH-VMI Version 3b
 /usr/local/mpich-vmi/bin mpicc, mpif77, mpif90
MPICH-VMI with Intel Compiler /usr/local/mpich-vmi-intel/bin mpicc, mpif77, mpif90
Portland Group (**) /fnal/ups/prd/pgi/v5_0/linux86/5.0/bin
 pgCC, pgcc, pgf77, pgf90
MPICH-GM with Portland Group Compiler
 /usr/local/mpich-gm-pgi/bin
 mpicc, mpiCC, mpif77, mpif90
MPICH-VMI with Portland Group Compiler /usr/local/mpich-vmi-pgi/bin mpicc, mpiCC, mpif77, mpif90

(*) Use the following: 

. /usr/local/etc/setups.sh       or       source /usr/local/etc/setups.csh
setup intel v8_1

to add the Intel compiler version 8.1 and libraries to your environment.

(**) Use the following: 

. /usr/local/etc/setups.sh       or       source /usr/local/etc/setups.csh
setup pgi

to add the Portland Group compiler and libraries to your environment and to access the license manager. If you'd prefer not to use "setup", be sure to set LM_LICENSE_FILE=/fnal/ups/prd/pgi/v5_0/license.dat.

The binaries that are generated using the PGI compilers are not GNU compatible. Applications compiled with the Portland Group and Intel compilers must be statically linked with their respective object libraries since the dynamically linkable libraries are not available from cluster worker nodes.

7. Launching jobs on the qcd cluster

 Two MPI versions are available on "qcd" - mpich-vmi from NCSA and
 mpich-gm from Myrinet.

 To build codes, use:

   /usr/local/mpich-vmi/bin/mpicc    # or mpiCC, mpif77, etc...
             or
   /usr/local/mpich-gm/bin/mpicc     # or mpiCC, mpif77, etc...

 To run codes, use:

    mpich-vmi:
      mpirun -np N  your.binary args
    mpich-gm:
      mpirun -np N your.binary args

 To redirect stdin on VMI jobs, use

      mpirun -np N -stdin stdin.file your.binary args

8. Compiling an MPI application

Two MPI versions are available on the Infiniband clusters - MVAPICH, which uses the Ohio State University MPI and Open MPI. We recommend using mvapich. Two MPI versions are available on the Myrinet (qcd) cluster - mpich-gm, which uses the Myrinet MPI and mpich-vmi.

To build codes, use 

$ /usr/local/mvapich/bin/mpicc     # or mpiCC, mpif77, etc...  (Infiniband)
             or 
$ /usr/local/openmpi/bin/mpicc     # or mpiCC, mpif77, etc...  (Infiniband)
             or
$ /usr/local/mpich-vmi/bin/mpicc   # or mpiCC, mpif77, etc...  (Myrinet)
             or
$ /usr/local/mpich-gm/bin/mpicc    # or mpiCC, mpif77, etc...  (Myrinet)
             or
One of the MPICH versions as compiled with the Portland Group or Intel compiler.
To use a compiler, first check that its executable path is in your shell's PATH environment variable. Use the unix env command to see how PATH is currently set. If the desired compiler is not in your PATH, edit your .bashrc (for bash or sh users) or .cshrc (for tcsh or csh users) to add the compiler's path. For example, to add mpiCC, mpicc and mpif77 to your path, add the following line to your shell's start-up script:

.bashrc .cshrc
export PATH=$PATH:/usr/local/mvapich/bin setenv PATH $PATH:/usr/local/mvapich/bin

Here is how to compile example program cpi.c which comes with the mvapich software distribution. 

$ cp /usr/local/mvapich/example/cpi.c .      # copy source to your directory
$ mpicc cpi.c -o cpi                         # compile
usqcd-webmaster@usqcd.org