Compilers and Communication Libraries

The table below lists compilers installed on the cluster. Unix man page documentation is available for individual compilers. The Portland Group compilers are not binary compatible with gnu meaning that they will not accept object files or libraries created by the gnu compilers. Compiler drivers, such as mpicc, are provided for use when compiling programs containing MPI message passing directives. The implementation of MPI is mpich using an abstract communication device implemented in GM, VMI and VAPI.

Compilers Available on LQCD

(*) Use the following:

. /usr/local/etc/setups.sh       or       source /usr/local/etc/setups.csh
setup intel v8_1

to add the Intel compiler version 8.1 and libraries to your environment.

(**) Use the following:

. /usr/local/etc/setups.sh       or       source /usr/local/etc/setups.csh
setup pgi

to add the Portland Group compiler and libraries to your environment and to access the license manager. If you'd prefer not to use "setup", be sure to set LM_LICENSE_FILE=/fnal/ups/prd/pgi/v5_0/license.dat.

(**)The binaries that are generated using the PGI compilers are not GNU compatible.

Applications compiled with the Portland Group and Intel compilers must be statically linked with their respective object libraries since the dynamically linkable libraries are not available from cluster worker nodes.

Compilers Available on KAON

(*) Use the following:

. /usr/local/etc/setups.sh       or       source /usr/local/etc/setups.csh
setup intel v9_1

to add the Intel compiler and libraries to your environment.

(**) Use the following:

. /usr/local/etc/setups.sh       or       source /usr/local/etc/setups.csh
setup pgi

to add the Portland Group compiler and libraries to your environment and to access the license manager. If you'd prefer not to use "setup", be sure to set LM_LICENSE_FILE=/fnal/ups/prd/pgi/v6.2-3/license.dat.

(**)The binaries that are generated using the PGI compilers are not GNU compatible.

Applications compiled with the Portland Group, Pathscale and Intel compilers must be statically linked with their respective object libraries since the dynamically linkable libraries are not available from cluster worker nodes.

Compiling an MPI application

Two MPI versions are available on the Infiniband (pion) cluster - mvapich, which uses the Ohio State University MPI and mpich-vmi, which uses the National Center for Supercomputing Applications (NCSA) MPI based on their VMI adapter layer. We recommend using mvapich; however, mpich-vmi will allow you to use the same binaries on the Myrinet and Infiniband clusters. Two MPI versions are also available on the Myrinet (qcd) cluster - mpich-gm, which uses the Myrinet MPI and mpich-vmi.

To build codes, use

$ /usr/local/mvapich/bin/mpicc     # or mpiCC, mpif77, etc...  (Infiniband)
             or 
$ /usr/local/mpich-vmi/bin/mpicc   # or mpiCC, mpif77, etc...  (Infiniband, Myrinet)
             or
$ /usr/local/mpich-gm/bin/mpicc    # or mpiCC, mpif77, etc...  (Myrinet)
             or
One of the MPICH versions as compiled with the Portland Group or Intel compiler.

To run codes, use

$ /usr/local/mvapich/bin/mpirun_rsh    (Infiniband)
             or
$ /usr/local/mpich-vmi/bin/mpirun      (Infiniband, Myrinet)
             or
$ /usr/local/mpich-gm/bin/mpirun       (Myrinet)

Please note that the syntax of these three versions of mpirun is different. Please use the following syntax:

mvapich: (Infiniband)
$ mpirun_rsh -rsh -np X your.binary arguments

mpich-vmi: (Infiniband)
$ mpirun -np X -specfile openib your.binary arguments

mpich-vmi: (Myrinet)
$ mpirun -np X -specfile myrinet your.binary arguments

mpich-gm: (Myrinet)
$ mpirun -np X your.binary arguments

Please note that "-specfile openib" for Infiniband or "-specfile myrinet" for Myrinet is critical for mpich-vmi. Your jobs will fail otherwise.

To use a compiler, first check that its executable path is in your shell's PATH environment variable. Use the unix env command to see how PATH is currently set. If the desired compiler is not in your PATH, edit your .bashrc (for bash or sh users) or .cshrc (for tcsh or csh users) to add the compiler's path. For example, to add mpiCC, mpicc and mpif77 to your path, add the following line to your shell's start-up script:

.bashrc	.cshrc
export PATH=$PATH:/usr/local/mvapich/bin	setenv PATH $PATH:/usr/local/mvapich/bin

Here is how to compile example program cpi.c which comes with the mvapich software distribution.

$ cp /usr/local/mvapich/example/cpi.c .      # copy source to your directory
$ mpicc cpi.c -o cpi                         # compile