FNAL - LQCD Documentation

New Users / Account Renewal

User Authentication

Kerberos and SSH Troubleshooting

Submitting jobs to the TORQUE Batch System

Project Allocations

Software Documentation Details

Hardware Details

Filesystem Details

Mass/Tape Storage Details

Transferring Files


Using GPUs


Contact Us

Filesystem Details

The layout of the various filesystems available at the Fermilab LQCD Computing Facility is illustrated in the following figure:

/home Filesystem

Each user has a different home directory on the ds, bc and pi0 clusters and this home directory is physically located on the machine ds2.fnal.gov, bc2.fnal.gov and pi0s.fnal.govrespectively. These home directories are mounted over NFS on all the worker nodes in their corresponding clusters. The user quota on these disks are several GBs and this space is backed up nightly.

The home directories corresponding to each cluster can be accessed between clusters from the cluster login head nodes (ds1.fnal.gov, bc1.fnal.gov and pi0.fnal.gov) under the /ClusternameHome directory. Thus ds, bc and pi0 cluster home directories can be accessed as /dshome, /bchome and /pi0home respectively, on different cluster login head nodes. NOTE: Not all home directory mounts may exist on each cluster login head node and if you require access to a missing home directory please email us at lqcd-admin@fnal.gov and we will respond to your request promptly.

This home area storage space can be used as "run" directory for light production or testing but it is not suitable for configurations or propagators.

Lustre Filesystem on the LQCD Clusters

Lustre is a scalable, secure, robust, highly-available cluster file system. The current Lustre Portable Operating System Interface (POSIX) is comparable with NFS and Lustre filesystem supports MPI-IO as well.

The Lustre Filesystem is mounted in the directory /lqcdproj on pi0.fnal.gov, bc1.fnal.gov, ds1.fnal.gov, lqcdsrm.fnal.gov and on all the cluster worker nodes. This filesystem is not backed up and is meant to be used as a volatile storage space. Quotas are enforced on the /lqcdproj file system and count against your project allocation. Please send email to lqcd-admin@fnal.gov to apply for a top level directory for your project and for quota maintenance. LQCD Lustre usage reports are updated regularly and are available here.

To transfer between Lustre using Globus Online please follow the steps listed on this page.

Please refer to the following page for more details about the various Lustre user commands. Our Lustre version is 2.5.3.

/project Filesystem

This storage area is typically used for approved projects. This filesystem is accessible from all cluster worker nodes via NFS using IP-over-Infiniband and is automatically backed up every night. This storage space is suitable for output logs, meson correlators, and other small data files and it should NOT be used for storing fields e.g configurations, quark propagators.

/pnfs/lqcd Filesystem

/pnfs/lqcd is the FNAL central dCache filesystem with tape backup facility and is intended for permanent storage of parameter files and results. Although this appears to be a standard disk area, it is not. Commands to manipulate file and directory metadata (for example, rm, mv, chmod, mkdir, rmdir) will work here, but commands like cp or cat will not. Instead, you will need to use dccp to copy files into and out of this area. dccp has syntax like cp, i.e.

                dccp source destination

If destination is in /pnfs/lqcd, the command will copy the source file to tape. If source is in /pnfs/lqcd, the command will copy the file on tape to the destination specified.

/pnfs/lqcd is accessible only from the ds, bc and pi0 cluster login head nodes, ds1.fnal.gov, bc1.fnal.gov and pi0.fnal.gov, respectively and lqcdsrm.fnal.gov.