Filesystem Details

The layout of the various filesystems available at the Fermilab LQCD Computing Facility is illustrated in the following figure:

/home Filesystem

Each user has a different home directory on the kaon, jpsi and ds clusters and this home directory is physically located on the machine kaon2.fnal.gov, jpsi2.fnal.gov and ds2.fnal.gov respectively. These home directories are mounted over NFS on all the worker nodes in their corresponding clusters. The user quota on these disks are several GBs and this space is backed up nightly.

The home directories corresponding to each cluster can be accessed between clusters from the cluster login head nodes (kaon1.fnal.gov, jpsi1.fnal.gov and ds1.fnal.gov) under the /ClusternameHome directory. Thus kaon, jpsi and ds cluster home directories can be accessed as /kaonhome, /jpsihome and /dshome respectively, on different cluster login head nodes. NOTE: Not all home directory mounts may exist on each cluster login head node and if you require access to a missing home directory please email us at lqcd-admin@fnal.gov and we will respond to your request promptly.

This home area storage space can be used as "run" directory for light production or testing but it is not suitable for configs or props.

Lustre Filesystem on the LQCD Clusters

Lustre is a scalable, secure, robust, highly-available cluster file system. It is designed, developed and maintained by Sun Microsystems, Inc. The current Lustre Portable Operating System Interface (POSIX) is comparable with NFS and Lustre filesystem supports MPI-IO as well.

The Lustre Filesystem is mounted in the directory /lqcdproj on ds1.fnal.gov, jpsi1.fnal.gov, kaon1.fnal.gov, lqcdsrm.fnal.gov and on all the D/s, J/psi and kaon worker nodes. This filesystem is not backed up and is meant to be used as a volatile storage space. Quotas are enfoced on the /lqcdproj file system and count against your project allocation. Please send email to lqcd-admin@fnal.gov to apply for a top level directory for your project and for quota maintenance. LQCD Lustre usage reports and plots updated daily are available here.

Please refer to the following page for more details about the various user commands.

/project Filesystem

This storage area is typically used for approved projects. This filesystem is accessible from all cluster worker nodes via the Lustre file-system and is automatically backed up every night. This storage space is suitable for output logs, meson correlators, and other small data files and it should NOT be used for storing fields e.g configs, quark propagators.

/pnfs/lqcd Filesystem

/pnfs/lqcd is the FNAL central dCache filesystem with tape backup facility and is intended for permanent storage of parameter files and results. Although this appears to be a standard disk area, it is not.  Commands to manipulate file and directory metadata (for example, rm, mv, chmod, mkdir, rmdir) will work here, but commands like cp or cat will not.  Instead, you will need to use dccp to copy files into and out of this area.  dccp has syntax like cp, i.e.
   
    dccp source destination

If destination is in /pnfs/lqcd, the command will copy the source file to tape.  If source is in /pnfs/lqcd, the command will copy the file on tape to the destination specified.

/pnfs/lqcd is accessible only from the kaon, jpsi and ds cluster login head nodes, kaon1.fnal.gov, jpsi1.fnal.gov and ds1.fnal.gov, respectively.

PVFS Filesystem on the KAON and J/psi clusters

The PVFS Filesystem available on the jpsi and kaon worker nodes as a temporary filesystem that gets created at the start of your batch job, by concatenating a set of local disks from the nodes allocated to your job, into a logical flat filesystem and deleted at the end of the batch job.

The PVFS filesystem is not created automatically for all the batch jobs. The users need to specify the following option as a part of their qsub command to enable the PVFS filesystem:

"-v PVFS=" or "-v PVFS=0" (to use the local disks from all the nodes of your batch job.)
or
"-v PVFS=x" (to use the local disk from the first x nodes of your batch job.) 

The PVFS filesystem upon creation can be accessed through the directory /pvfs from any of the nodes in your job. Please refer to the following link for more details about the PVFS filesystem.