Filesystem Details

The layout of the various filesystems available at the Fermilab LQCD Computing Facility is illustrated in the following figure:

/home Filesystem

Each user has a different home directory on the lqcd and kaon clusters and this home directory is physically located on the machine lqcd.fnal.gov and kaon2.fnal.gov respectively. These home directories are mounted over NFS on all the worker nodes in their corresponding clusters. The user quota on these disks are several GBs and this spaces is backed up nightly.

The home directories on the kaon cluster can be accessed from lqcd.fnal.gov under the directory /kaonhome, while the home directories on the qcd cluster can be accessed from kaon1.fnal.gov under the directory /lqcdhome.

This storage space can be used as "run" directory for light production or testing but it is not suitable for configs or props.

Lustre Filesytem on the LQCD Clusters

The Lustre Filesystem is mounted in the directory /lqcdproj on jpsi1, kaon1, lqcdsrm and on all the Jpsi, Kaon and Pion worker nodes. This filesystem is not backed up and is meant to be used as a volatile storage space. Quotas are enfoced on the /lqcdproj file system count against your project allocation. You need to send mail to lqcd-admin@fnal.gov to apply for a top level directory for your project and for quota maintenance.

Lustre is a scalable, secure, robust, highly-available cluster file system. It is designed, developed and maintained by Sun Microsystems, Inc. The current Lustre Portable Operating System Interface (POSIX) is comparable with NFS and Lustre filesystem supports MPI-IO as well.

Please refer to the following page for more details about the various user commands

/pnfs/volatile Filesystem

/pnfs/volatile filesystem is a dcache filesystem where the least recently used files are deleted without any warning. This filesystem is intended to be used for temporary storage of large data files and results that maybe reused on the compute clusters. None of the files on this filesystem is backed up. Any file that is not used within the last six weeks are deleted automatically.

Files are copied into and out of /pnfs/volatile with the dccp command; see "dccp -h" for more information. Example:

dccp data.file /pnfs/volatile/kurt/data.file

The unix command "ls" will list the meta data of the files and directories in /pnfs/volatile; mkdir will create subdirectories; and rm and rmddir will remove files. Shell wild cards can be used on these commands.

/pnfs/volatile is accessible from all the head nodes and worker nodes.

/pnfs/lqcd Filesystem

/pnfs/lqcd is the FNAL central dcache filesystem with tape backupi facility and it is intended for permanent storage of parameter files and results.

Files are copied into and out of /pnfs/lqcd with the dccp command; see "dccp -h" for more information.

Unix commands work in in /pnfs/lqcd as described above in the /pnfs/volatile system.

Files can be copied directly to and from /pnfs/lqcd to /pnfs/volatile using dccp command but we recommend that you use the qcopy script instead as it checks for prior existence of files, correct file sizes and automatically retries failed copies.

/pnfs/lqcd is accessible only from the lqcd and kaon head nodes.

/project Filesystem

This storage area is typically used for approved projects. This filesystem is accessible from all clusters nodes via nfs and it is automatically backed up every night. This storage space is suitable for output logs, meson correlators, and other small data files and it should NOT be used for storing fields e.g configs, quark propagators.

/data/raid Filesystem

This file systems is located on a Raid Storage System and it is not backed up. This filesystem is an unmanaged common storage area available to all users i.e. there is no user quota set on this disk and individual disks are subject to filling up. Users must use the commands rcp or rsync to move data files in and out of this storage space. It is recommended that you use fcp when transferring files from withing multiple batch jobs. This filesystem is suitable for the storage of configurations and propagators.

PVFS Filesytem on the KAON and JPSI Clusters

The PVFS Filesystem available on the jpsi, kaon and pion nodes is a temporary filesystem that gets created at the start of your batch job, by concatenating a set of local disks from the nodes allocated to your job, into a logical flat filesystem and deleted at the end of the batch job.

The PVFS filesystem is not created automatically for all the batch jobs. The users need to specify the following option as a part of their qsub command to enable the PVFS filesystem:

"-v PVFS=" or "-v PVFS=0" (to use the local disks from all the nodes of your batch job.)
or
"-v PVFS=x" (to use the local disk from the first x nodes of your batch job.) 

The PVFS filesystem upon creation can be accessed through the directory /pvfs from any of the nodes in your job. Please refer to the following link for more details about the PVFS filesystem.