The layout of the various filesystems available at the Fermilab
LQCD Computing Facility is illustrated in the following figure:
Each user has a different home directory on the ds, bc and pi0 clusters
and this home directory is physically located on the machine
These home directories are mounted over NFS on all the worker nodes
in their corresponding clusters.
The user quota on these disks are several GBs and this space is
backed up nightly.
The home directories corresponding to each cluster can be
accessed between clusters from the cluster login head nodes
pi0.fnal.gov) under the
Thus ds, bc and pi0 cluster home directories can be accessed as
respectively, on different cluster login head nodes.
NOTE: Not all home directory mounts may exist on each cluster
login head node and if you require access to a missing home directory please
email us at firstname.lastname@example.org and
we will respond to your request promptly.
This home area storage space can be used as "run" directory for
light production or testing but it is not suitable for configurations
Lustre Filesystem on the LQCD Clusters
Lustre is a scalable, secure, robust, highly-available cluster file system.
The current Lustre Portable Operating System Interface (POSIX) is comparable
with NFS and Lustre filesystem supports MPI-IO as well.
The Lustre Filesystem is mounted in the directory /lqcdproj on
lqcdsrm.fnal.gov and on all the
cluster worker nodes.
This filesystem is not backed up and is meant to be used as a volatile
Quotas are enforced on the
/lqcdproj file system and count
against your project allocation.
Please send email to email@example.com
to apply for a top level directory for your project and for quota maintenance.
LQCD Lustre usage reports are updated regularly and are available
To transfer between Lustre using Globus Online please follow the steps listed
on this page.
Please refer to the following page for more details about the various Lustre user commands.
Our Lustre version is 2.5.3.
This storage area is typically used for approved projects.
This filesystem is accessible from all cluster worker nodes via NFS using
IP-over-Infiniband and is automatically backed up every night.
This storage space is suitable for output logs, meson correlators, and
other small data files and it should NOT be used for storing fields
e.g configurations, quark propagators.
/pnfs/lqcd is the FNAL central
dCache filesystem with tape backup
facility and is intended for permanent storage of parameter files and results.
Although this appears to be a standard disk area, it is not.
Commands to manipulate file and directory metadata (for example,
rmdir) will work here, but commands like
cat will not.
Instead, you will need to use
dccp to copy files into and out
of this area.
dccp has syntax like
dccp source destination
destination is in
/pnfs/lqcd, the command
will copy the source file to tape.
source is in
/pnfs/lqcd, the command will
copy the file on tape to the destination specified.
/pnfs/lqcd is accessible only from the ds, bc and pi0
cluster login head nodes,