The layout of the various filesystems available at the
Fermilab LQCD Computing Facility is illustrated in the following figure:
Each user has a different home directory on the
ds, bc and pi0 clusters and this home directory is physically located on
the machine ds2.fnal.gov,
respectively. These home directories are mounted
over NFS on all the worker nodes in their corresponding clusters. The
user quota on these disks are several GBs and this space is backed up
The home directories corresponding to each cluster can
be accessed between clusters from the cluster login head nodes (ds1.fnal.gov,
directory. Thus ds, bc and pi0 cluster home directories can be
respectively, on different cluster login head nodes. NOTE: Not all home directory mounts
may exist on each cluster login head node and if you require access to
a missing home directory please email us at firstname.lastname@example.org and we will
respond to your request promptly.
This home area storage space can be used as "run"
directory for light production or testing but it is not suitable for
configurations or propagators.
Lustre Filesystem on the LQCD Clusters
Lustre is a scalable, secure, robust, highly-available
cluster file system. The current Lustre Portable Operating System
Interface (POSIX) is comparable with NFS and Lustre filesystem supports
MPI-IO as well.
The Lustre Filesystem is mounted in the directory /lqcdproj
and on all the cluster worker nodes. This filesystem is not
backed up and is meant to be used as a volatile storage space. Quotas
are enforced on the /lqcdproj
file system and count against your project allocation. Please send
email to email@example.com
to apply for a top level directory for your project and for quota
LQCD Lustre usage reports and plots updated daily are available here.
To transfer between Lustre using Globus Online please follow the steps listed on this page.
Please refer to the following page
for more details about the various Lustre user commands. Our Lustre version is 2.5.3.
This storage area is typically used for approved
projects. This filesystem is accessible from all cluster worker nodes
via NFS using IP-over-Infiniband and is automatically backed up every night.
This storage space is
suitable for output logs, meson correlators, and other small data files
and it should NOT be used for storing fields e.g configurations, quark
is the FNAL central dCache
tape backup facility and is intended for permanent storage of
parameter files and results. Although this appears to be a standard
disk area, it is not. Commands to manipulate file and directory
work here, but commands like cp or cat will
not. Instead, you will need to use dccp to
into and out of this area. dccp has
syntax like cp,
is in /pnfs/lqcd,
command will copy the source file to tape. If source
is in /pnfs/lqcd,
command will copy the file on tape to the destination specified.
is accessible only from the ds, bc and pi0 cluster
login head nodes, ds1.fnal.gov,
bc1.fnal.gov and pi0.fnal.gov,
respectively and lqcdsrm.fnal.gov.