The various filesystems available on the Fermilab LQCD Computing Facility are listed the table below




Area typically used for approved projects.
Visible on all cluster worker nodes via NFS file-system.
Backups nightly. Suitable for output logs, meson correlators and other small data files. NOT suitable for fields e.g configs, quark propagators.


Home area. Backups nightly. Visible on all cluster worker nodes via NFS.
Not suitable for configs or props.
Can be used as "run" directory for light production or testing.
Quota of about 6 to 10 GB per home directory.


Enstore Tape storage.
Visible on cluster login head nodes only.
Ideal for permanent storage of parameter files and results.
Must use special copy command: 'dccp'

/lqcdproj, /lustre1

Lustre storage. NO backups.
Visible on all cluster worker nodes.
Ideal for temporary storage (~month) of very large data files. NOT suitable for large number of small files. Disk space usage monitored and disk quotas enforced

The layout of the various filesystems available on the Fermilab LQCD Computing Facility is illustrated in the following figures.


Fig 1. Batch & filesystems layout for pi, bc and ds clusters.


Fig 2. Batch and filesystems layout for LQ1 Institutional cluster.


/home Filesystem


Each user has a home directory on the LQ1 and the rest of the clusters. These home directories are mounted over NFS on all the worker nodes in their corresponding clusters. For most part users do not have to worry about home directories in general. The user quota on these disks are several GBs and this space is backed up nightly.

The home directories corresponding to each cluster can also be accessed under the /ClusternameHome directory. Thus ds, bc and pi0 cluster home directories can be accessed as /dshome, /bchome and /pi0home respectively, on different cluster login head nodes. NOTE: Not all home directory mounts may exist on each cluster login head node and if you require access to a missing home directory please email us at and we will respond to your request promptly.

Lustre Filesystem

Lustre is a Lustre is a scalable, secure, robust, highly-available cluster file system. The current Lustre Portable Operating System Interface (POSIX) is comparable with NFS and Lustre filesystem supports MPI-IO as well.

The Lustre Filesystem is mounted at /lqcdproj and /lustre1 and your project will have a sub directory under the main mountpoint depending on your disk allocation. This filesystem is not backed up and is meant to be used as a volatile storage space. Quotas are enforced on the file system and count against your disk allocation. Please send email to to apply for a top level directory for your project and for quota maintenance.

To transfer between Lustre using Globus Online please follow the steps listed on this page.

Please refer to the following page for more details about the various Lustre user commands. Our Lustre version for /lustre1 is 2.12.2 and for /lqcdproj is 2.5.3

/project Filesystem

This limited storage area is typically used for approved projects. This filesystem is accessible from all cluster worker nodes via NFS as /project using IP-over-Infiniband and is automatically backed up every night. This storage space is suitable for output logs, meson correlators, and other small data files and it should NOT be used for storing fields e.g configurations, quark propagators.

/pnfs/lqcd Filesystem

/pnfs/lqcd is the central disk cache with tape backup facility and is intended for permanent storage of parameter files and results. Although this appears to be a standard disk area, it is not. Commands to manipulate file and directory metadata (for example, rm, mv, chmod, mkdir, rmdir) will work here, but commands like cp or cat will not. Instead, you will need to use dccp to copy files in and out of this area. dccp has syntax like cp, i.e.

dccp source destination

Fermi National Accelerator Laboratory
Managed by Fermi Research Alliance, LLC
for the U.S. Department of Energy Office of Science
Security, Privacy, Legal





peaceOpt2 item3 item2 item3 footerFermilabLogo