rosette128px1

The various filesystems available on the Fermilab LQCD Computing Facility are listed the table below

Area

Description

/project/xxxxx

Area typically used for approved projects.
Visible on all cluster worker nodes via NFS file-system.
Backups nightly. Suitable for output logs, meson correlators and other small data files. NOT suitable for fields e.g configs, quark propagators.

/home/<username>

Home area. Backups nightly. Visible on all cluster worker nodes via NFS.
Not suitable for configs or props.
Can be used as "run" directory for light production or testing.
Quota of 5 GB per home directory.

/pnfs/lqcd

Enstore Tape storage.
Visible on cluster login head nodes only.
Ideal for permanent storage of parameter files and results.
Must use special copy command: 'dccp'

/lustre1

Lustre storage. NO backups.
Visible on all cluster worker nodes.
Ideal for temporary storage (~month) of very large data files. NOT suitable for large number of small files. Disk space usage monitored and disk quotas enforced
rosette128px1

The layout of the various filesystems available on the Fermilab LQCD Computing Facility is illustrated in the following figures.

rosette128px1

/home Filesystem

rosette128px1

Each user has a home directory on the LQ1 cluster. This home directory is mounted over NFS on all the worker nodes. For most part users do not have to worry about home directories in general. The user quota on these disks are several GBs and this space is backed up nightly.

Lustre Filesystem

Lustre is a Lustre is a scalable, secure, robust, highly-available cluster file system. The current Lustre Portable Operating System Interface (POSIX) is comparable with NFS and Lustre filesystem supports MPI-IO as well.

The Lustre Filesystem is mounted at /lustre1 and your project will have a sub directory under the main mountpoint depending on your disk allocation. This filesystem is not backed up and is meant to be used as a volatile storage space. Quotas are enforced on the file system and count against your disk allocation. Please send email to lqcd-admin@fnal.gov to apply for a top level directory for your project and for quota maintenance.

To transfer between Lustre using Globus Online please follow the steps listed on this page.

Please refer to the following page for more details about the various Lustre user commands. Our Lustre version for /lustre1 is 2.12.2.

/project Filesystem

This limited storage area is typically used for approved projects. This filesystem is accessible from all cluster worker nodes via NFS as /project using IP-over-Infiniband and is automatically backed up every night. This storage space is suitable for output logs, meson correlators, and other small data files and it should NOT be used for storing fields e.g configurations, quark propagators.

/pnfs/lqcd Filesystem

/pnfs/lqcd is the central disk cache with tape backup facility and is intended for permanent storage of parameter files and results. Although this appears to be a standard disk area, it is not. Commands to manipulate file and directory metadata (for example, rm, mv, chmod, mkdir, rmdir) will work here, but commands like cp or cat will not. Instead, you will need to use dccp to copy files in and out of this area. dccp has syntax like cp, i.e.

dccp source destination

Fermi National Accelerator Laboratory
Managed by Fermi Research Alliance, LLC
for the U.S. Department of Energy Office of Science
footerFermilabLogo
Security, Privacy, Legal

 

 

 

 

peaceOpt2 item3