Staging Files from the Enstore Tape System

The interface to the Fermilab tape robot is via Enstore. Documentation includes the User's Guide.

Each project will be assigned a quota of tapes on the STK robot. The namespace of Enstore files in your project area appears as a UNIX file system mounted at directory
/pnfs/lqcd/projectName where projectName is the charge name assigned to your project. Enstore and the /pnfs/lqcd area are only accessible from the login nodes kaon1.fnal.gov, jpsi1.fnal.gov, ds1.fnal.gov, lqcd.fnal.gov,lqcdsrm.fnal.gov and not from the cluster worker nodes. Hence, files within the Enstore system must be first staged to the cluster login node disk using the dccp command before they can be copied to the /scratch area of the workers.

Managing /pnfs/lqcd Project Area

• /pnfs/lqcd/projectName looks like a standard directory, but you have to use a special command (dccp) to copy files into and out of this area.  However,  you can use file and directory manipulation commands, such as find, stat, mv, rm, mkdir, rmdir, chmod, chgrp, chown, etc. to locate tape files, print an inode's content, rename and delete files, manipulate subdirectories, change permissions, and so forth. Using standard UNIX commands means scripts to manage tape files are almost as unchanged from scripts that manage disk files.

• dccp has the semantics of cp, but wildcards are not allowed.  So, you will need to script the sequence of dccp commands to copy your files into /pnfs/lqcd/projectName.

• In dccp commands, the source and destination directories will determine whether files are copied to tape or read from tape.
     Commands like
      dccp  file  /pnfs/lqcd/projectName/subdir/file
     will copy a file to tape.  Commands like
      dccp  /pnfs/lqcd/projectName/subdir/file /destdir/file
     will copy a file from tape.

• dccp uses a disk layer (dCache) to cache files on their way to or from tape.  So when writing to tape, the file is actually stored on a dCache disk, and is subsequently written to tape as soon as practical.  A dccp command writing to tape will return success as soon as the file has been transfered to disk.  When reading from tape, dccp will not return until the tape has been mounted by the robot and the file read to disk.
  

• By default your files will be grouped together onto tapes that will hold only ProjectName "file family" files.  Enstore file families are explained in the next section below. When you delete a file (with rm), only the metadata is removed, but the file remains on tape.  Once all of the files are deleted from a tape, you may request that the tape be recycled.  Your allocation will be charged when the first file on a tape is written, and you will get a refund if a tape is recycled.   If you would like to have different groups of files - some which are archival and never deleted, and others that can be deleted and their containing tapes recycled - email us at lqcd-admin@fnal.gov and we will set you up with multiple file families.  File families are tied to subdirectories.
  

Enstore file families

The Enstore system contains more metadata for tape files beyond the standard UNIX inode information for files. The command below displays the extra information for the projectName area:

$ enstore pnfs --tags /pnfs/lqcd/projectName
.(tag)(library) = 9940
.(tag)(file_family) = lqcd
.(tag)(file_family_wrapper) = cpio_odc
.(tag)(file_family_width) = 1
.(tag)(storage_group) = lqcd
-rw-rw-r-- 11 11072 9540 4 Feb 11 14:29 /pnfs/lqcd/projectName/.(tag)(library)
-rw-rw-r-- 11 11072 9540 4 Feb 11 14:30 /pnfs/lqcd/projectName/.(tag)(file_family)
-rw-rw-r-- 11 11072 9540 8 Feb 11 14:31 /pnfs/lqcd/projectName/.(tag)(file_family_wrapper)
-rw-rw-r-- 11 11072 9540 1 Feb 11 14:31 /pnfs/lqcd/projectName/.(tag)(file_family_width)
-rw-rw-r-- 11 root  root 4 Feb 11 14:31 /pnfs/lqcd/projectName/.(tag)(storage_group)

The file_family tab offers a convenient way for a group to organize their data on tape. The file_family specifies the logical name for a set of tapes. File families may be specified on a per directory basis and are, by default, inherited from the parent dirctory when a new directory is created. Enstore maintains files belonging to separate file families on separate sets of tape cartriges. This feature facilitates shelving or the removal of little used data sets from the robot without affecting other data sets.