Staging Files from the Enstore Tape System

The interface to the Fermilab tape robot is via enstore. Documentation includes the User's Guide and User Notes.

Each project will be assigned a quota of tapes on the STK robot. The namespace of enstore files in your project area appears as a unix file system mounted at directory /pnfs/lqcd/projectName where projectName is the charge name assigned to your project. Enstore and the /pnfs area are only accessible from host lqd and not from cluster worker nodes. Hence, files within the enstore system must be first staged to lqcd disk using the encp command before they can be copied to the /scratch area of the workers.

Managing /pnfs/lqcd Project Area

You can use commands such as find to locate tape files, ls to list files, mkdir to make a new subdirectory, chmod to change permissions and and stat to print an inode's content. Using standard unix commands means scripts to manage tape files are almost as unchanged from scripts that manage disk files.

The enstore system contains more metadata for tape files beyond the standard unix inode information for files. The command below displays the extra information for the exampleProject area:

$ enstore pnfs --tags /pnfs/lqcd/exampleProject
.(tag)(library) = 9940
.(tag)(file_family) = lqcd
.(tag)(file_family_wrapper) = cpio_odc
.(tag)(file_family_width) = 1
.(tag)(storage_group) = lqcd
-rw-rw-r-- 11 11072 9540 4 Feb 11 14:29 /pnfs/lqcd/exampleProject/.(tag)(library)
-rw-rw-r-- 11 11072 9540 4 Feb 11 14:30 /pnfs/lqcd/exampleProject/.(tag)(file_family)
-rw-rw-r-- 11 11072 9540 8 Feb 11 14:31 /pnfs/lqcd/exampleProject/.(tag)(file_family_wrapper)
-rw-rw-r-- 11 11072 9540 1 Feb 11 14:31 /pnfs/lqcd/exampleProject/.(tag)(file_family_width)
-rw-rw-r-- 11 root  root 4 Feb 11 14:31 /pnfs/lqcd/exampleProject/.(tag)(storage_group)

The file_family tab offers a convenient way for a group to organize their data on tape. The file_family specifies the logical name for a set of tapes. File families may be specified on a per directory basis and are, by default, inherited from the parent dirctory when a new directory is created. Enstore maintains files belonging to separate file families on separate sets of tape cartriges. This feature facilitates "shelving" or the removal of little used data sets from the robot without affecting other data sets. An example of creating a new file family tag for a data set follows in the next session.

Copying files to enstore

The example below creates a new directory in the /pnfs/lqcd/exampleProject area called latAconfigs. A new file_family tag is created for this data set. Then, a single gauge configuration is copied to tape.

$ cd /pnfs/lqcd/exampleProject
$ mkdir latAconfigs
$ cd latAconfigs
$ enstore pnfs --file_family=exampleProject-latAconfigs
$ encp /data/raid1/data/myDataArea/latAconfigs/gf_latA_000030 .

Copying files from enstore

To copy a file from enstore tape to disk:

$ encp /pnfs/lqcd/exampleProject/latAconfigs/gf_latA_000130 /data/raid1/data/myDataArea/stage