Staging Files from the Enstore Tape System
The interface to the Fermilab tape robot is via Enstore.
Documentation includes the
Enstore/dCache User's Guide.
Each project will be assigned a quota of tapes on the
The namespace of Enstore files in your project area appears
as a UNIX file system mounted at directory
projectName is the charge name assigned to
Enstore and the
/pnfs/lqcd area are only
accessible from the login head nodes such as
PNFS is also available on our
data movement node.
Hence, files within the Enstore system must first be
staged to disk using the
dccp command before they can
be copied to the
/scratch area of the workers.
NOTE: We strongly encourage users to use our
lqcdsrm.fnal.gov data movement node for moving data
between tape, disk and remote locations.
This node has much faster network connections allowing better
And performing your transfers here does not bog down other users
on the login head nodes.
Managing /pnfs/lqcd Project Area
looks like a standard directory, but you have to use a
special command (
dccp) to copy files into and
out of this area.
However, you can use file and directory manipulation
commands, such as
find, stat, mv, rm, mkdir, rmdir,
chmod, chgrp, chown etc. to locate tape files,
print an inode's content, rename and delete files,
manipulate subdirectories, change permissions, and so forth.
Using standard UNIX commands means scripts to manage tape files
are almost unchanged from scripts that manage disk files.
dccp has the semantics of
except that wildcards are not allowed.
So, you will need to script the sequence of
commands to copy your files into
dccp commands, the source and destination
directories will determine whether files are copied to tape
or read from tape.
- Commands like
/pnfs/lqcd/projectName/subdir/file will copy a file
- Commands like
/destdir/file will copy a file from tape.
dccp uses a disk layer (dCache) to cache files
on their way to or from tape.
So when writing to tape, the file is actually stored on a
dCache disk, and is subsequently written to tape as soon
dccp command writing to tape will return success
as soon as the file has been transfered to disk.
When reading from tape,
dccp will not return
until the tape has been mounted by the robot and the file
read to disk.
- By default your files will be grouped together onto
tapes that will hold only
"file family" files.
Enstore file families are explained in the next section below.
When you delete a file (with
rm), only the
metadata is removed, but the file remains on tape.
Once all of the files are deleted from a tape, you may request
that the tape be recycled.
Your allocation will be charged when the first file on a tape
is written and you will get a refund if a tape is recycled.
If you would like to have different groups of files - some
which are archival and never deleted and others that can be
deleted and their containing tapes recycled - email us at
and we will set you up with multiple file families.
File families are tied to subdirectories.
Please visit the Frequently Asked Questions page for our responses to common user questions regarding best practices and access methods to the Fermilab Mass Storage system.
Enstore file families
The Enstore system contains more metadata for tape files beyond
the standard UNIX inode information for files.
The command below displays the extra information for the
$ enstore pnfs --tags /pnfs/lqcd/projectName
.(tag)(library) = 9940
.(tag)(file_family) = lqcd
.(tag)(file_family_wrapper) = cpio_odc
.(tag)(file_family_width) = 1
.(tag)(storage_group) = lqcd
-rw-rw-r-- 11 11072 9540 4 Feb 11 14:29 /pnfs/lqcd/projectName/.(tag)(library)
-rw-rw-r-- 11 11072 9540 4 Feb 11 14:30 /pnfs/lqcd/projectName/.(tag)(file_family)
-rw-rw-r-- 11 11072 9540 8 Feb 11 14:31 /pnfs/lqcd/projectName/.(tag)(file_family_wrapper)
-rw-rw-r-- 11 11072 9540 1 Feb 11 14:31 /pnfs/lqcd/projectName/.(tag)(file_family_width)
-rw-rw-r-- 11 root root 4 Feb 11 14:31 /pnfs/lqcd/projectName/.(tag)(storage_group)
file_family tab offers a convenient way for
a group to organize their data on tape.
file_family specifies the logical name for a
set of tapes.
File families may be specified on a per directory basis and are,
by default, inherited from the parent dirctory when a new
directory is created.
Enstore maintains files belonging to separate file families on
separate sets of tape cartriges.
This feature facilitates
shelving or the removal
of little used data sets from the robot without affecting
other data sets.