Transfering Files Between US Lattice QCD Sites
The three labs hosting US Lattice QCD hardware -
Fermilab (FNAL), Jefferson
Lab (JLab), and Brookhaven (BNL) - have different computer security
configurations. FNAL uses Kerberos, and both JLab and BNL use ssh-based
firewalls. Because of these differences, users must employ different
sets of
commands to access these facilities and to transfer files, depending
upon the
starting location.
At JLab and BNL, users must first login to one of
several site firewall
nodes using ssh, and then login via ssh to the desired head node
(qcdochosta
or qcdochostb at BNL, qcdi01 or qcdi02 at JLab). Transfering files
across
this firewall into either facility requires two "hops" - first, files must
be scp'd to the firewall node, second scp'd from the
firewall node to the
target head node. A much superior alternative is to use ssh tunneling,
which relies on ssh processes running on the user's node and on the
firewall
node to move data directly to or from the target head node. This is
discussed
further below.
To login to FNAL computers, or to move files to or from
FNAL machines, users
must employ Kerberos software. First, a user must obtain a Kerberos
ticket.
Once a ticket is held, kerberized versions of telnet, rsh, rcp,
rlogin, ssh,
and scp
may be used to access FNAL systems. This is discussed further
below.
Globus Online
Fermilab provides data transfer between its Lattice QCD Lustre filesystem through Globus Online. Please follow the simple steps listed here.
SSH Tunneling
Both BNL and JLab rely on ssh firewalls for cyber
security. The ssh utilities
include a very powerful port forwarding facility which
greatly
simplifies accessing the BNL and JLab lattice QCD head nodes through
these
firewalls. The ssh command syntax which is required for port forwarding
is cumbersome and very difficult to memorize. Fortunately, Steve Adler
at BNL
has implemented a set of Perl scripts (tunnel.pl)
which
greatly simplify port forwarding.
The tunnel.pl is installed in /usr/local/bin/ on the
head nodes at BNL, JLab, and
FNAL. Rather than invoking tunnel.pl directly, users should use these
commands:
- tssh - open a tunnel if necessary,
and login via ssh to the requested head node
- tscp - open a tunnel if necessary,
and copy a file via ssh to/from the requested head node
- tclose - close any open tunnels
The tunnel.pl script relies on a configuration file
which is unique to each
user, $HOME/.tunnel/config. The syntax of the
configuration file is explained
in the BNL
documentation (see the notes in "Installing the tunnel script") and in
the main
tunnel.pl documentation
(see
"How do I configure tunnel.pl?").
Here is a sample configuration file from lqcd.fnal.gov:
qcdochostb.qcdoc.bnl.gov:2002:ssh.qcdoc.bnl.gov:dholmgre:localbnl:qcdochostb qcdi01.jlab.org:2003:jlab.org:johndoe:localjlab1:qcdi01 qcdi02.jlab.org:2004:jlab.org:johndoe:localjlab2:qcdi02
The fields in the configuration file, separated by
colons, are as follows:
- Fully qualified domain name of target head node.
- TCP port number to use for the local end of tunnel.
Note - this port number must be unique for all users and all target
head nodes. If a non-unique number is used, a tunnel command may fail
with a "port in use" error message. The port numbers specified should
be greater than 1023.
- The fully qualified domain name of the ssh firewall
node.
- The username to use on the ssh firewall node. Note in
this example that different usernames are used at BNL and JLab.
- The local network name for the machine running the
tunnel scripts. Note - to avoid ssh errors similar to "REMOTE HOST
IDENTIFICATION HAS CHANGED!", you should use unique local network names
for each target head node. This requires the system administrator to
add aliases for "localhost" to /etc/hosts. On lqcd.fnal.gov, localbnl,
localjlab1, and localjlab2 are available aliases for localhost.
Additional aliases can be added on request. The local network name is
used by tssh and tscp to connect to
the "end" of the ssh tunnel connected to the local machine.
- An alias for the target head node. This allows you to
use a name for the destination head node which is easier to type than
the fully qualified domain name.
Examples
In all of the examples shown below, we have assumed that
the user has first
loaded his ssh private key via ssh-agent $SHELL and
ssh-add. Otherwise, on each command
invocation there will be an
additional prompt for ssh passphrase, or for the password on the local
node.
Logging into a Remote Head Node
In this example, a tunnel has not yet been opened on the
JLab firewall. So,
in response to tssh the user will be asked for his
password for the
JLab firewall. If successful, tssh will start an ssh
process on the firewall
node, then invoke ssh again to open a shell on qcdi02:
lqcd:~$ tssh qcdi02 Firing up tunnel to qcdi02.jlab.org through jlab.org using port 2004 ssh -e none -f -N -L 2004:qcdi02.jlab.org:22 johndoe@jlab.org johndoe@jlab.org's password: [johndoe@qcdi02 ~]$
Next, we exit from the shell on qcdi02,
returning to lqcd.
tssh to qcdi02 is invoked again.
This time, the tunnel
already exists, so no password is required:
[johndoe@qcdi02 ~]$ exit logout Connection to localjlab2 closed. lqcd:~$ tssh qcdi02 [johndoe@qcdi02 ~]$
On the BNL machine, according to the $HOME/.tunnel/config
file, a
different username is required. To login to a BNL head node, you must
explicitly give the remote username:
lqcd:~$ tssh dholmgre@qcdochostb Firing up tunnel to qcdochostb.qcdoc.bnl.gov through ssh.qcdoc.bnl.gov using port 2002 ssh -e none -f -N -L 2002:qcdochostb.qcdoc.bnl.gov:22 dholmgre@ssh.qcdoc.bnl.gov dholmgre@ssh.qcdoc.bnl.gov's password: -bash-2.05b$ hostname qcdochostb
The ssh port forwarding tunnels will continue to run on
the BNL and JLab
firewall nodes. When you are finished accessing these nodes, you should
close
the tunnels with tclose:
lqcd:~$ tclose Closing tunnel on port 2004 going to qcdi02.jlab.org through jlab.org Closing tunnel on port 2002 going to qcdochostb.qcdoc.bnl.gov through ssh.qcdoc.bnl.gov lqcd:~$
Notice that tclose will close all
existing tunnels.
Copying Files
In these examples, we assume that a tunnel has already
been established to the
target head nodes.
To copy a file from the local node to the target head
node, use tscp:
lqcd:~$ tscp test.test qcdi02:. test.test 100% |******************************************| 7 00:00
For BNL with our $HOME/.tunnel/config
file, you must give the
username:
lqcd:~$ tscp test.test dholmgre@qcdochostb:. test.test 100% |******************************************| 7 00:00
To copy files from the target head nodes, reverse the
arguments:
lqcd:~$ tscp qcdi02:test.test . test.test 100% |******************************************| 7 00:00 lqcd:~$ tscp dholmgre@qcdochostb:test.test . test.test 100% |******************************************| 7 00:00
Note that tscp takes standard scp
switches. So, to
recursively copy a directory, you could use:
lqcd:~$ tscp -r test.dir qcdi02:. file.a 100% |******************************************| 7 00:00 file.b 100% |******************************************| 7 00:00
Accessing Fermilab via Kerberos
Obtaining a Ticket - Brookhaven
From BNL, in order to login to the FNAL headnode,
lqcd.fnal.gov, or to transfer files into or out of
lqcd, you
first must obtain a kerberos ticket. To do this, use the
kinit command:
qcdochostb:~$ kinit -fr 7d johndoe@FNAL.GOV Password for johndoe@FNAL.GOV: qcdochostb:~$ qcdochostb:~$ klist -f Ticket cache: FILE:/var/krb5/security/creds/krb5cc_8386 Default principal: johndoe@FNAL.GOV
Valid starting Expires Service principal 06/23/12 18:16:00 06/24/12 18:15:54 krbtgt/FNAL.GOV@FNAL.GOV Renew until 06/30/12 18:15:54, Flags: FRIA qcdochostb:~$
Here, kinit has been used to obtain a
forwardable ticket which
is renewable for 7 days. Note that we have explicitly given the
principal
name, in this case, johndoe@FNAL.GOV. If you do not
give the
principal, instead username@FNAL.GOV will be used.
Users who have different login names at the three sites must include
their
principal in the kinit command.
A forwardable ticket may be used, once logged
into
lqcd.fnal.gov, to login to another FNAL system.
Kerberos tickets are valid for 24 hours. Renewable
tickets may be
renewed for an additional 24 hours at any time before expiration by
using
kinit -R
Note that renewing a ticket does not require you to type
in your password.
The "-r 7d" switch in this example requests a ticket
which may be
renewed in this manner for up to 7 days. After 7 days you have to
request a
new ticket and will need to type in your password. For bash
shell
users, here is a handy function which will automatically renew your
tickets
every hour:
function kinit() { env kinit -r 7d -f ${*-}; while true; do sleep `expr 3600 \* 8`; env kinit -R; done & }
If this function is included in your .bash_profile,
when you use the
kinit command you will instead execute this
function, which
first performs the actual kinit, and then subsequently
loops
and renews the ticket every hour.
Obtaining a Ticket - Jefferson Lab
Because of the networking configuration at JLab, addressless
kerberos
tickets must be used. This requires adding the "-n"
switch to
your kinit command:
qcdi01:~$ kinit -n -fr 7d johndoe@FNAL.GOV ****************************************************************** *** It seems you are about to type your password over the net. *** *** Are you SURE there's no other way to do this ? *** *** If not, is your connection encrypted end-to-end ? *** *** Talk with an expert or review the documentation at *** *** http://www.fnal.gov/docs/strongauth/html/user.html *** ******************************************************************
Password for johndoe@FNAL.GOV: qcdi01:~$ klist Ticket cache: /tmp/krb5cc_5270 Default principal: johndoe@FNAL.GOV
Valid starting Expires Service principal 06/23/05 18:29:00 06/24/05 20:29:00 krbtgt/FNAL.GOV@FNAL.GOV renew until 06/30/05 18:28:52 qcdi01:~$
Note that the JLab kinit will always
issue the warning in the
box above. Be sure that your connection to JLab (or to BNL) uses only
ssh, so
that all communications are encrypted.
You can renew the addressless tickets at JLab the same
way as at BNL.
Logging into Fermilab from Jefferson Lab or from
Brookhaven
Once you have your kerberos ticket, use fssh
to login to the
FNAL head node, lqcd.fnal.gov. This is a kerberized version
of ssh.
Examples follow:
qcdi01:~$ fssh lqcd.fnal.gov Last login: Thu Jun 23 16:20:21 2005 from lqcdp4e2.fnal.gov lqcd:~$
qcdochostb:~$ openssh/fssh johndoe@lqcd.fnal.gov Last login: Thu Jun 23 17:34:57 2005 from g865.jlab.org lqcd:~$
Note that the second example above shows the syntax to
use if your username at
JLab or BNL is not the same as your username at FNAL.
Transfering Files Using Kerberized SSH
Once you have a kerberos ticket at either BNL or JLab,
use
fscp, the kerberized version of scp,
to copy
files to or from FNAL:
qcdi01:~$ fscp test.test johndoe@lqcd.fnal.gov:. test.test 100% |******************************************| 7 00:00 qcdi01:~$ qcdi01:~$ fscp johndoe@lqcd.fnal.gov:test.test . test.test 100% |******************************************| 7 00:00
|