Transfering Files Between US Lattice QCD Sites

The three labs hosting US Lattice QCD hardware - Fermilab (FNAL), Jefferson Lab (JLab), and Brookhaven (BNL) - have different computer security configurations. FNAL uses Kerberos, and both JLab and BNL use ssh-based firewalls. Because of these differences, users must employ different sets of commands to access these facilities and to transfer files, depending upon the starting location.

At JLab and BNL, users must first login to one of several site firewall nodes using ssh, and then login via ssh to the desired head node (qcdochosta or qcdochostb at BNL, qcdi01 or qcdi02 at JLab). Transfering files across this firewall into either facility requires two "hops" - first, files must be scp'd to the firewall node, second scp'd from the firewall node to the target head node. A much superior alternative is to use ssh tunneling, which relies on ssh processes running on the user's node and on the firewall node to move data directly to or from the target head node. This is discussed further below.

To login to FNAL computers, or to move files to or from FNAL machines, users must employ Kerberos software. First, a user must obtain a Kerberos ticket. Once a ticket is held, kerberized versions of telnet, rsh, rcp, rlogin, ssh, and scp may be used to access FNAL systems. This is discussed further below.

Fermilab provides data transfer between its Lattice QCD Lustre filesystem through Globus Online. Please follow the simple steps listed here.

SSH Tunneling

Both BNL and JLab rely on ssh firewalls for cyber security. The ssh utilities include a very powerful port forwarding facility which greatly simplifies accessing the BNL and JLab lattice QCD head nodes through these firewalls. The ssh command syntax which is required for port forwarding is cumbersome and very difficult to memorize. Fortunately, Steve Adler at BNL has implemented a set of Perl scripts (tunnel.pl) which greatly simplify port forwarding.

The tunnel.pl is installed in /usr/local/bin/ on the head nodes at BNL, JLab, and FNAL. Rather than invoking tunnel.pl directly, users should use these commands:

• tssh - open a tunnel if necessary, and login via ssh to the requested head node
• tscp - open a tunnel if necessary, and copy a file via ssh to/from the requested head node
• tclose - close any open tunnels

The tunnel.pl script relies on a configuration file which is unique to each user, $HOME/.tunnel/config. The syntax of the configuration file is explained in the BNL documentation (see the notes in "Installing the tunnel script") and in the main tunnel.pl documentation (see "How do I configure tunnel.pl?").

Here is a sample configuration file from lqcd.fnal.gov:

qcdochostb.qcdoc.bnl.gov:2002:ssh.qcdoc.bnl.gov:dholmgre:localbnl:qcdochostb
qcdi01.jlab.org:2003:jlab.org:johndoe:localjlab1:qcdi01
qcdi02.jlab.org:2004:jlab.org:johndoe:localjlab2:qcdi02

The fields in the configuration file, separated by colons, are as follows:

1. Fully qualified domain name of target head node.
2. TCP port number to use for the local end of tunnel. Note - this port number must be unique for all users and all target head nodes. If a non-unique number is used, a tunnel command may fail with a "port in use" error message. The port numbers specified should be greater than 1023.
3. The fully qualified domain name of the ssh firewall node.
4. The username to use on the ssh firewall node. Note in this example that different usernames are used at BNL and JLab.
5. The local network name for the machine running the tunnel scripts. Note - to avoid ssh errors similar to "REMOTE HOST IDENTIFICATION HAS CHANGED!", you should use unique local network names for each target head node. This requires the system administrator to add aliases for "localhost" to /etc/hosts. On lqcd.fnal.gov, localbnl, localjlab1, and localjlab2 are available aliases for localhost. Additional aliases can be added on request. The local network name is used by tssh and tscp to connect to the "end" of the ssh tunnel connected to the local machine.
6. An alias for the target head node. This allows you to use a name for the destination head node which is easier to type than the fully qualified domain name.

Examples

In all of the examples shown below, we have assumed that the user has first loaded his ssh private key via ssh-agent $SHELL and ssh-add. Otherwise, on each command invocation there will be an additional prompt for ssh passphrase, or for the password on the local node.

Logging into a Remote Head Node

In this example, a tunnel has not yet been opened on the JLab firewall. So, in response to tssh the user will be asked for his password for the JLab firewall. If successful, tssh will start an ssh process on the firewall node, then invoke ssh again to open a shell on qcdi02:

lqcd:~$ tssh qcdi02
Firing up tunnel to qcdi02.jlab.org through jlab.org using port 2004
ssh -e none -f -N -L 2004:qcdi02.jlab.org:22 johndoe@jlab.org
johndoe@jlab.org's password: 
[johndoe@qcdi02 ~]$ 

Next, we exit from the shell on qcdi02, returning to lqcd. tssh to qcdi02 is invoked again. This time, the tunnel already exists, so no password is required:

[johndoe@qcdi02 ~]$ exit
logout
Connection to localjlab2 closed.
lqcd:~$ tssh qcdi02
[johndoe@qcdi02 ~]$ 

On the BNL machine, according to the $HOME/.tunnel/config file, a different username is required. To login to a BNL head node, you must explicitly give the remote username:

lqcd:~$ tssh dholmgre@qcdochostb
Firing up tunnel to qcdochostb.qcdoc.bnl.gov through ssh.qcdoc.bnl.gov using port 2002
ssh -e none -f -N -L 2002:qcdochostb.qcdoc.bnl.gov:22 dholmgre@ssh.qcdoc.bnl.gov
dholmgre@ssh.qcdoc.bnl.gov's password: 
-bash-2.05b$ hostname
qcdochostb

The ssh port forwarding tunnels will continue to run on the BNL and JLab firewall nodes. When you are finished accessing these nodes, you should close the tunnels with tclose:

lqcd:~$ tclose
Closing tunnel on port 2004 going to qcdi02.jlab.org through jlab.org
Closing tunnel on port 2002 going to qcdochostb.qcdoc.bnl.gov through ssh.qcdoc.bnl.gov
lqcd:~$

Notice that tclose will close all existing tunnels.

Copying Files

In these examples, we assume that a tunnel has already been established to the target head nodes.

To copy a file from the local node to the target head node, use tscp:

lqcd:~$ tscp test.test qcdi02:.
test.test            100% |******************************************|     7       00:00    

For BNL with our $HOME/.tunnel/config file, you must give the username:

lqcd:~$ tscp test.test dholmgre@qcdochostb:.
test.test            100% |******************************************|     7       00:00    

To copy files from the target head nodes, reverse the arguments:

lqcd:~$ tscp qcdi02:test.test .
test.test            100% |******************************************|     7       00:00    
lqcd:~$ tscp dholmgre@qcdochostb:test.test .
test.test            100% |******************************************|     7       00:00    

Note that tscp takes standard scp switches. So, to recursively copy a directory, you could use:

lqcd:~$ tscp -r test.dir qcdi02:.
file.a               100% |******************************************|     7       00:00    
file.b               100% |******************************************|     7       00:00    

Obtaining a Ticket - Brookhaven

From BNL, in order to login to the FNAL headnode, lqcd.fnal.gov, or to transfer files into or out of lqcd, you first must obtain a kerberos ticket. To do this, use the kinit command:

qcdochostb:~$ kinit -fr 7d johndoe@FNAL.GOV
Password for johndoe@FNAL.GOV: 
qcdochostb:~$ 
qcdochostb:~$ klist -f
Ticket cache:  FILE:/var/krb5/security/creds/krb5cc_8386
Default principal:  johndoe@FNAL.GOV

Valid starting     Expires            Service principal
06/23/12 18:16:00  06/24/12 18:15:54  krbtgt/FNAL.GOV@FNAL.GOV
        Renew until 06/30/12 18:15:54, Flags:  FRIA
qcdochostb:~$ 

Here, kinit has been used to obtain a forwardable ticket which is renewable for 7 days. Note that we have explicitly given the principal name, in this case, johndoe@FNAL.GOV. If you do not give the principal, instead username@FNAL.GOV will be used. Users who have different login names at the three sites must include their principal in the kinit command.

A forwardable ticket may be used, once logged into lqcd.fnal.gov, to login to another FNAL system.

Kerberos tickets are valid for 24 hours. Renewable tickets may be renewed for an additional 24 hours at any time before expiration by using

    kinit -R

Note that renewing a ticket does not require you to type in your password. The "-r 7d" switch in this example requests a ticket which may be renewed in this manner for up to 7 days. After 7 days you have to request a new ticket and will need to type in your password. For bash shell users, here is a handy function which will automatically renew your tickets every hour:

function kinit() {
    env kinit -r 7d -f ${*-};
    while true; do
        sleep `expr 3600 \* 8`;
        env kinit -R;
    done &
}

If this function is included in your .bash_profile, when you use the kinit command you will instead execute this function, which first performs the actual kinit, and then subsequently loops and renews the ticket every hour.

Obtaining a Ticket - Jefferson Lab

Because of the networking configuration at JLab, addressless kerberos tickets must be used. This requires adding the "-n" switch to your kinit command:

qcdi01:~$ kinit -n -fr 7d johndoe@FNAL.GOV
******************************************************************
*** It seems you are about to type your password over the net. ***
*** Are you SURE there's no other way to do this ?             ***
*** If not, is your connection encrypted end-to-end ?          ***
*** Talk with an expert or review the documentation at         ***
*** http://www.fnal.gov/docs/strongauth/html/user.html         ***
******************************************************************

Password for johndoe@FNAL.GOV: 
qcdi01:~$ klist
Ticket cache: /tmp/krb5cc_5270
Default principal: johndoe@FNAL.GOV

Valid starting     Expires            Service principal
06/23/05 18:29:00  06/24/05 20:29:00  krbtgt/FNAL.GOV@FNAL.GOV
        renew until 06/30/05 18:28:52
qcdi01:~$ 

Note that the JLab kinit will always issue the warning in the box above. Be sure that your connection to JLab (or to BNL) uses only ssh, so that all communications are encrypted.

You can renew the addressless tickets at JLab the same way as at BNL.

Logging into Fermilab from Jefferson Lab or from Brookhaven

Once you have your kerberos ticket, use fssh to login to the FNAL head node, lqcd.fnal.gov. This is a kerberized version of ssh. Examples follow:

qcdi01:~$ fssh lqcd.fnal.gov
Last login: Thu Jun 23 16:20:21 2005 from lqcdp4e2.fnal.gov
lqcd:~$


qcdochostb:~$ openssh/fssh johndoe@lqcd.fnal.gov
Last login: Thu Jun 23 17:34:57 2005 from g865.jlab.org
lqcd:~$  

Note that the second example above shows the syntax to use if your username at JLab or BNL is not the same as your username at FNAL.

Transfering Files Using Kerberized SSH

Once you have a kerberos ticket at either BNL or JLab, use fscp, the kerberized version of scp, to copy files to or from FNAL:

qcdi01:~$ fscp test.test johndoe@lqcd.fnal.gov:.
test.test            100% |******************************************|     7       00:00    
qcdi01:~$ 
qcdi01:~$ fscp johndoe@lqcd.fnal.gov:test.test .
test.test            100% |******************************************|     7       00:00