Troubleshooting Kerberos and SSH problems
1. Kerberos kinit Problems
2. SSH Problems
1. Kerberos kinit Problems
Error message:
kinit(v5):
Cannot find KDC for requested realm while getting initial
credentials
Problem: /etc/krb5.conf
file doesn't contain .FNAL.GOV information
Solutions:
1. Replace /etc/krb5.conf
with your OS-specific Fermilab-supplied version of krb5.conf
2. Modify /etc/krb5.conf,
adding
Fermi-specific stanzas as instructed here.
3. If user does not have
permission to modify /etc/krb5.conf,
copy
Fermilab-supplied version into home area, and do
export
KRB5_CONFIG=$HOME/krb5.conf
to tell all Kerberos
commands to use the user's copy of krb5.conf.
Related
problem:
On Macintosh computers
(OS-X operating system), Kerberos is installed on
all recent versions.
However, there are two locations and names for krb5.conf,
/etc/krb5.conf
and
/Library/Preferences/edu.mit.Kerberos
(Note: the file in /Library is
named edu.mit.Kerberos,
not
krb5.conf.)
Either will work, but you
should only have one. Updated 10/11/2012: Several Mac OS-X 10.6.8 users have reported that only the /etc/krb5.conf file worked for them.
Error message:
kinit: Unable to acquire credentials for 'user@FNAL.GOV': Cannot contact any KDC for realm 'FNAL.GOV'
Problem:
You are behind a firewall or are using an internet connection which has a "NAT" (Network Address Translation), such as on a home wireless router
Solution Steps:
1. Ping one of the Fermilab Kerberos Authentication servers (such as krb-fnal-1.fnal.gov) to make sure you can reach the server at the other end. If successful move to step 2.
2. Request an addressless Kerberos ticket as follows:
kinit -a
If you do
klist -a
you should see as the last line
Addresses: (none)
Error message:
kinit:
Preauthentication
failed while
getting initial credentials
Problem: kinit fails with
preauthentication error
Solutions:
1. Usually the problem is simply that you have typed in your kerberos
password incorrectly. Try again.
2. If you have lost your kerberos password, call the Fermilab Service Desk,
630-840-2345, during business hours to have the password reset.
3. Occassionally this is not a password problem, but a problem with your
system's clock. Make sure that the date
command returns a time
correct to within 5 minutes.
Error message:
kinit:
krb5_get_init_creds:
Too large
time skew
Problem: kinit fails with time skew
message
Solution:
1. Your system clock must be within 5 minutes of the correct value.
Error message:
kinit:
KDC
has no support for encryption
type while getting initial credentials
Problem: kinit fails with complaint
about encryption type
Solution:
1. This problem appears on recent Ubuntu and related Linux
distributions.
To fix, edit /etc/krb5.conf
file, and in the [libdefaults] section
add
allow_weak_crypto
= true
Error message:
kinit:
Client
not found in Kerberos
database while getting initial credentials
Problem: kinit fails with database
complaint
Solutions:
1. Your kerberos principal may differ from your username on your local
system. Use kinit
username@FNAL.GOV where username is your
Fermilab kerberos principle. kinit by
itself will use your local
username.
2. Your Fermilab ID or visitor ID has expired. Please see the renewal
instructions at Accounts
and
Passwords webpage.
Error message:
kinit:
Client's
entry in database has
expired
Problem: kinit fails because of an
expired password
Solutions:
1. You must change your kerberos password once a year with the kpasswd
command (the same command you used to change your initial kerberos
password). You can try to change your password, even if it is
expired, by using kpasswd on
your local machine. More detailed instructions are available here.
2. You can call the Fermilab Service
Desk, 630-840-2345, and request that
they reset your kerberos password.
Error message:
kinit: Password incorrect
Problem: If you are sure your Kerberos password is correct but you are on a MAC OS 10.10 (Yosemite) kinit will fail because the Kerberos pass phrase is DES encoded, which Yosemite no longer accepts.
Solutions:
You will need to reset your Kerberos password as follows:
1. If you have access to a non-Yosemite machine with Kerberos client software installed, do kinit to obtain a Kerberos ticket and then SSH to one of our head nodes, for example ds1.fnal.gov. Once there do a kpasswd to change your Kerberos password.
Alternatively, you may call the Fermilab Service Desk at 630-840-2345, and request that they reset your kerberos password.
2. Update the krb5.conf file on your Yosemite machine wih the latest version from http://computing.fnal.gov/authentication/krb5conf/OSX/krb5.conf.
3. On your Yosemite machine, go ahead and attempt kinit and SSH, which should now work.
2. SSH Problems
Error message:
ssh_dispatch_run_fatal: Connection to 131.225.202.32: unexpected internal error
Problem:
You may have just upgraded to the new 'El Capitan' Mac OS X version or changed the SSH options on your client.
Solution Steps:
1. Try to ssh -o GSSAPIKeyExchange=no to one of our servers (such as pi0.fnal.gov). If it works then
2. Add GSSAPIKeyExchange no to your SSH client config file (eg /etc/ssh/ssh_config).
Error message:
permission
denied
OR
CryptoCard RB-1
Press
ENTER
and compare this challenge to the one on your display
.
. . .
ssh login failures will be
indicated by a permission denied
message.
If none of the solutions below fixes your problem please
email the output of the command ssh -vvv
pi0.fnal.gov to
lqcd-admin@fnal.gov
for further assistance.
Problem: Kerberos client and SSH using different credential cache file locations
Solution:
We have mostly encountered this on MAC 10.9.x versions where Kerberos clients are installed from two different sources. In such a situation Kerberos client binaries end up in /opt/local/bin and in /usr/bin. Use the Kerberos client ( kinit) installed in /usr/bin
to obtain a Kerberos ticket. Also make sure there is a subdirectory .ssh in your home directory. Make sure the subdirectory has a file named config with the following lines:
Host *.fnal.gov
GSSAPIAuthentication yes
GSSAPIDelegateCredentials yes
GSSAPITrustDNS yes
Problem: Not having a kerberos
ticket granting ticket (TGT), or
having an expired TGT
Solution:
Verify with the klist -f
command that you have a ticket. If you
don't have a ticket, or have an expired ticket, get a new ticket
with kinit
as follows:
lqcdp4ee:~$
kinit
-fr 7d username@FNAL.GOV
lqcdp4ee:~$
klist
-f
Ticket cache:
/tmp/krb5cc_1234
Default
principal: username@FNAL.GOV
Valid
starting
Expires
Service principal
08/17/12
09:31:16 08/18/12 11:31:16 krbtgt/FNAL.GOV@FNAL.GOV
renew until
08/24/12 09:31:09, Flags: FRIA
Normal output, indicating that a forwardable, renewable, ticket exists.
Check the expiration time - if the current time is past the expiration,
login attempts will fail.
lqcdp4ee:~$
klist -f
klist:
No credentials cache file found (ticket cache /tmp/krb5cc_5598)
If you see the above message you do not have a Kerberos ticket.
Use kinit
to get a ticket before attempting to login.
Kerberos tickets expire after 24 hours. If you include the
-r
7d switch on your kinit
command line, you will receive
a renewable ticket. Renewable tickets may be renewed by
typing kinit
-R before they expire at the end of any 24
hour period. Tickets are renewable for up to the period
specified in the -r switch,
to a maximum of 7 days.
Another useful switch to kinit is -f, which
asks for a
forwardable ticket. If you have a forwardable ticket, once
you login to a Fermilab machine, say pi0.fnal.gov,
you
can
then login from pi0 to
another machine, say ds1.fnal.gov,
without doing a new kinit. It
is in general a bad idea to
use kinit
on any machine but your local system, as your
password may be captured as it traverses the internet. The
only time typing a kinit
password is safe on a remote machine
is when you are using an encrypted connection, like with ssh.
Problem: Not having an account on
the target machine, or having
an account on the target machine under a different username.
Solution:
A permission
denied error will occur if you do not have an account
on the target machine, or if your username on the target machine
differs from your username on your local machine. Try
ssh
username@pi0.fnal.gov
or
ssh
-l username pi0.fnal.gov
where username is your Fermilab username (the same name that you
used in your kinit
command). If this fails, send e-mail to
lqcd-admin@fnal.gov
and ask that the administrators verify that you have a valid
account on the Fermilab lattice QCD systems.
Problem: Using an internet
connection which has a "NAT" (Network
Address Translation), such as on a home wireless router
Solution:
Nearly all home routers, wired or wireless, have a "NAT" function,
which results in your local system having a different local network
address than what is presented to remote machines. This allows you
to have multiple local machines and only one external IP address.
You local addresses will generally be something like 192.168.X.Y,
or 10.X.Y.Z, when a NAT is present.
With a NAT, your ssh logins may fail with Incorrect net
address.
To fix this, use "addressless" tickets. First, use kdestroy
to delete your current ticket. Then, use kinit with -a, -A,
or -n
to request an addressless ticket. The switch required
varies with kerberos versions, so use man kinit
on your local
system to determine which of these three switches to use.
For Mac OS users, please be
aware that between versions 10.5 and
10.6 Apple changed the switch for addressless tickets from
-A
to -a.
So
if you have recently upgraded from Leopard (10.5)
to Snow Leopard (10.6) and are still using -A you will
need to
change to -a.
Also,
the default behaviour on Mac OS is to supply
addressless tickets, so you should also be able to simply drop the
-A
or -a
switch entirely.
Problem: Using an ssh client
which does not have Kerberos
authentication enabled
Solution:
Some versions of ssh will not attempt to perform kerberos
authentication. In this case, you will receive a permission
denied
error. To enable kerberos
authentication, try the following -o switch:
ssh
-o "GSSAPIAuthentication yes" username@pi0.fnal.gov
The quotation marks are required. If this form of ssh succeeds,
you can configure your local system to always attempt to use
kerberos authentication by editing either $HOME/.ssh/config
or
/etc/ssh/ssh_config
and adding these lines:
Host
*.fnal.gov
GSSAPIAuthentication
yes
GSSAPIDelegateCredentials
yes
The GSSAPIDelegateCredentials
line is necessary if you want to
use X-windows clients on the remote (Fermilab) system.
Note that you may also need a -X or -Y switch
on your ssh command
to enable X forwarding.
Problem: Setting up an ssh tunnel to access kaon1 via a single hop. (NOTE: kaon1 has been decomissioned and no longer being supported.)
Solution:
As with other facilities that use ssh gateway nodes, you may find it
convenient to use ssh tunneling to access kaon1 via a single hop. An example
command to use on your local machine to setup a tunnel is:
ssh -X -f -N -L 2222:kaon1.fnal.gov:22 username@lqcd.fnal.gov
where "username" is your Fermilab username. After this command returns, you
can login to kaon1 by using
ssh -p 2222 username@localhost
Note that "2222" is the port number of the end of the tunnel on your local
machine. You can use any large number (> 1000) for this port.
The syntax is slightly different for the scp command:
scp -P 2222 username@localhost:file file
scp -P 2222 file username@localhost:file
If you get a "Permission denied" error message then follow the steps listed below:
Step1: Generate a ssh public private key pair if you do not have one using the
ssh-keygen command which will save the keys at $HOME/.ssh/ on the machine
where the ssh-keygen command is executed. If you already have a private public ssh key pair then go to step 2.
Step2: Append the contents of the public ssh key file ($HOME/.ssh/id_rsa.pub and
$HOME/.ssh/id_dsa.pub and $HOME/.ssh/identity.pub) to
the $HOME/.ssh/authorized_keys file on kaon1.
See also
//fnal/transfer.html
for a description of the "tunnel.pl" tool which will setup ssh tunnels for
you.
If you still have any questions or issues with kaon please
send email to lqcd-admin@fnal.gov.
|
