Secure access to remote systems

Author: Adrian Jackson
Posted: 5 Sep 2016 | 10:35

06/09/16: As pointed out by my colleague Stephen in the comments after this post, the way to solve most of these issues is to tunnel the key authentication and therefore bypass the need to have private keys anywhere but on my local machine.  I'm always learning :)

Password vs key

Having to remember a range of passwords for systems that I don't use regularly is hard. 

You can use a password manager, but that only helps if I'm only ever trying to log in from my own laptop. If I have to log in from someone else's machine for any reason then I'd need to know the password.

At any one time I probably have access to something like 15-20 remote systems. I won't be heavily using all of these, I'll probably use 5 or 6 regularly, but I'll have access to a wider range. These systems could be places that people have given me access to to help install software, debug parallel codes, copy data from, run jobs etc.

I spend a lot of my time working from home and, unlike most of EPCC (and the world it seems), I use a Windows laptop for my day to day work. I do have a linux VM installed on it that I use for some things, but in general it's Windows + SSH client + X window server that gets the job done for me day-to-day.

All these systems generally have a different username, because they often have different usename policies (length, required components, can't be shared between projects etc), and in theory they should all have different passwords too. This is where the issues come in.

Remote copies

I often have to copy data between different remote machines.  This requires knowing the passwords for each system, unless I want to copy the data back to my laptop first then copy it out to the other remote machine (which isn't really feasible for large datasets), but remembering 15 or 20 unique passwords is not easy. 

It can be done if you theme your passwords, or use some kinds of system when deciding passwords to easily link a password to a machine (i.e. incorporate something about the machine in the password, and have the rest as a standard string; something like the first, third, and fifth letters from the name of the machine mixed with where the system is based, and some numbers).  But this can lead to insecure passwords.

The obvious solution I hear you shouting is SSH keys. I can create an SSH key that will let me connect to a remote system without having to remember the password to that system. I simply have to remember the password to the SSH key and the job is done, especially as I can use a single SSH key on many systems.

However here is where I have my security dilemma, one that only occured to me recently. If I simply use the SSH key to access remote systems from my laptop it's probably very secure. My private SSH key remains local, and barring me doing something stupid with a phishing email or virus it's unlikely anyone will ever get access to it. But what if I need to access one remote system from another, or copy data from one remote system to another?

Security of keys?

Now I either have to remember the password again, or I need to copy my private key to the remote system so I can use it to facilitate access between them. And suddenly the one thing that can enable access to all systems I have accounts on is no longer under my control. It's sitting on a filesystem on a remote machine. I can (should/must) set the filesystem permissions on that key so that it can't be read by other users, but that does not protect me from people who have higher levels of permissions on those systems.

After all, someone with root or sudo access to the remote system could easily view my key file. It may sounds like paranoid dillusions but I don't know who is administering a lot of the systems I get access to, I don't know how secure they are, whether they have been hacked... We've even had compromised linux boxes within the University where attackers had root privileges for a number of months before the hack was discovered.

So, it may not be a case that there are dark forces administering machines I have access too, but I can't fully trust that my files are secure across all the systems I log in to.  If each system has a separate password/account then the compromising of one system doesn't impact all the other systems I'm using. If I'm using the same SSH key on all systems then if that key is compromised all my accounts could be accessible. Even if I change my SSH passwords on those systems, if the public key is still present then access will be granted.

Guest accounts

Indeed, this is something we have had to watch out for on ARCHER where we have shared access SSH accounts. In general it is forbidden to share your ARCHER account with anyone. However, we have a set of guest accounts that we use for training courses. These aren't shared between people at any one time, but they are re-used between courses. To ensure the security of the account we change the passwords for each course. However, if someone were to place a public SSH key into the account, they would be able to access it later even if the SSH password had changed.

To combat this we have to ensure we fully clean the guest accounts (delete all new files from them, even the ones in hidden directories like the .ssh files) when we close them, after they've been used for a course.


Furthermore, it actually doesn't even require a system to be compromised for people to gain access to a private key. I did a quick unscientific test on a number of systems I have access to, and found a number of private SSH keys (or a file that looked like a private key) that were publically readable on some of the systems that I had the potential to copy should I want to. This was true even on a very secure system I've access to where otherwise SSH access is strictly controlled (only from specific hosts, only newer SSH protocols are allowed, password format is strictly controlled, etc).

If keys are copied and left readable by anyone on the system then compromising, and subsequently using, that key is much easier. I should point out my test didn't verify that the files I could see actually were valid SSH keys, I didn't copy or open them, but given the filenames used there is a good chance that they were. Of course, you can protect against this if you are administering systems: configuring directory permissions to not let others access users' directories by default will stop this, unless users themselves relax those permissions.

If users are careless there is no way to revoke those keys, unlike passwords to the system where, as a system administrator, you have the tools to change passwords for users if you realise there has been a security issue.

Key password

But wait, I hear you say, I've got a password on my SSH key, even if someone takes it they can't use it. True, provided you've not set up passwordless SSH keys, your SSH key should be protected by a password. However, sys admins (should) put effort in to making sure the SSH password you use on a system is not weak (password length, complexity, expiry dates, etc).  No such controls are in place for SSH key passwords.

This could mean that your SSH key password is much easier to crack than your SSH password for a given machine. Furthermore, it is hard to brute-force SSH passwords as hosts generally have controls over how many failed login attempts are allowed for an account. No such control exists if you have someone's SSH key, you can run a brute force dictionary attack against the password at your leisure.

What to do?

This does not mean SSH keys are necessarily insecure. However, it also doesn't mean they are necessarily safe. I use SSH keys sparingly, on systems I'm regularly accessing, and I make my SSH key as long and varied as my passwords. I only copy my private SSH key to local systems. This still isn't 100% safe, but it does facilitate my work, and there has to be a balance between security and usability.

It also doesn't mean using password and no SSH keys is necessarily secure either. Having different passwords across a range of systems is also challenging and many people have the same password everywhere, or similar passwords. I try to use a variant on each system, but with the variability linked to the systems (not necessarily around the name of the systems, but why I'm using them), but it's not a fool-proof system and can lead to weaker passwords across the systems than if I was using a unique password everywhere.


However, these are all just my musings. I'd be keen to hear feedback from others on security issues and what we should be using for accessing machines? Am I making stupid assumptions in the above? Have I missed a crucial detail?  Are there other systems of access control I could be using?  What do you use to manage access to a set of machine? Is everything better on a Mac (:))? Let me know...


Adrian Jackson, EPCC
Adrian on Twitter: @adrianjhpc

Image: Alexander Shirokov, iStock




Stephen Booth's picture

Adrian, SSH is designed to counter the type of problem you are worrying about but you have to use it correctly.
The key rule to never let your private key leave the machines you trust like your laptop. Instead of installing a key-file everywhere use the -A flag each time you log into a new machine.
With this flag instead of using a key installed on the machine you are leaving the authentication session is forwarded back down the chain to the machine you started on. The authentication is crptographic so the private key never leaves your trusted machine.
In principal if one of the machines in the chain is compromised (untrustworthy) then any text you type on that machine or machines further down the chain might be capturable. Using normal passwords is therefore only as secure as the weakest link in the chain.
Use an ssh-agent (like pageant) so that the ssh-key decryption password only ever gets typed into a pop-up window on your trusted machine.
If you have a private key on a machine and you unlock that key then the system administrators of the machine you unlock the key on can pull the unlocked key out of system memory. The step you are missing is that it is not necessary for the private key to ever leave machines you can trust.

This does not help if you want to borrow a machine you don't entirely trust. There is no solution in this case. If you don't trust the machine you are
sitting at then don't use it there is no other way to stay secure.

Thanks Stephen! I've never been able to get key tunnelling to work satisfactorily from the windows SSH clients I've been using, but I should some more effort into fixing that.

Blog Archive