When using ssh to connect to a server the ssh client checks the identity of the server by using one or more so called known_hosts files. In these files the public keys are stored on a host name or IP address basis. This article shows you how you can make proper use of this mechanism.

Instructions

Open the ~/.ssh/config file and add the following options:

UserKnownHostsFile ~/.ssh/known_hosts ~/.ssh/known_hosts_fixed
HashKnownHosts no
CheckHostIP no

By setting UserKnownHostsFile to ~/.ssh/known_hosts ~/.ssh/known_hosts_fixed we achieve two things. First, ssh use will use the specified two files when searching for known public keys of servers. Second, when connecting to a new server for the first time ssh will ask us if we want to add the public key of the server to the list of known hosts. If we choose yes ssh will add the key to the ~/.ssh/known_hosts file. It won’t touch the second file. The ~/.ssh/known_hosts_fixed is the file which we will maintain manually.

We also set both HashKnownHosts and CheckHostIP to no. This way our list of known hosts will require only one entry per domain name and the list will be easily maintainable by humans. If you want to read more about the HashKnownHosts and the CheckHostIP options see the next section.

Then delete your current known_hosts file in the ~/.ssh/ folder. If you want, do not forget to make a backup of it.

After this fetch the public keys of all the servers which you are connecting to frequently. You can achieve this by using the ssh-keyscan command:

user@laptop:~/tmp$ ssh-keyscan github.com
github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==
user@laptop:~/tmp$

ssh-keyscan may return multiple public keys. In this case I recommend picking one of the public keys in the following order:

1. rsa (highest priority)
2. ecdsa
3. ed25519 (lowest priority)

If you have shell access to the server you can also find the public keys in the /etc/ssh directory:

user@server:/etc/ssh$ ll
<...>
-rw-------  1 root root  668 Aug 18  2017 ssh_host_dsa_key
-rw-r--r--  1 root root  613 Aug 18  2017 ssh_host_dsa_key.pub
-rw-------  1 root root  227 Aug 18  2017 ssh_host_ecdsa_key
-rw-r--r--  1 root root  185 Aug 18  2017 ssh_host_ecdsa_key.pub
-rw-------  1 root root  419 Aug 18  2017 ssh_host_ed25519_key
-rw-r--r--  1 root root  105 Aug 18  2017 ssh_host_ed25519_key.pub
-rw-------  1 root root 1,7K Aug 18  2017 ssh_host_rsa_key
-rw-r--r--  1 root root  405 Aug 18  2017 ssh_host_rsa_key.pub
user@server:/etc/ssh$

Store the fetched public keys in the ~/.ssh/known_hosts_fixed file (one public key per line). When reading the public key directly from the file system of the server you will have to add the hostname prefix (e.g. github.com) on your own.

Then you are good to go. Ideally you would now implement a mechanism to sync the ~/.ssh/config and ~/.ssh/known_hosts_fixed files across the devices you are using. I use Ansible for this task.

Details about HashKnownHosts and CheckHostIP

HashKnownHosts indicates that ssh should hash host names and addresses when they are added to e.g. ~/.ssh/known_hosts. This helps to prevent information leaks about internal hosts names if the contents of this file are disclosed:

user@laptop:~/tmp$ git clone git@github.com:golang/website.git
Cloning into 'website'...
The authenticity of host 'github.com (140.82.121.4)' can't be established.
RSA key fingerprint is SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'github.com,140.82.121.4' (RSA) to the list of known hosts.
<...>
user@laptop:~/tmp$ 

When executing this command ssh will add the following two entries to ~/.ssh/known_hosts when connecting to github.com for the first time:

|1|XbQtt3GpqVvirItg5N3htKC1Zs8=|ns1qjyRmV5uj72aBhj3s/pOoFII= ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==
|1|G3CWe+lhdS9SD8CUcAvGrjoz51I=|frft9KUf+CAhU7yW0DDr1nEBF0o= ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==

Because we want the known_hosts to be maintainable by humans and we do not have any problems if the host names of e.g. github.com are disclosed to the public we set HashKnownHosts to no.

By the way. Why did ssh add two lines to the known_hosts file anyway? This is where we have to take a look at the CheckHostIP option. If set to yes, ssh will additionally check the servers IP address in the list of known hosts. This allows it to detect if a public host key changed due to DNS spoofing and will add addresses of destination hosts to ~/.ssh/known_hosts in the process. Also when HashKnownHosts is set to yes a public key can only be stored for exactly one host name or IP address.

Furthermore sites like github.com also make use of DNS load balancing. Suppose CheckHostIP is set to yes. When connecting to github.com for the first time ssh would ask us to add the public key of github.com and 140.82.121.4 to the known_hosts file. When connecting a second time the DNS might resolve github.com to another IP address (e.g. 140.82.121.4). ssh would then ask us if it should add another public key for 140.82.121.4. Or it might even report a problem (I still have to find out ;-).

Therefore we set both HashKnownHosts and CheckHostIP to no. This way our list of known hosts will require only one entry per domain name and the list will be easily maintainable by humans.

Further documentation

• Details about the known_hosts file format are documented in the man page of sshd ($ man 8 sshd). This man page is part of the OpenSSH server package under most Linux distributions.
• $ man 5 ssh_config