I remember the good times, when I had a single password for all (the two?) services I used. Then came a phase of managing passwords on sticky notes stuck below the keyboard (much more secure than stuck to the monitor!), until I migrated to one of the first note-taking apps and had passwords written in clear like on the sticky notes but copy/pasteable. Those were the days!
Then came the first “real” password manager which for me was 1Password. It worked, I synced with a local store (i.e. sans Internet) until the company decided to convert my “paid for life” (which never exists) into a subscription model. I spent days thinking and rethinking and settled for Enpass which I’ve used since on macOS and iOS. It’s okay, and does what I want from it, but its enshitification is in progress. There are others, lots of others, and I looked closely at KeePassXC, but can’t seem to make the jump.
One of the password managers I kept an eye upon for a while was pass, a.k.a. passwordstore. It must have been a dozen years ago that I fell for the cool little tool which lives in my terminal, but as there wasn’t an iOS pendant, I kept pushing it aside. Interestingly, for years now, I’ve mentioned it in my Ansible trainings when I very briefly explain the passwordstore lookup plugin when going through a short list of lookups.
Moving forward, I stumbled across a program which piqued my interest yesterday: Pass for iOS, and decided to take a closer look to see whether the combination of that plus pass might tickle me again.
pass the standard unix password manager
pass calls itself the standard unix password manager, and with that it is in line with ed, the standard unix text editor. Both have no frills and no AI.
In pass individual passwords live in gpg-encrypted files whose name is the title of the site or resource the password is for. These files can be organized into hierarchies, for instance:
All these files live in a directory (defaults to ~/.password-store) and are encrypted with one or more gpg keys. The pass command can insert a new entry, generate a password, mv (i.e. rename) or rm (delete) passwords, etc. Pass can optionally manage its store as a Git repository which in turn can be cloned or pushed from/to elsewhere, which is what I’m doing with a forgejo repository, so my workflow would typically be:
$pass insert testing/t2
Enter password for testing/t2:
Retype password for testing/t2:
[trunk c6b675d] Add given password for testing/t2 to store.
1 file changed, 0 insertions(+), 0 deletions(-)
create mode 100644 testing/t2.gpg
$pass git push
Enumerating objects: 10, done.
Counting objects: 100% (10/10), done.
Delta compression using up to 12 threads
Compressing objects: 100% (7/7), done.
Writing objects: 100% (8/8), 1.45 KiB | 1.45 MiB/s, done.
Total 8 (delta 3), reused 0 (delta 0), pack-reused 0 (from 0)
To ssh://forgejo.example.org/jp/password-store.git
d677b28..c6b675d trunk ->trunk
With pass I can edit entries, use multiline passwords (the first line is the password, the rest is just text, but see below for passforios), view a password or copy it to the clipboard. Here I want to print the password to stdout and have to first authenticate to the gpg-agent, with pinentry set to tty so that we see its output here:
$pass testing/t2
Please enter the passphrase to unlock the OpenPGP secret key:
"PW Store (password-store) <passwordstore@example.org>"
255-bit ECDH key, ID F8BE1A46,
created 2026-08-11 (main key ID 3BE34DFA).
Passphrase:
bla
pass can find names of passwords inside the tree, and it will grep (i.e. search) a string within the store, decrypting each file in turn which can obviously take a while. It will also find strings within passwords.
$pass find hop
Search Terms: hop
└── testing
└── shoppe
$pass grep colour
testing/shoppe:
colour: yellow
Documentation for pass is very good and simple to follow with more details in its manual page including an example how to set up Git for it.
customization
I have created a small script which permits me to centralize pass’ directory and a gnupg home specifically for pass. It lives in my $HOME/bin and looks like this:
iOS is the operating system I typically carry along in the pocket of my pants, so if I want to use pass I need something which can access the same password store.
I’ll admit to have been skeptical about Pass for iOS but am very pleasantly surprised at how smooth the app appears to work. Setting up the repository took longer than I imagined, but the problem was the person tapping on glass: I forgot to change the branch name from the app’s setting of master to the actual name I use.
Using Universal Clipboard I quickly and securely copied the GPG public and private keys over to the phone and pasted them into the corresponding dialog. Other mechanisms for adding the keys include downloading from a URL, iTunes File Sharing, or tapping on a Yubikey. On first use, I’m asked for the password of the PGP key and the device can optionally store that for me like it can do for the password of the Git repository URL, which avoids me having to retype it at every use.
On the device I can edit and create passwords, generate (random or XKCD style) new ones, delete them, and a swipe down on a directory of passwords has the device pull/push to the Git repository. Nice detail: a red badge indicates that a sync has to be performed because I changed an entry on the device.
Passwords edited on the device are marked appropriately in Git’s log.
$pass git log
commit 73a1c0a66112e09f318e4497c3b3e6ba0f6e0d01 (HEAD ->trunk, origin/trunk)Author: passforios <jp-jPhone@example.org>Date: Tue Aug 11 10:35:17 2026 +0200
Edit password for testing/t2.gpg using Pass for iOS.
commit c6b675de1fee70a5a597a19c9585fd5b789e4586
Author: Jan-Piet Mens <jp@example.org>Date: Tue Aug 11 10:17:47 2026 +0200
Add given password for testing/t2 to store.
...
I mentioned multiline passwords earlier, and the testing/shoppe password is one:
Note how Pass for iOS formats that in the screenshot at the top of the page. I like. Yes, and those are YAML-like multiline fields with either | (keep newlines) or > (squash newlines).
Earlier I mentioned an Ansible lookup plugin which uses pass as one of its backends.
$ansible localhost -m debug -amsg="{{ lookup('passwordstore', 'testing/shoppe') }}"Please enter the passphrase to unlock the OpenPGP secret key:
"PW Store (password-store) <passwordstore@example.org>"
255-bit ECDH key, ID F8BE1A46,
created 2026-08-11 (main key ID 3BE34DFA).
Passphrase:
localhost | SUCCESS =>{ "msg": "eeTiebaip0diHahd,aiYithoh"
}
The lookup plugin can also generate a new password if the requested key isn’t there:
$pass ansible/l1
Error: ansible/l1 is not in the password store.
$ansible localhost -m debug -amsg="{{ lookup('passwordstore', 'ansible/l1', missing='create') }}"localhost | SUCCESS =>{ "msg": "Rm8PxmtETPWsXKix"
}
$pass ansible/l1
Rm8PxmtETPWsXKix
lookup_pass: First generated by ansible on 11/08/2026 12:01:15
We’d also be able to use pass to store a secret for Ansible Vault, leveraging vault_password_file in an ansible.cfg or its corresponding environment variable. Let’s generate a new password:
$pass generate -n ansible/vaultsecret
[trunk 2c8a9f4] Add generated password for ansible/vaultsecret.
1 file changed, 0 insertions(+), 0 deletions(-)
create mode 100644 ansible/vaultsecret.gpg
The generated password for ansible/vaultsecret is:
VxpWJRFTIEY2A8ImiTKJ2EPS4
Now we configure vault_password_file in an ansible.cfg with an executable program to have Ansible Vault use our pass password for creating the AES256 key it will use to encrypt/decrypt its data:
I’ve not yet decided whether or not I will migrate most of my passwords to pass (it would entail quite a bit of work), but the combination of pass and Pass for iOS does appeal to me, in spite of GnuPG:
CLI for the win.
Versioning. (pass git log -p to see what the previous password was – yes, I have needed that in the past)
After setting up gpg if I haven’t already done so, pass handles the underlying encryption and decryption transparently, which is a big win.
Sync (pull/push) from iOS, the latter with a nice interface.
Shell script with few dependencies.
There’s an import extension which supports over 60 different password managers. A test with a small KeePassXC file was disappointing, as the extension imported one entry into several files, one for each line of a KeePassXC note, for instance. Quite undesireable!
However, a test with several dozen passwords exported as JSON from Enpass was impressively good, the extension creating multi-line passwords in pass corresponding to the (lowercased) field names in Enpass, the latter supporting arbitrary fields in a password entry; this is both very flexible as well as awful when moving to a new system. ;-) Attachments in Enpass entries are also imported as base64 blobs into the pass entry.
gopass, the “slightly more awesome standard unix password manager for teams”, and its source code. Also uses GPG and is said to be a drop-in replacement for pass, and it can be configured to use age.
A few years ago I began using Emacs Org-mode and reveal.js to generate slide decks I use in trainings, and I’ve been pleased with the results. (I believe it was Carsten who got me hooked.)
It took me a bit to get the settings just the way I wanted to have them, but since then it’s been a matter of typing make in a command line to generate the HTML from the .org file using this in my Makefile, and the resulting *.html is what I open in a browser for display.
For printing and handouts I want a PDF, but that’s always required a bit of manual work to convert the HTML to PDF, because the tools I tried back then didn’t produce the result I could get manually by opening in a Chrome browser with a ?print-pdf attached the the URL and then a judicious print-to-PDF and save.
My idea was to automate that, and I was prompted a few days ago to revisit decktape.
I shy away from installing NPM stuff because like Maven, Rust, and whatnot, they typically make me download half an internet, so I began by using the docker image as described at decktape. However I disliked the fonts and the overall impression of the generated PDF. “Disliked” is possibly unfair, but it just looked different…
It dawned on me then, obviously, that the docker image was running some incantation of Chrome on Linux, whereas I’m used to Chrome on Mac and its fonts, etc., so I then did actually install decktape locally with the resulting PDF looking exactly like what I was used to!
I’ve now got “automation” to (mostly) get the PDF output the way I wish to have it with a small script I’ve named runoff.sh (yes, the name is intentional :-)
The above launches an HTTP server with a random’ish port, takes note of the process’ PID, runs decktape, and finally kills off the HTTP server on error and at exit.
I said I’ve got things “mostly” how I want them, but there are two things missing so far:
the slide deck pages have no page number on them (and a few students have previously complained about that)
I’m still looking for a decent 2nup program; Ghostscript worked for quite a while quite well, but I had to stop using that because something broke; I can’t recall exactly what it was, so I will revisit. pdf2ps | psnup works somewhat, but the output is not half as nice as the original, and URLs are broken.
LeoMoon Wiki-Go calls itself a modern, feature-rich, databaseless, flat-file wiki platform built with Go, and it is all that.
Wiki-Go is a single-binary Go program, but there is a docker image for those who want it. I download the binary for my platform and launch it; this creates a data/ directory and its configuration file. If need be I configure a port number for it plus a timezone and the odd other setting and relaunch the binary.
The program has all the features I need and more, and what I most appreciate is that I can simply drop Markdown files, images, videos, etc. into a directory to have Wiki-Go serve them. I can also move documents to other pages, so for instance, when I’ve finished working on the Cascade page, I’ll move it into DNS/.
The program also boasts an interactive Markdown editor which certainly fulfills all I expect of one, including being able to format text, add links, insert images, etc.
In addition to wiki pages, the program supports the creation of Kanban boards and Link management pages, all of which are stored in *.md Markdown files.
There is a REST API with examples, which might be interesting to programmatically add comments, say, but if I have access to the server, I copy files into their directories.
There’s a web-based settings dialog where I can add users and configure roles, but I can also add these to the YAML configuration file manually:
This is really a lovely piece of work, and it is open source, and IMO its major features: Markdown in flat files on the file system, which makes it easy for me to synchronize the data directory onto my laptop and use a separate instance of Wiki-Go on that.
Assuming we want a host’s private key to be generated on a node and reside on the node only, and assuming our SSH CA (certification authority) is on the Ansible controller, we can use delegation to localhost for the bits which should happen on the controller.
In the following simple playbook I use a block in which I create a temporary unique directory locally into which the public key from the host is copied and signed, and the always portion of the block ensures the directory is cleared out even on error within the block. The public portion of the SSH key pair is returned in openssh_keypair’s metadata.
-hosts:d13gather_facts:yesremote_user:jpvars:dirname:"/etc/ssh"tasks:-name:Generate ED25519 SSH host key on nodecommunity.crypto.openssh_keypair:backend:"opensshbin"path:"{{dirname}}/ssh_host_ed25519_key"comment:"ansible-made™forhost{{inventory_hostname}}"type:"ed25519"become:trueregister:keydata-debug:var=keydata-block:-name:Create a local temporary directoryansible.builtin.tempfile:prefix:"jp"suffix:"cert"state:directoryregister:pdelegate_to:localhost-name:Save public host key to local temporary fileansible.builtin.copy:content:"{{keydata.public_key}}"dest:"{{p.path}}/{{keydata.filename|basename}}"delegate_to:localhost-name:Sign SSH certificate on local copy of public host keycommunity.crypto.openssh_cert:identifier:"{{inventory_hostname}}"public_key:"{{p.path}}/{{keydata.filename|basename}}"principals:"{{[ansible_fqdn]+ansible_all_ipv4_addresses}}"path:"{{p.path}}/{{keydata.filename|basename}}-cert.pub"serial_number:10signing_key:"../CA/ssh-ca"use_agent:truetype:"host"valid_from:"+0m"valid_to:"+53w"delegate_to:localhost-name:Install signed certificate on target nodeansible.builtin.copy:src:"{{p.path}}/{{keydata.filename|basename}}-cert.pub"dest:"{{dirname}}"mode:0444become:truealways:-name:Remove local temporary directoryansible.builtin.file:path:"{{p.path}}"state:absentdelegate_to:localhost
The openssh_cert module is able to sign the public key and create a certificate without specifying a password because I have the CA key in my SSH agent.
I left the debug in there so we can see the returned metadata when the host key is generated or read if it need not be changed:
TASK [Generate ED25519 SSH host key on node] *************************************************
changed: [d13]
TASK [debug] *********************************************************************************
ok: [d13] => {
"keydata": {
"changed": true,
"comment": "ansible-made™ for host d13",
"failed": false,
"filename": "/etc/ssh/ssh_host_ed25519_key",
"fingerprint": "SHA256:3zm2UIyqO0xsxejndKdg9+Rpm4JeGuxamlsZwNOGX2Y",
"public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBMNncuSW1uwBwCQTb7ZOH61ChYQiJVYtGR/SyuD3Mcl",
"size": 256,
"type": "ed25519"
}
}
TASK [Create a local temporary directory] ****************************************************
changed: [d13 -> localhost]
TASK [Save public host key to local temporary file] ******************************************
changed: [d13 -> localhost]
TASK [Sign SSH certificate on local copy of public host key] *********************************
changed: [d13 -> localhost]
TASK [Install signed certificate on target node] *********************************************
changed: [d13]
TASK [Remove local temporary directory] ******************************************************
changed: [d13 -> localhost]
Getting the playbook to alter sshd’s configuration to include the new certificate and restart it on a handler or two ought to be simple enough.
When I ssh into a server for the first time, I’m confronted with a dialog which asks me to verify I’m actually talking to the machine I expect to be talking to.
$ssh -l jp 192.0.2.65
The authenticity of host '192.0.2.65 (192.0.2.65)' can't be established.
ED25519 key fingerprint is SHA256:4WTRnq2OR1m03TpnHCfkFdlh1gN/PBXE4vDi0WnjFEc.
No matching host key fingerprint found in DNS.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])?
It is likely that the majority of users cross their fingers and type ‘yes’, which is not really a clever response. This Trust on First Use (TOFU) is what permits SSH to ensure that my SSH client verifies which server it’s talking to. I ought to have asked the administrator of the server to tell me its fingerprint, and if I am the administrator I ought to know how to do this: (in the following examples, a shell prompt % indicates I’m working as `root)
If the two fingerprints compare equal, I can trust that I am connecting to the correct server and can continue with ‘yes’ or I paste a known host fingerprint into the prompt: trust on first use is accomplished. (Utilities such as ssh-keyscan gather public keys from remote hosts, but I still ought to verify out-of-band whether I’m talking to the correct machine. SSH fingerprints can also be in the DNS, what can’t?, but that’s a different story.)
The session then possibly continues with me being asked for the target user’s password which, if entered correctly, grants me access to the machine.
SSH key pairs
If I create an SSH key pair, install my public key in the correct location (typically $HOME/.ssh/authorized_keys on the target node), and present the private key upon connection, then I don’t need to type the target user’s password; instead I enter the key’s passphrase, a hopefully much more complicated combination of words, to unlock the private key. I say “hopefully”, because this passphrase is what encrypts the private key so that it cannot be used without it. (Note that I typically generate keys with a comment in them so as to more easily keep track of them (-C), and specify which file (-f) they should be written to. The comment can be read from the public key file and will later be visible in the SSH agent.)
$ssh-keygen -t ecdsa -C"JP's demo key"-f demokey
Generating public/private ecdsa key pair.
Enter passphrase for "demokey" (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in demokey
Your public key has been saved in demokey.pub
$ssh-copy-id -i demokey.pub jp@192.0.2.65
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "demokey.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
jp@192.0.2.65's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh -i ./demokey 'jp@192.0.2.65'"
and check to make sure that only the key(s) you wanted were added.
$ssh -i demokey jp@192.0.2.65
Enter passphrase for key 'demokey':
jp@node:~$
The private SSH key still needs unlocking on use because I set a passphrase on it when creating it.
I can avoid having to do that at every use of the key, by launching an SSH agent which I feed with my private key and it will then no longer requests a passphrase on use. Desktop systems typically launch an agent at startup, but I do so explicitly here to demonstrate:
In order for this to work, the server needs a copy of the public key I use in one of the locations specified by SSHd’s AuthorizedKeysFile configuration which defaults to .ssh/authorized_keys and .ssh/authorized_keys2. Public keys can also be sourced from a command on a node.
This is well known and has worked for very many years, but the required procedures for public key authentication to work come with some disadvantages:
a copy of my SSH public key has to be available for each user I want to login as on a node
TOFU typically causes the host fingerprint to be stored on my client (in the known_hosts file)
when a host key rolls, I get a big warning
What’s with this warning? Well, if the server’s SSH host key changes, for instance because the server has been re-installed without restoring its original host keys, or because an administrator has forcefully re-generated them, my client will loudly complain that the server’s host keys have changed. Let’s see this in action.
The server’s admin re-generates SSH host keys on the server and restarts sshd
% rm -f /etc/ssh/ssh_host_*
% ssh-keygen -A
ssh-keygen: generating new host keys: RSA ECDSA ED25519
% systemctl restart sshd
Now I try and access the same server again, as above (my user key is still in the agent). The connection fails, I then remove the offending host key from the known_hosts file using ssh-keygen, obtain the server’s host key fingerprint to re-validate TOFU, and finally accept the host key into my known_hosts again.
$ssh -l jp 192.0.2.65 df-h /
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:3zm2UIyqO0xsxejndKdg9+Rpm4JeGuxamlsZwNOGX2Y.
Please contact your system administrator.
Add correct host key in /Users/jpm/.ssh/known_hosts to get rid of this message.
Offending ED25519 key in /Users/jpm/.ssh/known_hosts:649
Host key for 192.0.2.65 has changed and you have requested strict checking.
Host key verification failed.
$ssh-keygen -R 192.0.2.65
#Host 192.0.2.65 found: line 649
/Users/jpm/.ssh/known_hosts updated.
Original contents retained as /Users/jpm/.ssh/known_hosts.old
$ssh -l jp 192.0.2.65 df-h /
The authenticity of host '192.0.2.65 (192.0.2.65)' can't be established.
ED25519 key fingerprint is SHA256:3zm2UIyqO0xsxejndKdg9+Rpm4JeGuxamlsZwNOGX2Y.
No matching host key fingerprint found in DNS.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.0.2.65' (ED25519) to the list of known hosts.
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 19G 1.2G 17G 7% /
Admittedly this probably won’t occur very frequently, but in larger environments, this is something our users need to be made aware of, including how to correctly remedy the situation.
For those of us with a small handful of nodes or requiring connections to systems we have no root privileges on, working with keys as described so far is likely sufficient respectively the only choice, however for those with dozens or even hundreds of servers and full control there-over, we can make all of the disadvantages above go away with an SSH CA (certification authority) and SSH certificates. That sounds complicated, but it isn’t.
SSH Certification Authority
Those familiar with X.509 certificates, their complexity, and certification authorities might well have begun to groan now, but rest assured an SSH CA is something quite simple: all we need is an SSH key pair, and a few additional options for the ssh-keygen utility we’re already familiar with!
SSH certificates have existed in OpenSSH since version 5.4 released in March 2010, and the certificate format is based on data formats which implementations already support – one reason OpenSSH designed it this way rather than using the far more complicated X.509 format. The following looks like an SSH public key file is actually an OpenSSH public certificate.
For the following experiments I will use a single SSH CA key pair for users and hosts. (Two key pairs, one of hosts another for users, are sometimes used for better separation, but the principles described in the following are applied in the same way.)
Let me begin by enumerating some of the advantages of using SSH certificates:
no more deploying of public SSH keys into server authorized_keys files; users can forget about ssh-copy-id, editing $HOME/.ssh/authorized_keys manually, etc.
no danger of a user adding a key to an authorized_keys file for a key which shouldn’t have access at all
a server’s host keys can be rolled (replaced) without the scary “WARNING” on clients as we saw earlier; in fact there will no longer be host keys added to users’ known_hosts files!
no more TOFU (Trust on First Use) confirmation is required, as users will implicitly trust servers and vice-versa
principal names on user certificates dictate as which user(s) a user’s key can login as
remote commands can be enforced (user restrictions are bound to certificates and do not need to be added to a target user’s $HOME/.ssh/authorized_keys)
source IP prefixes can be limited, e.g. connecting as user root is permitted from 192.0.2.53/32 only.
certificates are valid for a specified time frame only, e.g. 30 minutes, 24 days, 34 weeks, etc. and they expire automatically (watch for clock drift)
a single line of configuration in the global known hosts files (e.g. /etc/ssh/ssh_known_hosts) suffices for all users on the client (but this line can also be in my personal $HOME/.ssh/known_hosts if I prefer)
in order to connect to a node, ssh -i needs the secret key file (jane) alongside its certificate (jane-cert.pub); if the certificate file isn’t available password authentication will be offered
So in our example the SSH CA will be used to sign host keys and user keys. To clarify, the CA is an SSH key pair which will be trusted by our nodes.
I proceed as follows.
On the CA machine, i.e. the system on which I will be “operating” the certification authority, I create a dedicated directory for the CA key pair. All we need on this system are the OpenSSH utilities, and we will be signing public keys on this system. I generate a CA key pair of algorithm ECDSA which generates short but strong keys; use any algorithm you prefer.
$umask 077;mkdir CA
$ssh-keygen -t ecdsa -C"JP's SSH CA"-f CA/ssh-ca
Generating public/private ecdsa key pair.
Enter passphrase for "CA/ssh-ca" (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in CA/ssh-ca
Your public key has been saved in CA/ssh-ca.pub
The key fingerprint is:
SHA256:A5ZBb5b/GbAv03EAb8fmDzv4p+q0g8Ulxrt8QZpbamM JP's SSH CA
$chmod-w CA/ssh-ca*
Jane creates a key pair for herself (or she uses an existing key pair she already has, e.g. in $HOME/.ssh/id_rsa or id_ecdsa). Like myself, she likes specifying a comment for the key and the file name into which it should be saved. Jane will send her public key (*.pub) to the CA for signing; as the public key is public it can be transmitted by any (also insecure) means.
$ssh-keygen -t ecdsa -C"Jane's key"-f jane
Generating public/private ecdsa key pair.
Enter passphrase for "jane" (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in jane
Your public key has been saved in jane.pub
The key fingerprint is:
SHA256:2WH263LauVfk5XvxHvrdvNt0y9OgaHBLSvcQ+R6u/KE Jane's key
Now I sign that user’s public key with our SSH CA key. I give the certificate an identity (-I), specify principals it may be used for (-n) (i.e. users on target hosts), a serial number (-z) for my own use and a validity (-V), in this example ending one week from now. I send the public certificate in jane-cert.pub back to Jane who then places the file adjacent to her private key file.
$ssh-keygen -s CA/ssh-ca -I"Jane Jolie"-n jane -z 001 -V +1w jane.pub
Enter passphrase for "CA/ssh-ca":
Signed user key jane-cert.pub: id "Jane Jolie" serial 1 for jane valid from 2026-03-27T13:57:00 to 2026-04-03T14:58:39
$ssh-keygen -L-f jane-cert.pub
jane-cert.pub:
Type: ecdsa-sha2-nistp256-cert-v01@openssh.com user certificate
Public key: ECDSA-CERT SHA256:2WH263LauVfk5XvxHvrdvNt0y9OgaHBLSvcQ+R6u/KE
Signing CA: ECDSA SHA256:A5ZBb5b/GbAv03EAb8fmDzv4p+q0g8Ulxrt8QZpbamM (using ecdsa-sha2-nistp256)
Key ID: "Jane Jolie"
Serial: 1
Valid: from 2026-03-27T13:57:00 to 2026-04-03T14:58:39
Principals:
jane
Critical Options: (none)
Extensions:
permit-X11-forwarding
permit-agent-forwarding
permit-port-forwarding
permit-pty
permit-user-rc
I then copy our CA’s public SSH key to the target server and configure it into the SSH server’s configuration. This ensures the SSH service on this node will trust SSH keys signed by this CA. I don’t restart sshd yet.
$[copy CA's public key to server]
% install -m444 /tmp/ssh-ca.pub /etc/ssh/ssh-ca.pub
% echo "TrustedUserCAKeys /etc/ssh/ssh-ca.pub" >> sshd_config
I then obtain the node’s host key(s) and sign them with our CA, again specifying a validity, an identifier of choice, a serial number, and a principal (the node’s hostname(s)) for the key. Note the -h option with which we sign a host certificate.
% [ copy server's /etc/ssh/ssh_host_ed25519_key.pub to CA ]
$ssh-keygen -h-s CA/ssh-ca -V +52w -I deadbeef01 -z 1000 -n alice.example.com ssh_host_ed25519_key.pub
Enter passphrase for "CA/ssh-ca":
Signed host key ssh_host_ed25519_key-cert.pub: id "deadbeef01" serial 1000 for alice.example.com valid from 2026-03-27T14:04:00 to 2027-03-26T14:05:47
$ssh-keygen -L-f ssh_host_ed25519_key-cert.pub
ssh_host_ed25519_key-cert.pub:
Type: ssh-ed25519-cert-v01@openssh.com host certificate
Public key: ED25519-CERT SHA256:ddxT1zL+HhHpIT5qWPdMJ6GC1SbVp2ij/2Sca5xA3RE
Signing CA: ECDSA SHA256:A5ZBb5b/GbAv03EAb8fmDzv4p+q0g8Ulxrt8QZpbamM (using ecdsa-sha2-nistp256)
Key ID: "deadbeef01"
Serial: 1000
Valid: from 2026-03-27T14:04:00 to 2027-03-26T14:05:47
Principals:
alice.example.com
Critical Options: (none)
Extensions: (none)
I install the signed certificate(s) alongside the SSH server’s host key(s)
$[ copy host's key certificate to the server ]
% install -m444 tmpfile /etc/ssh/ssh_host_ed25519_key-cert.pub
and configure the SSH server to use the new certificate and restart sshd
and finally add a reference to the host’s certificate authority in the client’s known_hosts file, either the global one in /etc/ssh/known_hosts or my own in ~/.ssh/known_hosts. Here I overwrite the file as I no longer need the fingerprints it contained, but I can also keep the file’s content and append or prepend this line. The @cert-authority line has a glob-style pattern used to match trusted nodes which present host keys signed with this (our CA’s) public key. This is how we add trust to our client for nodes’ SSH host keys signed by our CA.
We created an SSH CA by generating an SSH key pair, and we have signed one or more public user keys and public host keys. Then on a node, we wired up the public key of our SSH CA and made the server trust it.
On our client machines, we added a single line of configuration to our known_hosts file (user-specific or the global one), and this line will cause our clients to trust host keys signed by our CA.
We can now test.
An initial connection
In order to test, I create a user on the target system
% useradd -c "Jane Jolie" -m jane
we can now connect to the server. For debugging purposes I specify options (-o) to ensure ssh uses exactly the files I specify here. In particular I want to make sure we’re informed when server host keys cannot be validated with the ask option. If all goes well I will not be asked to trust a server’s fingerprint, and after the SSH session terminates, there will be no additional line in the UserKnownHostsFile. (This last file contains one line only, the @cert-authority line.)
$ssh -oUserKnownHostsFile=./known_hosts -oStrictHostKeyChecking=ask \-oIdentitiesOnly=true-i jane -l jane alice.example.com uname Linux
the server logs in auth.log (here OpenBSD) but likewise in a systemd journal. Note we can read out the fingerprint of the used key, its identity (Jane Jolie), its serial number (001 used during signing is simply an integer which is why it’s shown here as 1), and the public key fingerprint of the signing CA.
sshd-session[3099]: Accepted publickey for jane from 192.0.2.42 port 17087 ssh2: ECDSA-CERT SHA256:2WH263LauVfk5XvxHvrdvNt0y9OgaHBLSvcQ+R6u/KE ID Jane Jolie (serial 1) CA ECDSA SHA256:A5ZBb5b/GbAv03EAb8fmDzv4p+q0g8Ulxrt8QZpbamM
If I want to connect by IP address as well as host name, I have to sign a node’s host key(s) with more than one principal name in them, and the entry in known_hosts needs these added comma-separated to it
if I try to use Jane’s key and certificate to login as, say, ansible on the target node, that fails because Jane’s user key certificate contains a principal “jane” only. (Add more comma-separated principal names when signing the user key in the -n option)
$ssh -i jane -l ansible alice.example.com
sshd-session[3648]: error: Certificate invalid: name is not a listed principal
a user’s certificate can be forced to invoke a particular utility on the server instead of what the user wanted. Here we add a forced date, and in spite of the user wanting to invoke uname, date is executed on the remote node.
$ssh-keygen -s CA/ssh-ca -I"Jane Jolie"-n jane -z 2 -V +1w -O force-command=/usr/bin/date jane.pub
$ssh -i jane -l jane alice.example.com uname Fri Mar 27 13:43:58 UTC 2026
permissible source CIDR masks can be embedded into a user certificate, and the server logs violations
$ssh-keygen -s CA/ssh-ca -I"Jane Jolie"-n jane -z 3 -V +1w -O force-command=/usr/bin/date -O source-address=192.0.2.0/24 jane.pub
sshd-session[3854]: cert: Authentication tried for jane with valid certificate but not from a permitted source address (192.168.1.100).
it’s worth looking at the description of AuthorizedPrincipals in which I can specify a file that lists principal names accepted for certificate authentication; there is also AuthorizedPrincipalsCommand with which I can specify a command which retrieves the list from an arbitrary source, similar to what we did with AuthorizedKeysCommand.
Revocation
Certificates can expire as we saw above, but sometimes we want to actually revoke a key, which we can do with an revocation file which contains any number of revoked keys.
I create this file on my CA machine:
$ssh-keygen -k-f revoked jane.pub
Revoking from jane.pub
$file revoked
revoked: OpenSSH key/certificate revocation list, format 1, version 0, generated Tue Apr 7 14:24:33 2026
I install the revoked file on the target node and configure it in sshd_config:
Key revocation lists can contain a complex set of rules specifying which keys are revoked. See the KEY REVOCATION LISTS section in [ssh-keygen(1)20.
Checklist
I’ve tried to assemble a bit of a checklist of things we verify particularly if something doesn’t work as expected.
on the node:
CA’s public key must be on server, readable by sshd, and configured in TrustedUserCAKeys
server’s host key needs signing and must be placed alongside host key file(s)
sshd_config requires HostCertificate pointing to the certificate of the host key
server needs restarting
if the node’s certificate changes, the service needs to be restarted
on the client:
user’s SSH key needs to be signed by CA and placed adjacent to the key file
user’s SSH key may be in SSH agent; if it isn’t and the key is encrypted, SSH will as usual ask for its passphrase if there is one
known_hosts needs correct @cert-authority; pay attention to principal names (test with * to permit any hostname)
the node name or address used when connecting to server must match one of the principal names in host key certificate on server
If I do get prompted to confirm the host fingerprint, it is possible the certificate has expired, and I am told so.
$ssh -oStrictHostKeyChecking=ask -oIdentitiesOnly=true-i jane -l root 192.0.2.65
Certificate invalid: expired
The authenticity of host '192.0.2.65 (192.0.2.65)' can't be established.
ED25519 key fingerprint is SHA256:3zm2UIyqO0xsxejndKdg9+Rpm4JeGuxamlsZwNOGX2Y.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? ^C
on the other hand, if I’m prompted for a password, it is quite likely that a constraint in the certificate cannot be validated. Here I’m tyring to login as a user for which the certificate has no principal configure. The server log shows it.
$ssh -i jane -l ansible 192.0.2.65
ansible@192.0.2.65's password:
% journalctl -f
sshd-session[6622]: error: Certificate invalid: name is not a listed principal
if the server had been configured to permit only public key authentication (no passwords), Jane would have seen the following diagnostic and not had the possibility to enter a password.
$ssh -i jane -l ansible 192.0.2.65
ansible@192.0.2.65: Permission denied (publickey).
We accomplish this by configuring /etc/ssh/sshd_config as follows:
PubkeyAuthentication yes
PasswordAuthentication no
ChallengeResponseAuthentication no
adding a user’s key to an SSH agent shows that both it and the cert have been added (both need to be removed with -d if desired)
$ssh-add jane
Enter passphrase for jane:
Identity added: jane (Jane's key)
Certificate added: jane-cert.pub (Jane Jolie)
$ssh-add -l 256 SHA256:2WH263LauVfk5XvxHvrdvNt0y9OgaHBLSvcQ+R6u/KE Jane's key (ECDSA)
256 SHA256:2WH263LauVfk5XvxHvrdvNt0y9OgaHBLSvcQ+R6u/KE Jane's key (ECDSA-CERT)
during signing of user certificates features can be enabled or disabled. Here we clear all options, then add permission for agent forwarding and port forwarding. Note that pseudo-tty (PTY) allocation is therefore explicitly disabled. We can login to the target node, but do not get a PTY allocated, i.e. we have no console
$ssh-keygen -U\-s CA/ssh-ca \-I"Jane Jolie"\-n jane,root \-z 4 \-V +1w \-O clear \-O extension:permit-agent-forwarding \-O extension:permit-port-forwarding \
jane.pub
Signed user key jane-cert.pub: id "Jane Jolie" serial 4 for jane,root valid from 2026-03-28T15:01:00 to 2026-04-04T16:02:02
$ssh-keygen -L-f jane-cert.pub
jane-cert.pub:
Type: ecdsa-sha2-nistp256-cert-v01@openssh.com user certificate
Public key: ECDSA-CERT SHA256:2WH263LauVfk5XvxHvrdvNt0y9OgaHBLSvcQ+R6u/KE
Signing CA: ECDSA SHA256:A5ZBb5b/GbAv03EAb8fmDzv4p+q0g8Ulxrt8QZpbamM (using ecdsa-sha2-nistp256)
Key ID: "Jane Jolie"
Serial: 4
Valid: from 2026-03-28T15:01:00 to 2026-04-04T16:02:02
Principals:
jane
root
Critical Options: (none)
Extensions:
permit-agent-forwarding
permit-port-forwarding
$ssh -l jane o78
PTY allocation request failed on channel 0
$ssh -l jane o78 uname OpenBSD
And the “magical” part of all this? The signing user’s SSH certificate configures these capabilities: neither the key, nor an entry in a user’s authorized_keys file on the target system have been added.
Automate host key certificate distribution?
Being able to easily sign host keys got me thinking that it would likely be easy’ish to do this automatically. I quickly shredded my idea of an CGI (Common Gateway Interface) shell script which invoked ssh-keygen when I came accross sshkey-tools, a well-documented Python module with which I can programmatically issue and sign SSH certificates.
An initial sign.py was quickly created, and I then wrapped that into a BottlePy HTTP server, as proof of concept:
on the machine containing the CA, I launch my small host key bot (hkbot.py) for which I don’t yet have a logo but patent is pending. :-)
$ ./hkbot.py
Bottle v0.13.4 server starting up (using WSGIRefServer())...
Listening on http://0.0.0.0:8870/
Hit Ctrl-C to quit.
on a client node onto which I want to have a host key signed, I upload one of host’s public keys to the bot which will determine the key type (the Python module does that automatically), and sign a certificate which it returns together with very basic shell commands to install the CA’s public key, the newly issued certificate, and with which to patch /etc/ssh/sshd_config with the necessary statements.
# curl -sSf -F hostkey=@/etc/ssh/ssh_host_ed25519_key.pub http://192.0.2.140:8870 | sh
Extracting CA public key to /etc/ssh/ssh-ca.pub
Extracting certificate to /etc/ssh/ssh_host_ed25519_key-cert.pub
Patching /etc/ssh/sshd_config
Validating /etc/ssh/sshd_config
after verifying sshd_config I restart the service
# systemctl restart sshd
I should now be able to seamlessly login to the machine. Using -v to show what the server sends, and explicitly setting ask to make sure I’m informed should a host key not be verifiable.
$ ssh -v -o IdentitiesOnly=true -i jane -l root 192.0.2.65
...
debug1: Server host certificate: ssh-ed25519-cert-v01@openssh.com SHA256:4WTRnq2OR1m03TpnHCfkFdlh1gN/PBXE4vDi0WnjFEc, serial 27 ID "deadbeef01" CA ecdsa-sha2-nistp256 SHA256:A5ZBb5b/GbAv03EAb8fmDzv4p+q0g8Ulxrt8QZpbamM valid from 2026-04-02T10:19:46 to 2026-04-02T11:19:46
...
root@node:~#
I keep an eye on the auth log (syslog) or the journal (systemd) for incoming connections on the machine
Apr 02 10:22:07 d13 sshd-session[1058]: Accepted publickey for root from 192.0.2.140 port 54872 ssh2: ECDSA-CERT SHA256:2WH263LauVfk5XvxHvrdvNt0y9OgaHBLSvcQ+R6u/KE ID Jane Jolie (serial 4) CA ECDSA SHA256:A5ZBb5b/GbAv03EAb8fmDzv4p+q0g8Ulxrt8QZpbamM
It’s a bit primitive, and there are lots of bits missing for real-life, but I was able to scratch an itch. I later scratched a further itch when deploying SSH host keys and certificates with Ansible, generating keys on the node and signing certificates on the controller.
The title of this post is “SSH certificates: the better SSH experience”, and I really believe SSH certificates provide the better and more seamless operation for environments in which I have control over the nodes:
No more need for TOFU (Trust On First Use), as clients and servers implicitly trust each other.
SSH certificates allow me to issue short-lived keys to users (you may login in the next 20 minutes, after that not again). (If the user remains logged in, there’s no automation to kick them out.)
The use of SSH certificates means I don’t have to clean up and remove public user keys from servers (i.e. remove entries from authorized_keys files); once the validity of the certificate expires, the user is locked out.
Certificates can contain forced-commands, we can permit or deny creation of PTY on the remote, and the target users (principals) and permissible source addresses can be specified.
I demonstrated above that some of this can be automated, and there is a very interesting project called Smallstep SSH with tools to enable all that and much more.
This is neat for hosts that you own, e.g. where you have control of the CA and access to its private key which one needs for signing hosts/user keys. The “traditional” known_hosts/authorized keys was made for multi-user systems where you want secure connections between systems where you do not have root privileges. But nowadays, most usage falls into the first category, I assume. And a CA makes that easier.
I learn from Job Snijders about an I-D for the SSH Certificate Format, in draft-miller-ssh-cert-06.
I verified that SSH certificates also work very well with -SK (security key) keys, for instance with FIDO2 tokens. I sign the key’s public key as we did above, and that’s it.
