How to renew Puppet CA and server certificates in place

In the following, I will describe what we did to renew both CA and Puppet server certificates. Despite that this problem should occur on every Puppet server running for a prolonged amount of time, I found remarkably few resources on the net (that did not involve completely replacing the CA) – so I'm going to share our findings. 

Act before it's too late

It is important to pay immediate attention to messages like:

Certificate 'Puppet CA: server1.fcio.net' will expire on 2017-10-01T11:09:06UTC
Certificate 'puppet.fcio.net' will expire on 2017-10-03T11:13:58UTC

As long as the certificates are only soon to expire, it is possible to roll out a new CA certificate using Puppet itself. Later on, the only rescue will be to regenerate all certificates and install them manually on each client. But since the certificates are still valid albeit soon-to-be-expired, we have a better plan. We generate new certificates with extended expiration times based on the original keys. All certificates stay compatible to each other — a prerequisite for a slick roll over. Fasten seat belts and make sure to back up the SSL directory before proceeding.

Reverse-generate missing signing requests

Before getting into business, we have to check if there are certificate signing requests (CSR) available. CSRs should be found unter /var/lib/puppet/ssl/ca/ca_csr.pem and /var/lib/puppet/ssl/certificate_requests/puppet.fcio.net.pem. Ours seem to have got lost over time. No problem! A missing CSR can be generated from an existing certificate. Here is how we did it for the CA certificate:

cd /var/lib/puppet/ssl/ca
openssl x509 -x509toreq -in ca_crt.pem -signkey ca_key.pem \
  -out ca_csr.pem

The same procedure applies to recreating a missing CSR for a Puppet server certificate in certs/puppet.fcio.net.pem. I assume the the Puppet server's private key lives in private_keys/puppet.fcio.net.pem and the CSR goes into certificate_requests/puppet.fcio.net.pem.  Note, that those requests aren't perfect: the certificate CA extensions are missing, but we'll fill them in during the next step.

Certificate renewal

Next, we generate new certificates from the existing private key and the signing request. To get the necessary X509v3 extensions into the CA certificate, we first create a suitable OpenSSL config file snippet:

cat > extension.cnf <<_EOT_
[CA_extensions]
basicConstraints = critical,CA:TRUE
nsComment = "Puppet Ruby/OpenSSL Internal Certificate"
keyUsage = critical,keyCertSign,cRLSign
subjectKeyIdentifier = hash
_EOT_

With the extensions config in place, we generate the new certificate:

openssl x509 -req -days 3650 -in ca_csr.pem -signkey ca_key.pem \
  -out ca_crt.pem -extfile extension.cnf -extensions CA_extensions

We can now use the CA certificate to sign a new Puppet server certificate:

cd /var/lib/puppet/ssl
openssl x509 -req -days 3650 -in certificate_requests/puppet.fcio.net.pem -CA ca/ca_crt.pem \
  -CAkey ca/ca_key.pem -CAserial ca/serial -out certs/puppet.fcio.net.pem

Of course, the new certificates do nothing unless they are actually used. In our case we had to restart the Puppet server so they got picked up.

Distribute the CA certificate

Puppet clients need the CA certificate to be locally available so that they can verify other certificates against it. We copy the newly generated ca_crt.pem into some Puppet module and let Puppet place it on all clients:

 file { '/var/lib/puppet/ssl/certs/ca.pem': 
   source => 'puppet:///path/to/ca_crt.pem', 
   owner => 'puppet', 
   group => 'puppet', 
 }

That's it! No more warnings, everyone happy. :-)