Creating a Certificate Authority With Hashicorp Vault and Terraform
One of the things that Vault is good at is being a Certificate Authority and creating certificates for a mTLS based infrastructure. The client and Server need to have appropriate certificates before any communication can start. This gives your infrastructure one more layer of security.
There are good examples by Hashicorp on how to create a Certificate Authority with Vault. These are based on using the GUI, the Vault CLI or the raw rest apis. However when I tried to create a CA using Terraform I found the vault_pki* provider APIs to be lacking in any kind of real documentation about how to string the calls together. There are several steps to create and install the Certificate Authority certs on Vault. This article attempts to de-mystify this process.
This walkthrough will guide you through:
- Creating a simple vault server instance to run this demo on.
- Creating a CA and Intermediate CA
- Creating Roles to create certificates for Servers and Client access
- Restarting your Vault server with Certificates enabled
It is assumed that you have already done some vault administration and have used terraform already. If you aren’t going to use Terraform, Hashicorp has documentation on how to do this with the CLI, API and the GUI.
The code snippets shown below are part of a GitHub repo. You should clone this, as it provides full scripts for work that I will describe.
If you look at the vault_pki* resources you will see that there are numerous warnings about the sensitivity of the API. This terraform script will run and will output the private key of your root CA. This is an incredibly sensitive piece of information. When you prepare for a production run of creating a CA key, be very careful of where you run these scripts and remote state.
Setup
- Clone the github repository:
git clone git@github.com:stvdilln/vault-ca-demo.git - Install a current copy of vault somewhere in your path.
- Install a current copy of terraform (v0.12.xx+) somewhere in your path.
- jq is installed on machine.
Overview
There are a few stages to running this demo, maybe I tried to get too much into the demo, but if you follow along, this will provide all the information for setting up a CA in vault, and using those certs to create Server and Client certs for TLS. This bootstrap process requires us to first start vault without SSL and then later restarting Vault after we have created certificates. The flow of the demonstration is:
- Start a local vault server (without ssl) and unseal it.
- Run terraform against this server and create the Certificate Authorities
- Create Roles in Vault to issue Server and Client certificates
- Stop Vault and Restart in TLS mode
- Set Vault client environment variables, and access server with TLS.
Running a Local Vault Server
The typical instructions for playing with vault tell you to run vault server -dev , this will be insufficient for this demonstration as you cannot configure TLS in -dev mode. You will need to start the server in one terminal window and then issue commands against it in another.
# On the "server" bash shell
vault server -config vault-server-no-tls.hcl# Open a new bash shell, the "client" bash shell
steved$ export VAULT_ADDR=http://127.0.0.1:8200
steved$ vault operator init -key-shares 1 -key-threshold 1Unseal Key 1: izZoYuJEv/dkrSkJ56IqOVFMco8+79nbxBSNMNGM0ds=Initial Root Token: s.1ZB3zxzqvbOCIy4wX5rZyyJF... snip ...steved$ export VAULT_TOKEN={your token}
steved$ vault operator unseal
{your Unseal key}
# Save these secrets, you will need them later
Creating the Root and Intermediate CA certificates
Going through the the demonstration code, let’s look at the meat of root_ca.tf which creates the root certificate authority.
I create the root certificate which I will use for the root CA. I use tls_self_signed_cert for the primary reason that I was not able to get vault_pki_secret_backend_root_cert to output the very sensitive private key for the certificate. Without having a private key, you can’t load the certificate into another Vault server, or HSM, or other PKI infrastructure. Without the exported private key, you may be maintaining this vault server instance for years. Where you want to keep your master root CA might not even be in Vault, but an HSM or other storage so that a compromised vault server doesn’t destroy your tls infrastructure.
Ok, now all the scary warnings are over. We have created a Root self signed certificate, the next significant bit of code is where we load the Certificate and it’s private key into vault. This demonstrates that your root certificate and its key can be generated outside of Vault, and outside of terraform. If you were to create a deeper intermediate cert chain, you might not ever expose the root cert to Vault. Also note that Vault Enterprise has HSM support, so you might want to look at that as the source for the root certificate.
Install Root CA
We install our root CA into the root mount point with the following:
Create Intermediate CA.
The intermediate CA is created in the project file int.tf. It is a 3 step process. Please look at intermediate_ca.tf for details of each of the api/resources used.
vault_pki_secret_backend_intermediate_cert_request: is called to create a Certificate Signing Request (CSR). Behind the scenes this creates a private key that is stored in the Vault mount (pki-int-ca).
The CSR output is fed into resource vault_pki_secret_backend_root_sign_intermediate which passes the CSR to the Root CA, which signs it and returns the certificate.
The signed cert is then installed into the Vault mount pki-int-ca.
In this flow the private key is never exposed to the public or terraform. In the above snippet see that when installing the Cert the certificate key is not provided, but was created and saved when we did the CSR request. This is more secure but limits this intermediate CA to only creating certificates within vault.
Create Roles to Generate Certificates
In order to create non-certificate authority certs you must create ‘roles’ for the pki-int-ca mount point. The roles establish what certificates are allowed to be created in vault. You can control what domains you create certs for, and many other aspects of the generated certificates. Using RBAC on the roles will allow you to control exactly who, creates exactly what certificates.
The terraform example code creates two roles (in file roles.tf).
Role #1 is for generating certificates for servers. It allows you to specify a name eg: vault-dev.mydomain.com and IP addresses eg: 127.0.0.1. To access vault via TLS either the subject name or the IP_SAN (Subject Alternative Name) must match, or you will get an error from the client.
Role #2 is for generating certificates for clients. It takes common names in email format joe.user@mydomain.com.
Run the Terraform Scripts
In the “client” shell window (that has VAULT_ADDR configured) run:
terrafom init
terraform applyYou have now created the Root CA, Intermediate CA and roles that allow certificates to be created. You can create certificates with the UI at http://localhost:8200/
I find that generating certificates with vault to be a bit clumsy, as you may want to generate a new cert with the same common name for any number of reasons. So using the Vault UI or CLI makes more sense.
I have created a script file create-certs which will create server and client certificates, that we will use to re-start vault with TLS fully enabled.
Run the create-certs file now:
# when prompted for export password just hit enter
./create-certsClient certs are created in ./output/client-certs, and server certs are in ./output/server-certs. Note that I export the certs into .txt files so that you can see all of the fields in the certificate and how they match the settings in the terraform files.
Restarting Vault with TLS enabled
In the “server” shell, stop the vault process and restart with:
# In the "server" shell, run this:
vault server -config vault-server-tls.hcl
# Switch over to the "client" shell
# There is a script that sets the TLS settings needed by the client
source ./env-vars-vault-client-tls.sh
vault operator unseal
Congratulations, it’s been a long journey, at this point you have setup vault to be a fully functional certificate server and have configured mutual TLS connection between the vault client and the server.
If you have comments, post them here. If you have changes to the demo code, send me a PR. I’m stvdilln on github, keybase.