HashiCorp Vault
The vault_v1 role deploys a HashiCorp Vault instance in the
Kubernetes cluster. This is not to be confused with (upcoming) support
for using HashiCorp Vault as a backend for storing LCM secrets.
For more details about the software, check out the HashiCorp Vault website. The deployment happens via the Vault Helm Chart.
Note
The Vault integration is considered ready for production. It’s deployed in a highly available fashion (n=3) and comes with support for backups and monitoring. The initial unseal keys are written to the file system and must be rotated as soon as possible and the fresh keys then stored appropriately. Note that one can create a fresh root token as long as one has the necessary unseal key(s).
Backups
Backups have become opt-out instead of opt-in because of their importance. The role deploys the backup-creator and -shifter tandem which pushes an encrypted snapshot to an S3 bucket.
If you want to use this built-in
backup mechanism (which you should unless you have an alternative),
create a copy of
managed-k8s/templates/vault_backup_s3_config.template.yaml, place it
as config/vault_backup_s3_config.yaml and fill in the gaps.
Enable backups by setting yk8s.k8s-service-layer.vault.enable_backups to true.
This configuration must be stored in your cluster key-value secrets engine
under kv/data/vault-backup-s3-config.
Inserting the Vault backup S3 config into Vault can be automated by storing the
configuration at config/vault_backup_s3_config.yaml and then triggering
the Vault update script:
$ ./managed-k8s/tools/vault/update.sh
Alternatively, you can also manually insert your configuration into vault.
Consider the official docs for restore instructions.
If no backups are required, disable backups by setting yk8s.k8s-service-layer.vault.enable_backups to false.
Credential management
Unseal key(s) and an initial root token are cached in the cluster
repository (etc/vault_unseal.key and
etc/vault_root_token). It is up to you to rotate and
store them securely. The role requires privileges to create certain
policies and approles so for now we rely on using a root token (which is
kind of ugly).
The root token is either read from the file system (
etc/vault_root_token) or from the environment variableVAULT_TOKEN.If vault is sealed, then the role will attempt to read the unseal keys from the file system (
etc/vault_unseal.key). If they cannot be found, ask the (human) operators to unlock vault for you.
Monitoring
Vault offers metrics and the role deploys a corresponding service monitor including required means of authentication and authorization.
The Prometheus-Vault integration leverages a long-lived periodic token to access the Vault metrics endpoint. This token is automatically generated and managed by a Kubernetes CronJob and Pod. The CronJob is scheduled to run every Thursday at 1:00 PM, creating a new Pod to renew the token. This automated process eliminates the need for manual renewal and reduces the risk of misconfiguration.
No alerting rules have been defined yet but one probably wants to keep an eye on vault’s performance, response error rate and seal status.
API endpoints and certificates
By default, vault listens on a cluster internal API endpoint whose
authenticity is ensured by a self-signed PKI. If you need to use that
endpoint you can find the certificate in the secret
vault-ca-internal. The LCM does not fetch the certificate for
you. Additionally, one can define external (or public) endpoints. Place
the DNS names in the dnsnames list and ensure that the records and
your ingress controller are properly configured. Furthermore you have to
specify a (Cluster)Issuer in external_ingress_issuer_name and,
if required, change the value of external_ingress_issuer_kind.
Note: We cannot assume the existence of a publically available vault endpoint but must be able to interact with the vault cluster from the orchestrator. As a consequence we cannot make use of ansible’s built-in vault modules but instead we have to jump into the vault pods to execute commands as our second-best option.
Vault Configuraton
See yk8s.vault