HashiCorp Vault

The vault_v1 role deploys a HashiCorp Vault instance in the Kubernetes cluster. This is not to be confused with (upcoming) support for using HashiCorp Vault as a backend for storing LCM secrets.

For more details about the software, check out the HashiCorp Vault website. The deployment happens via the Vault Helm Chart.

Note

The Vault integration is considered ready for production. It’s deployed in a highly available fashion (n=3) and comes with support for backups and monitoring. The initial unseal keys are written to the file system and must be rotated as soon as possible and the fresh keys then stored appropriately. Note that one can create a fresh root token as long as one has the necessary unseal key(s).

Backups

Backups have become opt-out instead of opt-in because of their importance. The role deploys the backup-creator and -shifter tandem which pushes an encrypted snapshot to an S3 bucket. If you want to use this built-in backup mechanism (which you should unless you have an alternative), create a copy of managed-k8s/templates/vault_backup_s3_config.template.yaml, place it as config/vault_backup_s3_config.yaml and fill in the gaps. Disable backups by setting enable_backups = false. Consider the official docs for restore instructions.

Credential management

Unseal key(s) and an initial root token are cached in the cluster repository (inventory/.etc/vault_unseal.key and inventory/.etc/vault_root_token). It is up to you to rotate and store them securely. The role requires privileges to create certain policies and approles so for now we rely on using a root token (which is kind of ugly).

  • The root token is either read from the file system (inventory/.etc/vault_root_token) or from the environment variable VAULT_TOKEN.

  • If vault is sealed, then the role will attempt to read the unseal keys from the file system (inventory/.etc/vault_unseal.key). If they cannot be found, ask the (human) operators to unlock vault for you.

Monitoring

Vault offers metrics and the role deploys a corresponding service monitor including required means of authentication and authorization. No alerting rules have been defined yet but one probably wants to keep an eye on vault’s performance, response error rate and seal status.

API endpoints and certificates

By default, vault listens on a cluster internal API endpoint whose authenticity is ensured by a self-signed PKI. If you need to use that endpoint you can find the certificate in the secret vault-ca-internal. The LCM does not fetch the certificate for you. Additionally, one can define external (or public) endpoints. Place the DNS names in the dnsnames list and ensure that the records and your ingress controller are properly configured. Furthermore you have to specify a (Cluster)Issuer in external_ingress_issuer_name and, if required, change the value of external_ingress_issuer_kind.

Note: We cannot assume the existence of a publically available vault endpoint but must be able to interact with the vault cluster from the orchestrator. As a consequence we cannot make use of ansible’s built-in vault modules but instead we have to jump into the vault pods to execute commands as our second-best option.

Vault Configuraton

# --- KUBERNETES SERVICE LAYER : HASHICORP VAULT ---
# ansible prefix: "yaook_vault_"
[k8s-service-layer.vault]
# Enable HashiCorp Vault management.
# NOTE: On the first run, the unseal keys and the root token will be printed IN
# PLAINTEXT on the ansible output. The unseal keys MUST BE SAVED IN A SECURE
# LOCATION to use the Vault instance in the future!
enabled = false

# Create a publically reachable ingress resource for the API endpoint of vault.
#ingress = false

# Version of the Helm Chart to use
#chart_version = "0.20.1"

# Namespace to deploy the vault in (will be created if it does not exist, but
# never deleted).
#namespace = "k8s-svc-vault"

# Extra DNS names for which certificates should be prepared.
# NOTE: to work correctly, there must exist an ingress of class `nginx` and it
# must allow ssl passthrough.
#dnsnames = [..]

# If set to true, the Vault is configured to be exposed via yaook/operator
# infra-ironic, that is, via the integrated DNSmasq to all nodes associated.
# The default is false. This can be enabled in non-infra-ironic clusters,
# without significant damage.
# NOTE: To work in infra-ironic clusters, this requires the vault to be in the
# same namespace as the infra-ironic instance.
# NOTE: if you enable this, you MUST NOT set the service_type to ClusterIP; it
# will default to NodePort and it must be at least NodePort or LoadBalancer for
# the integration to work correctly.
#management_cluster_integration = false

# Number of unseal key shares to generate upon vault initialization.
# NOTE: On the first run, the unseal keys and the root token will be printed IN
# PLAINTEXT on the ansible output. The unseal keys MUST BE SAVED IN A SECURE
# LOCATION to use the Vault instance in the future!
#init_key_shares = 5

# Threshold for the Shamir's Secret Sharing Scheme used for unsealing, i.e. the
# number of shares required to unseal the vault after a restart
# NOTE: On the first run, the unseal keys and the root token will be printed IN
# PLAINTEXT on the ansible output. The unseal keys MUST BE SAVED IN A SECURE
# LOCATION to use the Vault instance in the future!
#init_key_threshold = 2

# Scheduling key for the vault instance and its resources. Has no default.
#scheduling_key =

# Storage class for the vault file storage backend.
#storage_class = "csi-sc-cinderplugin"

# Storage size for the vault file storage backend.
#storage_size = "8Gi"

# If `ingress=True` and `dnsnames` is not empty, you have to tell the LCM which (Cluster)Issuer to use
# for your ACME service.
#external_ingress_issuer_name = ""

# Can be `Issuer` or `ClusterIssuer`, depending on the kind of issuer you would like
# to use for externally facing certificates.
#external_ingress_issuer_kind = "ClusterIssuer"

# If `true`, then an additional backup service will be deployed which creates snapshots and stores
# them in an S3 bucket.
#enable_backups = true

# Credentials to access an S3 bucket to which the backups will be written. Required if `enable_backups = true`.
# You can find a template in `managed-k8s/templates/vault_backup_s3_config.template.yaml`.
#s3_config_file = "vault_backup_s3_config.yaml"

# Type of the Kubernetes Service of the Vault
# NOTE: You may set this to LoadBalancer, but note that this will still use the internal certificate.
# If you want to expose the Vault to the outside world, use the ingress config above.
#service_type = "ClusterIP"

# Node port to use for the Service which exposes the active Vault instance
# See NOTE above regarding exposure of the Vault.
#active_node_port = 32048