Restoring Kubernetes’ ability to sign certificates
Note
Requires at least version 6.0
Since YAOOK/k8s migrated to Hashicorp Vault for managing PKI and therefore clears the Kubernetes cluster root CA key from the control plane, YAOOK/k8s clusters are not able to respond to certificate signing requests (CSRs) anymore since access to PKI keys ceased. [1]
While in the long term we want to integrate Vault via a Kubernetes custom signer [2]
which would redirect CSRs in Kubernetes directly to Vault,
for now we provide a workaround fix to restore the CSR functionality.
The fix must be explicitly turned on
by setting [kubernetes.controller_manager].enable_signing_requests=true
in the config.
Enabling the fix
Attention
This weakens the security of your deployment!
Enable signing requests in the config
Set
[kubernetes.controller_manager].enable_signing_requests=true
in config/config.tomlConfigure k8s_control_plane Vault policy
Executing ‘tools/vault/init.sh’ will take care of that. This assumes you already have a Vault instance running and reachable from your YAOOK/k8s shell environment.
Note
Requires a Vault root token
./managed-k8s/tools/vault/init.sh
The Kubernetes cluster root CA key is made available through the
k8s-pki/cluster-root-ca
secret in the Vault kv2 store of the cluster. The script adds a Vault policy rule (among other things) that grants read-only access to it for the control plane nodes role.
Create the Kubernetes cluster root CA and backup its key
Executing ‘tools/vault/mkcluster-root.sh’ will take care of that.
Note
Requires a Vault root token
./managed-k8s/tools/vault/mkcluster-root.sh
The script creates the Kubernetes cluster root CA and backs up its private key to
k8s-pki/cluster-root-ca
in the Vault kv2 store of the cluster [3].Build the cluster using the apply-all action
This will copy and configure the Kubernetes cluster root CA key from Vault’s kv2 store on all control plane nodes.
Recreate the Kubernetes cluster root CA and backup its key
You must perform a complete root CA rotation, see Root Certificate Authority Rotation
Executing the preparation phase of ‘tools/vault/rotate-root-ca-root.sh’ will backup the Kubernetes cluster root CA key to
k8s-pki/cluster-root-ca
in the Vault kv2 store of the cluster [3]. when the CA is recreated.Tip
SHORTCUT
If you have access to the current CA key you may skip the root CA rotation and instead manually upload it to Vault’s kv2 store:
clustername="$(tomlq --raw-output .vault.cluster_name config/config.toml)" vault kv put -mount="yaook/${clustername}/kv" k8s-pki/cluster-root-ca private_key=@current-ca.key
Tip
SHORTCUT
If you previously had the fix enabled and disabled it again and since then did not change the Kubernetes cluster root CA you may try to undelete the CA key in Vault’s kv2 store:
clustername="$(tomlq --raw-output .vault.cluster_name config/config.toml)" export VAULT_TOKEN=$root_token ca_key_version="$(vault kv get -format=json -mount=yaook/${clustername}/kv k8s-pki/cluster-root-ca | jq .data.metadata.version)" vault kv undelete -versions="${ca_key_version}" -mount=yaook/${clustername}/kv k8s-pki/cluster-root-ca
Run at least the
k8s-master
tag of the apply-k8s-core actionAFLAGS="--tags k8s-master" ./managed-k8s/actions/apply-k8s-core.sh
This will copy and configure the Kubernetes cluster root CA key from Vault’s kv2 store on all control plane nodes.
Optional: Check that certificate signing is functional again now
See the Kubernetes documentation on how to do that.
The Kubernetes cluster root CA key can be read with
VAULT_TOKEN=$root_token vault kv get -format=json -mount=yaook/${clustername}/kv k8s-pki/cluster-root-ca
.
Disabling the fix
Disable signing requests in the config
tomlq --in-place --toml-output '.kubernetes.controller_manager.enable_signing_requests=false' config/config.toml
Nothing to care about, go ahead as normal
Run at least the
k8s-master
tag of the apply-k8s-core actionNote
Requires a Vault root token
AFLAGS="--tags k8s-master" ./managed-k8s/actions/apply-k8s-core.sh
This will delete and deconfigure the Kubernetes cluster root CA key on all control plane nodes AND delete the backup in Vault’s kv2 store (this requires a Vault root token).
Note
The key backup is not destroyed [4], you may still undelete it again later:
Note
Requires a Vault root token
clustername="$(tomlq --raw-output .vault.cluster_name config/config.toml)" ca_key_version="$(vault kv get -format=json -mount=yaook/${clustername}/kv k8s-pki/cluster-root-ca | jq .data.metadata.version)" vault kv undelete -versions="${ca_key_version}" -mount=yaook/${clustername}/kv k8s-pki/cluster-root-ca
If you wish to completely remove the key backup from Vault, run:
Note
Requires a Vault root token
clustername="$(tomlq --raw-output .vault.cluster_name config/config.toml)" vault kv destroy -mount=yaook/${clustername}/kv \ -versions="0,$( vault kv metadata get -format=json -mount=yaook/${clustername}/kv k8s-pki/cluster-root-ca \ | jq '.data.versions | keys_unsorted[] | tonumber' | tr '\n' ',' )" \ k8s-pki/cluster-root-ca \ && vault kv metadata delete -mount=yaook/${clustername}/kv k8s-pki/cluster-root-ca
https://developer.hashicorp.com/vault/docs/secrets/kv/kv-v2#deleting-and-destroying-data