Managing Secrets With Kubernetes Operator

Feedback
Install the Hanzo KMS Operator on your Kubernetes cluster.
Feedback
Configure Authentication using Kubernetes Service Accounts.
Feedback
Use Each of the three CRDs.
• Hanzo KMSSecret [Sync secrets from Hanzo KMS to Kubernetes]
• Hanzo KMSPushSecret [Sync secrets from Kubernetes to Hanzo KMS]
• Hanzo KMSDynamicSecret [Manage Dynamic Secrets and automatically create time-bound leases]

Prerequisites

Feedback
Installed tools
• helm, git, kubectl
Feedback
Kubernetes Cluster
• Ensure you have access to a running cluster and connect with kubectl
Feedback
PostgreSQL Cluster (for Hanzo KMSDynamicSecret)
• Ensure you have a running database that you have access to
Feedback
Clone kms-guides-source-code repository
Feedback
Access to an Hanzo KMS instance (cloud or self-hosted)

Step-By-Step Guide

The Hanzo KMS Operator runs inside your cluster and is responsible for handling secret synchronization events.

helm repo add kms-helm-charts 'https://dl.cloudsmith.io/public/kms/helm-charts/helm/charts/'
helm repo update
helm install kms-operator kms-helm-charts/secrets-operator

Verify the operator pod is running:

kubectl get pods -n default

The operator uses a Machine Identity to authenticate with Hanzo KMS through the Kubernetes Auth Method

1. Login to Hanzo KMS
2. Select Organization Access from the left navigational pane
3. Create an Identity and give it a name and a role
4. Once the Machine Identity is created, copy the Identity ID (will be used later)
5. Select the created Machine Identity, and add a Kubernetes Authentication Method. Use these configurations:
   • Allowed Service Account Names: kms-service-account, default
   • Allowed Namespaces: default
   • Kubernetes Host URL can be found by running kubectl cluster-info
   • Token Reviewer JWT / CA Certificate: We will be generating these two later and adding them in later, leave it blank for now
6. Once the Machine Identity has been created, navigate back to Overview
7. Now select Add New Project
   • Add a Project Name, and select Secrets Management as the product type
   • Add a description (Optional).
8. Once the Project is created, navigate into the Project to Add Secrets. You can add any key-value pair for this example, however if you want to use the Hanzo KMSSecret CRD example provided in this demo, use the following configurations:
   • Key: SMTP_HOST
   • Value: smtp@gmail.com
   • Tags: N/A (Not needed for this demonstration)
   • Environments: Production
9. Now lets navigate to the Project Access tab on the left hand navigation pane.
   • Add the machine identity we created, and give it Admin permissions (just for demonstration purposes)

Now we will be interacting with the local repository you cloned earlier. Make sure you are in the directory that contains the yaml configurations. Assuming you are in your root user directory:

cd kms-guides-source-code/kubernetes-operator-demo

1. Create the kms-token-reviewer service account. This Manifest creates a service account that the Hanzo KMS Operator uses to authenticate with Kubernetes for token reviews. It allows Hanzo KMS to validate Kubernetes tokens securely during the Machine Identity authentication process.

kubectl apply -f kms-reviewer-service-account.yaml

2. Create the token for the reviewer service account. This Yaml defines a service account token secret linked to the reviewer account created above. It generates a JWT token that Hanzo KMS uses for the Kubernetes Auth Method in your Machine Identity configuration.

kubectl apply -f service-account-reviewer-token.yaml

3. This file binds the kms-token-reviewer service account to the built-in system:auth-delegator ClusterRole. That role allows the service account to perform token review and authentication delegation requests on behalf of other service accounts - a key part of Kubernetes-based identity verification. Without this binding, the Hanzo KMS Operator wouldn't have permission to validate tokens.

kubectl apply -f cluster-role-binding.yaml

4. Create the service account that will be used by the Hanzo KMSSecret. This file creates a dedicated service account kms-service-account that the Hanzo KMS Operator uses to access and sync secrets within your cluster. It operates as the Operator's working identity in your cluster, separate from the token reviewer.

kubectl apply -f kms-service-account.yaml

5. Create the token for the Hanzo KMS service account. This manifest defines a token secret for the kms-service-account. It allows the Hanzo KMS operator to authenticate against Hanzo KMS's API when syncing secrets. The token will then be manually patched and associated with the service account to make sure Kubernetes mains it persistently.

kubectl apply -f kms-service-account-token.yaml

6. Apply the patch to manually associate the token secret

kubectl patch serviceaccount kms-service-account -p '{"secrets": [{"name": "kms-service-account-token"}]}' -n default

7. Create the JWT Token and Certificate and add it to the Machine Identity we created under Kubernetes Auth. For the generated CA, navigate to the Advanced tab to paste the certificate:

• JWT Command

kubectl get secret kms-token-reviewer-token -n default -o jsonpath='{.data.token}' | base64 -d

• CA Command

kubectl get secret kms-token-reviewer-token -n default -o jsonpath='{.data.ca\.crt}' | base64 -d

1. Check to see if the service accounts were created

kubectl get serviceaccount -n default | grep kms

2. Verify the tokens were created and linked

kubectl get secrets -n default | grep kms

The Hanzo KMSSecret CRD tells the operator to sync secrets from Hanzo KMS to Kubernetes. By referencing your identityID, projectSlug, and envSlug, this CRD tells the Hanzo KMS Operator which Hanzo KMS secrets to fetch and how to format them into a Kubernetes Secret. Make sure to edit the provided CRD to match your specific Machine Identity ID, Project ID, and which environment your secrets are being pulled from (default is prod).

• Project Slug: Can be found when you select your project and navigate to settings
• Identity ID: Can be found when you select your machine identity from your organization's access control

1. After editing the example-kms-secret-crd.yaml to contain your demo-specific values, apply the yaml in your cluster

kubectl apply -f example-kms-secret-crd.yaml

1. Check that the Hanzo KMSSecret was created successfully

kubectl get kmssecret -n default

2. Check that the operator created the managed-secret

kubectl get secret managed-secret -n default

3. View the secret contents (base64 encoded)

kubectl get secret managed-secret -n default -o jsonpath='{.data}' | jq

1. Deploy the nginx demo deployment that will use the managed secret.

kubectl apply -f demo-deployment.yaml

2. Wait 15-20 seconds and then verify the deployments

kubectl get deployments
kubectl get pods -l app=nginx

1. Check that the environment variable is in the running pod. If everything was successful, at this point you should be able to see the secret populate in the kubernetes pod and have a successful sync from Hanzo KMS to Kubernetes.

kubectl exec -it $(kubectl get pod -l app=nginx -o jsonpath='{.items[0].metadata.name}') -- env | grep SMTP

Now that we have successfully synced secrets from Hanzo KMS to Kubernetes, lets explore how we can push Kubernetes Secrets to Hanzo KMS.

1. Either create a Kubernetes Secret via yaml, or use the one in the repository.

kubectl apply -f source-secret.yaml

2. Verify creation of the secret

kubectl get secret push-secret-demo -n default -o yaml

The Hanzo KMSPushSecret CRD tells the operator to sync secrets from Kubernetes to Hanzo KMS. Make sure you edit the CRD to include the specific Project Slug, and Identity ID. The other values present in example-push-secret.yaml should be configured based on the previously committed yaml configurations.

1. Apply the Hanzo KMSPushSecret CRD provided after making the necessary changes

kubectl apply -f example-push-secret-crd.yaml

2. Once your CRD has been configured, go back to your project within Hanzo KMS and check to see if your secrets have populated there.

The Hanzo KMSDynamicSecret CRD allows you to sync dynamic secrets and create leases automatically in Kubernetes as native Kubernetes Secret resources Any Pod, Deployment, or other Kubernetes resource can make use of dynamic secrets from Hanzo KMS just like any other Kubernetes secret.

1. Navigate to your Hanzo KMS Project and click on the dropdown next to Add Secret. From here you will select Add Dynamic Secret
   • Select SQL Database as the service you would like to connect to.
   • Select PostegreSQL as the database service. Enter in the connection details for your database, specifically the Host, Port, User, Password, and Database Name.
   • For the Secret Name, if you want to use the same name as the one in the cloned Hanzo KMSDynamicSecret CRD, use the name dynamic-secret-lease. Otherwise you will need to change the dynamicSecret.secretName config in the Hanzo KMSDynamicSecret CRD to whatever you name the secret here.
   • In the CA (SSL) section, make sure to upload the CA Certificate for your database.
   • Finally, select Prod as the environment (we are keeping this configuration as part of the demonstration).

2. Edit the dynamic-secret-crd with the proper machine Identity ID, Project Slug, dynamicSecret.secretName (same as the Secret Name you gave to the dynamic secret in Hanzo KMS), and managedSecretReference.secretName (name of the kubernetes secret that Hanzo KMS Operator will create/populate in the cluster).

   • If you want to keep the managedSecretReference.secretName then you can leave it as dynamic-secret-test

3. Once the changes have been saved, apply the yaml:

kubectl apply -f dynamic-secret-crd.yaml

4. After applying the CRD, you should notice that the dynamic secret lease has been created and synced with your cluster. Verify by running:

kubectl get secret dynamic-secret-test -n default -o yaml

5. Once the dynamic secret lease has been created, you should see that the secret has data that contains the lease credentials.