Kubernetes Operator

How to use Hanzo KMS to inject, push, and manage secrets within Kubernetes clusters

The Hanzo KMS Operator is a collection of Kubernetes controllers that streamline how secrets are managed between Hanzo KMS and your Kubernetes cluster. It provides multiple Custom Resource Definitions (CRDs) which enable you to:

Sync secrets from Hanzo KMS into Kubernetes (Hanzo KMSSecret).

Push new secrets from Kubernetes to Hanzo KMS (Hanzo KMSPushSecret).

Manage dynamic secrets and automatically create time-bound leases (Hanzo KMSDynamicSecret).

When these CRDs are configured, the Hanzo KMS Operator will continuously monitors for changes and performs necessary updates to keep your Kubernetes secrets up to date. It can also automatically reload dependent Deployments resources whenever relevant secrets are updated.

If you are already using the External Secrets operator, you can view the integration documentation for it here.

The following Kubernetes minor releases are currently supported. The latest operator version is tested against each Kubernetes version. It may work with other versions of Kubernetes, but those are not officially supported.

1.33
1.32
1.31
1.30
1.29

The Hanzo KMS Kubernetes Operator has been tested successfully in the following hosted Kubernetes environments:

Amazon Elastic Kubernetes Service (EKS)
Google Kubernetes Engine (GKE)
Microsoft Azure Kubernetes Service (AKS)
Oracle Container Engine for Kubernetes (OKE)
Red Hat OpenShift

It may work in other Kubernetes distributions, but those are not officially supported. Please report any issues here.

Install

The operator can be installed via Helm. Helm is a package manager for Kubernetes that allows you to define, install, and upgrade Kubernetes applications.

Install the latest Helm repository

helm repo add kms-helm-charts 'https://dl.cloudsmith.io/public/kms/helm-charts/helm/charts/'

helm repo update

The operator can be installed either cluster-wide or restricted to a specific namespace. If you require stronger isolation and stricter access controls, a namespace-scoped installation may make more sense.

When installing the operator cluster-wide, the operator will watch and manage CRDs across all namespaces in the cluster. This is the default installation method and the quickest way to get started with using the operator. Cluster-wide installations are useful for:

Simplified Management: A single operator instance manages secrets across all namespaces
Centralized Operations: One deployment to monitor, update, and maintain
Cross-Namespace Flexibility: Easily manage secrets for applications spanning multiple namespaces
Quick Setup: Works out of the box with no additional RBAC configuration required

helm install --generate-name kms-helm-charts/secrets-operator

The operator can be configured to watch and manage secrets in a specific namespace instead of having cluster-wide access. This is useful for:

Enhanced Security: Limit the operator's permissions to only specific namespaces instead of cluster-wide access
Multi-tenant Clusters: Run separate operator instances for different teams or applications
Resource Isolation: Ensure operators in different namespaces don't interfere with each other
Development & Testing: Run development and production operators side by side in isolated namespaces

For multiple namespace-scoped installations, only the first installation should install CRDs. Subsequent installations should set installCRDs: false to avoid conflicts as CRDs are cluster-wide resources.

Single Namespace

# First namespace installation (with CRDs)
helm install operator-namespaced kms-helm-charts/secrets-operator \
  --namespace single-namespace \
  --set scopedNamespaces=single-namespace \
  --set scopedRBAC=true

  scopedNamespaces: single-namespace
  scopedRBAC: true
  installCRDs: true

Multiple Namespaces

 helm install operator-1 kms-helm-charts/secrets-operator \
  --namespace ns1 \
  --set scopedNamespaces=ns1 \
  --set scopedRBAC=true \
  --set installCRDs=true # Only install CRDs once in the cluster (default is true)

helm install operator-namespace2 kms-helm-charts/secrets-operator \
  --namespace ns2 \
  --set scopedNamespaces=ns2 \
  --set scopedRBAC=true \
  --set installCRDs=false # Do not install CRDs in subsequent namespace installations

  scopedNamespaces: ns1
  scopedRBAC: true
  installCRDs: false

  scopedNamespaces: ns2
  scopedRBAC: true
  installCRDs: false

Multiple namespaces with one operator installation

  helm install operator kms-helm-charts/secrets-operator \
  --namespace operator-namespace \
  --set "scopedNamespaces={ns1,ns2,ns3}" \
  --set scopedRBAC=true

  scopedNamespaces:
    - ns1
    - ns2
    - ns3
  scopedRBAC: true

Using your own service account

By default a service account is created for the operator based on the operator release name. You can bring your own service account by setting controllerManager.serviceAccount.create to false and setting controllerManager.serviceAccount.name to the name of the service account you want to use in your values.yaml file.

Example values.yaml file:

controllerManager:
  serviceAccount:
    create: false
    name: my-service-account
# other values...

Please note that if you set controllerManager.serviceAccount.create to false, the service account needs to already exist in the namespace you are installing the operator in.

Custom service accounts are supported in chart version 0.10.11 and above. Please upgrade your helm chart to 0.10.11 or above before attempting to use custom service accounts.

Custom Resource Definitions

Currently the operator supports the following CRD's. We are constantly expanding the functionality of the operator, and this list will be updated as new CRD's are added.

Hanzo KMSSecret: Sync secrets from Hanzo KMS to a Kubernetes secret.

Hanzo KMSPushSecret: Push secrets from a Kubernetes secret to Hanzo KMS.

Hanzo KMSDynamicSecret: Sync dynamic secrets and create leases automatically in Kubernetes.

Metrics and Prometheus

The operator exposes Prometheus metrics on /metrics for monitoring reconciliation performance, errors, and resource utilization.

Configuration

Enable the ServiceMonitor during installation. This will create a prometheus ServiceMonitor resource in the same namespace as the operator.

telemetry:
  serviceMonitor:
    enabled: true
    # ... other telemetry configuration (optional) ...

Enable ServiceMonitor for Prometheus Operator. Defaults to false.

Additional labels for ServiceMonitor. Defaults to {}.

Scheme to use for the ServiceMonitor. Defaults to https.

Port to use for the ServiceMonitor. Defaults to https.

Path to use for the ServiceMonitor. Defaults to /metrics.

Scrape interval. Defaults to 30s.

Scrape timeout. Defaults to 10s.

Bearer token file. Defaults to /var/run/secrets/kubernetes.io/serviceaccount/token.

telemetry:
  serviceMonitor:
    enabled: true

    selectors: {}
    scheme: https
    port: https
    path: /metrics
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 30s
    scrapeTimeout: 10s

Available Metrics

The operator exposes standard controller-runtime metrics. For a complete list of available metrics, see the Kubebuilder metrics reference.

Key metrics to monitor:

controller_runtime_reconcile_total - Reconciliation count

controller_runtime_reconcile_errors_total - Error count

controller_runtime_reconcile_time_seconds - Reconciliation duration

Controllers: Hanzo KMSSecret, Hanzo KMSPushSecret, Hanzo KMSDynamicSecret

Example Prometheus Setup

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace

helm repo add kms-helm-charts 'https://dl.cloudsmith.io/public/kms/helm-charts/helm/charts/'

helm install kms-secrets-operator kms-helm-charts/secrets-operator \
  --set telemetry.serviceMonitor.enabled=true

kubectl get servicemonitor

Check that the ServiceMonitor appears in your operator's namespace.

kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090

Open http://localhost:9090/targets and verify the operator target shows UP.

Example Queries

# Total reconciliations
controller_runtime_reconcile_total

# P99 latency
histogram_quantile(0.99, rate(controller_runtime_reconcile_time_seconds_bucket[5m]))

# Memory usage (MB)
process_resident_memory_bytes / 1024 / 1024

General Configuration

Private/self-signed certificate

To connect to Hanzo KMS instances behind a private/self-signed certificate, you can configure the TLS settings in the CRD to point to a CA certificate stored in a Kubernetes secret resource.

---
spec:
  hostAPI: https://app.kms.hanzo.ai/api
  tls:
    caRef:
      secretName: custom-ca-certificate
      secretNamespace: default
      key: ca.crt
---

Advanced Templating

With the KMS Secrets Operator, you can use templating to dynamically generate secrets in Kubernetes. The templating is built on top of Go templates, which is a powerful and flexible template engine built into Go.

Please be aware that trying to reference non-existing keys will result in an error. Additionally, each template field is processed individually, which means one template field cannot reference another template field.

Please note that templating is currently only supported for the Hanzo KMSPushSecret and Hanzo KMSSecret CRDs.

Available helper functions

The KMS Secrets Operator exposes a wide range of helper functions to make it easier to work with secrets in Kubernetes.

Function	Description	Signature
`decodeBase64ToBytes`	Given a base64 encoded string, this function will decode the base64-encoded string.	`decodeBase64ToBytes(encodedString string) string`
`encodeBase64`	Given a string, this function will encode the string to a base64 encoded string.	`encodeBase64(plainString string) string`
`pkcs12key`	Extracts all private keys from a PKCS#12 archive and encodes them in PKCS#8 PEM format.	`pkcs12key(input string) string`
`pkcs12keyPass`	Same as pkcs12key. Uses the provided password to decrypt the PKCS#12 archive.	`pkcs12keyPass(pass string, input string) string`
`pkcs12cert`	Extracts all certificates from a PKCS#12 archive and orders them if possible. If disjunct or multiple leaf certs are provided they are returned as-is. Sort order: `leaf / intermediate(s) / root`.	`pkcs12cert(input string) string`
`pkcs12certPass`	Same as `pkcs12cert`. Uses the provided password to decrypt the PKCS#12 archive.	`pkcs12certPass(pass string, input string) string`
`pemToPkcs12`	Takes a PEM encoded certificate and key and creates a base64 encoded PKCS#12 archive.	`pemToPkcs12(cert string, key string) string`
`pemToPkcs12Pass`	Same as `pemToPkcs12`. Uses the provided password to encrypt the PKCS#12 archive.	`pemToPkcs12Pass(cert string, key string, pass string) string`
`fullPemToPkcs12`	Takes a PEM encoded certificates chain and key and creates a base64 encoded PKCS#12 archive.	`fullPemToPkcs12(cert string, key string) string`
`fullPemToPkcs12Pass`	Same as `fullPemToPkcs12`. Uses the provided password to encrypt the PKCS#12 archive.	`fullPemToPkcs12Pass(cert string, key string, pass string) string`
`filterPEM`	Filters PEM blocks with a specific type from a list of PEM blocks..	`filterPEM(pemType string, input string) string`
`filterCertChain`	Filters PEM block(s) with a specific certificate type (`leaf`, `intermediate` or `root`) from a certificate chain of PEM blocks (PEM blocks with type `CERTIFICATE`).	`filterCertChain(certType string, input string) string`
`jwkPublicKeyPem`	Takes an json-serialized JWK and returns an PEM block of type `PUBLIC KEY` that contains the public key. See here for details.	`jwkPublicKeyPem(jwkjson string) string`
`jwkPrivateKeyPem`	Takes an json-serialized JWK and returns an PEM block of type `PRIVATE KEY` that contains the private key. See here for details.	`jwkPrivateKeyPem(jwkjson string) string`
`toYaml`	Takes an interface, marshals it to yaml. It returns a string, even on marshal error (empty string).	`toYaml(v any) string`
`fromYaml`	Function converts a YAML document into a `map[string]any`.	`fromYaml(str string) map[string]any`

Sprig functions

The KMS Secrets Operator integrates with the Sprig library to provide additional helper functions.

We've removed expandEnv and env from the supported functions for security reasons.

Global configuration

To configure global settings that will apply to all CRD instances (Hanzo KMSSecret, Hanzo KMSPushSecret, and Hanzo KMSDynamicSecret), you can define these configurations in a Kubernetes ConfigMap. For example, you can configure all CRD instances to fetch secrets from a single backend API without specifying the hostAPI parameter for each instance.

Available global properties

Property	Description	Default value
hostAPI	If `hostAPI` in a CRD instance is left empty, this value will be used	https://app.kms.hanzo.ai/api
tls.caRef.secretName	If `tls.caRef.secretName` in a CRD instance is left empty, this value will be used	-
tls.caRef.secretNamespace	If `tls.caRef.secretNamespace` in a CRD instance is left empty, this value will be used	-
tls.caRef.key	If `tls.caRef.key` in a CRD instance is left empty, this value will be used	-

Applying global configurations

All global configurations must reside in a Kubernetes ConfigMap named kms-config in the namespace kms-operator-system. To apply global configuration to the operator, copy the following yaml into kms-config.yaml file.

apiVersion: v1
kind: Namespace
metadata:
  name: kms-operator-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: kms-config
  namespace: kms-operator-system
data:
  hostAPI: https://example.com/api # <-- global hostAPI
  tls.caRef.secretName: custom-ca-certificate # <-- global TLS CA secret name
  tls.caRef.secretNamespace: default # <-- global TLS CA secret namespace
  tls.caRef.key: ca.crt # <-- global TLS CA secret key

Then apply this change via kubectl by running the following

kubectl apply -f kms-config.yaml

Troubleshoot operator

If the operator is unable to fetch secrets from the API, it will not affect the managed Kubernetes secret. It will continue attempting to reconnect to the API indefinitely. The Hanzo KMSSecret resource uses the status.conditions field to report its current state and any errors encountered.

$ kubectl get kmsSecrets
NAME                     AGE
kmssecret-sample   12s

$ kubectl describe kmsSecret kmssecret-sample
...
Spec:
...
Status:
  Conditions:
    Last Transition Time:  2022-12-18T04:29:09Z
    Message:               Hanzo KMS controller has located the Hanzo KMS token in provided Kubernetes secret
    Reason:                OK
    Status:                True
    Type:                  secrets.kms.hanzo.ai/LoadedHanzo KMSToken
    Last Transition Time:  2022-12-18T04:29:10Z
    Message:               Failed to update secret because: 400 Bad Request
    Reason:                Error
    Status:                False
    Type:                  secrets.kms.hanzo.ai/ReadyToSyncSecrets
Events:                    <none>

Uninstall Operator

The managed secret created by the operator will not be deleted when the operator is uninstalled.

Uninstall Hanzo KMS Helm repository

helm uninstall <release name>

On this page