Distribute Credentials Securely Using Secrets

This page shows how to securely inject sensitive data, such as passwords and encryption keys, into Pods.

Before you begin

You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:

Convert your secret data to a base-64 representation

Suppose you want to have two pieces of secret data: a username my-app and a password 39528$vdg7Jb. First, use a base64 encoding tool to convert your username and password to a base64 representation. Here's an example using the commonly available base64 program:

echo -n 'my-app' | base64
echo -n '39528$vdg7Jb' | base64

The output shows that the base-64 representation of your username is bXktYXBw, and the base-64 representation of your password is Mzk1MjgkdmRnN0pi.

Create a Secret

Here is a configuration file you can use to create a Secret that holds your username and password:

apiVersion: v1
kind: Secret
metadata:
  name: test-secret
data:
  username: bXktYXBw
  password: Mzk1MjgkdmRnN0pi
  1. Create the Secret

    kubectl apply -f https://k8s.io/examples/pods/inject/secret.yaml
    
  2. View information about the Secret:

    kubectl get secret test-secret
    

    Output:

    NAME          TYPE      DATA      AGE
    test-secret   Opaque    2         1m
    
  3. View more detailed information about the Secret:

    kubectl describe secret test-secret
    

    Output:

    Name:       test-secret
    Namespace:  default
    Labels:     <none>
    Annotations:    <none>
    
    Type:   Opaque
    
    Data
    ====
    password:   13 bytes
    username:   7 bytes
    

Create a Secret directly with kubectl

If you want to skip the Base64 encoding step, you can create the same Secret using the kubectl create secret command. For example:

kubectl create secret generic test-secret --from-literal='username=my-app' --from-literal='password=39528$vdg7Jb'

This is more convenient. The detailed approach shown earlier runs through each step explicitly to demonstrate what is happening.

Create a Pod that has access to the secret data through a Volume

Here is a configuration file you can use to create a Pod:

apiVersion: v1
kind: Pod
metadata:
  name: secret-test-pod
spec:
  containers:
    - name: test-container
      image: nginx
      volumeMounts:
        # name must match the volume name below
        - name: secret-volume
          mountPath: /etc/secret-volume
          readOnly: true
  # The secret data is exposed to Containers in the Pod through a Volume.
  volumes:
    - name: secret-volume
      secret:
        secretName: test-secret
  1. Create the Pod:

    kubectl apply -f https://k8s.io/examples/pods/inject/secret-pod.yaml
    
  2. Verify that your Pod is running:

    kubectl get pod secret-test-pod
    

    Output:

    NAME              READY     STATUS    RESTARTS   AGE
    secret-test-pod   1/1       Running   0          42m
    
  3. Get a shell into the Container that is running in your Pod:

    kubectl exec -i -t secret-test-pod -- /bin/bash
    
  4. The secret data is exposed to the Container through a Volume mounted under /etc/secret-volume.

    In your shell, list the files in the /etc/secret-volume directory:

    # Run this in the shell inside the container
    ls /etc/secret-volume
    

    The output shows two files, one for each piece of secret data:

    password username
    
  5. In your shell, display the contents of the username and password files:

    # Run this in the shell inside the container
    echo "$( cat /etc/secret-volume/username )"
    echo "$( cat /etc/secret-volume/password )"
    

    The output is your username and password:

    my-app
    39528$vdg7Jb
    

Modify your image or command line so that the program looks for files in the mountPath directory. Each key in the Secret data map becomes a file name in this directory.

Project Secret keys to specific file paths

You can also control the paths within the volume where Secret keys are projected. Use the .spec.volumes[].secret.items field to change the target path of each key:

apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
  - name: mypod
    image: redis
    volumeMounts:
    - name: foo
      mountPath: "/etc/foo"
      readOnly: true
  volumes:
  - name: foo
    secret:
      secretName: mysecret
      items:
      - key: username
        path: my-group/my-username

When you deploy this Pod, the following happens:

  • The username key from mysecret is available to the container at the path /etc/foo/my-group/my-username instead of at /etc/foo/username.
  • The password key from that Secret object is not projected.

If you list keys explicitly using .spec.volumes[].secret.items, consider the following:

  • Only keys specified in items are projected.
  • To consume all keys from the Secret, all of them must be listed in the items field.
  • All listed keys must exist in the corresponding Secret. Otherwise, the volume is not created.

Set POSIX permissions for Secret keys

You can set the POSIX file access permission bits for a single Secret key. If you don't specify any permissions, 0644 is used by default. You can also set a default POSIX file mode for the entire Secret volume, and you can override per key if needed.

For example, you can specify a default mode like this:

apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
  - name: mypod
    image: redis
    volumeMounts:
    - name: foo
      mountPath: "/etc/foo"
  volumes:
  - name: foo
    secret:
      secretName: mysecret
      defaultMode: 0400

The Secret is mounted on /etc/foo; all the files created by the secret volume mount have permission 0400.

Define container environment variables using Secret data

You can consume the data in Secrets as environment variables in your containers.

If a container already consumes a Secret in an environment variable, a Secret update will not be seen by the container unless it is restarted. There are third party solutions for triggering restarts when secrets change.

Define a container environment variable with data from a single Secret

  • Define an environment variable as a key-value pair in a Secret:

    kubectl create secret generic backend-user --from-literal=backend-username='backend-admin'
    
  • Assign the backend-username value defined in the Secret to the SECRET_USERNAME environment variable in the Pod specification.

    apiVersion: v1
    kind: Pod
    metadata:
      name: env-single-secret
    spec:
      containers:
      - name: envars-test-container
        image: nginx
        env:
        - name: SECRET_USERNAME
          valueFrom:
            secretKeyRef:
              name: backend-user
              key: backend-username
    
  • Create the Pod:

    kubectl create -f https://k8s.io/examples/pods/inject/pod-single-secret-env-variable.yaml
    
  • In your shell, display the content of SECRET_USERNAME container environment variable.

    kubectl exec -i -t env-single-secret -- /bin/sh -c 'echo $SECRET_USERNAME'
    

    The output is similar to:

    backend-admin
    

Define container environment variables with data from multiple Secrets

  • As with the previous example, create the Secrets first.

    kubectl create secret generic backend-user --from-literal=backend-username='backend-admin'
    kubectl create secret generic db-user --from-literal=db-username='db-admin'
    
  • Define the environment variables in the Pod specification.

    apiVersion: v1
    kind: Pod
    metadata:
      name: envvars-multiple-secrets
    spec:
      containers:
      - name: envars-test-container
        image: nginx
        env:
        - name: BACKEND_USERNAME
          valueFrom:
            secretKeyRef:
              name: backend-user
              key: backend-username
        - name: DB_USERNAME
          valueFrom:
            secretKeyRef:
              name: db-user
              key: db-username
    
  • Create the Pod:

    kubectl create -f https://k8s.io/examples/pods/inject/pod-multiple-secret-env-variable.yaml
    
  • In your shell, display the container environment variables.

    kubectl exec -i -t envvars-multiple-secrets -- /bin/sh -c 'env | grep _USERNAME'
    

    The output is similar to:

    DB_USERNAME=db-admin
    BACKEND_USERNAME=backend-admin
    

Configure all key-value pairs in a Secret as container environment variables

  • Create a Secret containing multiple key-value pairs

    kubectl create secret generic test-secret --from-literal=username='my-app' --from-literal=password='39528$vdg7Jb'
    
  • Use envFrom to define all of the Secret's data as container environment variables. The key from the Secret becomes the environment variable name in the Pod.

    apiVersion: v1
    kind: Pod
    metadata:
      name: envfrom-secret
    spec:
      containers:
      - name: envars-test-container
        image: nginx
        envFrom:
        - secretRef:
            name: test-secret
    
  • Create the Pod:

    kubectl create -f https://k8s.io/examples/pods/inject/pod-secret-envFrom.yaml
    
  • In your shell, display username and password container environment variables.

    kubectl exec -i -t envfrom-secret -- /bin/sh -c 'echo "username: $username\npassword: $password\n"'
    

    The output is similar to:

    username: my-app
    password: 39528$vdg7Jb
    

Example: Provide prod/test credentials to Pods using Secrets

This example illustrates a Pod which consumes a secret containing production credentials and another Pod which consumes a secret with test environment credentials.

  1. Create a secret for prod environment credentials:

    kubectl create secret generic prod-db-secret --from-literal=username=produser --from-literal=password=Y4nys7f11
    

    The output is similar to:

    secret "prod-db-secret" created
    
  2. Create a secret for test environment credentials.

    kubectl create secret generic test-db-secret --from-literal=username=testuser --from-literal=password=iluvtests
    

    The output is similar to:

    secret "test-db-secret" created
    
  3. Create the Pod manifests:

    cat <<EOF > pod.yaml
    apiVersion: v1
    kind: List
    items:
    - kind: Pod
      apiVersion: v1
      metadata:
        name: prod-db-client-pod
        labels:
          name: prod-db-client
      spec:
        volumes:
        - name: secret-volume
          secret:
            secretName: prod-db-secret
        containers:
        - name: db-client-container
          image: myClientImage
          volumeMounts:
          - name: secret-volume
            readOnly: true
            mountPath: "/etc/secret-volume"
    - kind: Pod
      apiVersion: v1
      metadata:
        name: test-db-client-pod
        labels:
          name: test-db-client
      spec:
        volumes:
        - name: secret-volume
          secret:
            secretName: test-db-secret
        containers:
        - name: db-client-container
          image: myClientImage
          volumeMounts:
          - name: secret-volume
            readOnly: true
            mountPath: "/etc/secret-volume"
    EOF
    
  4. Apply all those objects on the API server by running:

    kubectl create -f pod.yaml
    

Both containers will have the following files present on their filesystems with the values for each container's environment:

/etc/secret-volume/username
/etc/secret-volume/password

You could further simplify the base Pod specification by using two service accounts:

  1. prod-user with the prod-db-secret
  2. test-user with the test-db-secret

The Pod specification is shortened to:

apiVersion: v1
kind: Pod
metadata:
  name: prod-db-client-pod
  labels:
    name: prod-db-client
spec:
  serviceAccount: prod-db-client
  containers:
  - name: db-client-container
    image: myClientImage

References

What's next

Last modified August 24, 2023 at 6:38 PM PST: Use code_sample shortcode instead of code shortcode (e8b136c3b3)