AWS Secrets Manager in Kubernetes: Secret Rotation & Reloader

Secrets are easy to create in AWS — but consuming them cleanly from a Kubernetes pod is another matter. This is a hands-on guide to wiring Amazon EKS into AWS Secrets Manager and SSM Parameter Store using the Secrets Store CSI Driver and the AWS Secrets and Configuration Provider (ASCP), then closing the loop with automatic rotation and Stakater Reloader so your workloads always run with the latest credentials.

Handling Secrets and Parameters on AWS EKS

Security best practices demand that personal and sensitive data — passwords, tokens, API keys — is properly protected. On AWS these details are typically stored in AWS Secrets Manager or the AWS Systems Manager Parameter Store (SSM Parameter Store).

Secrets are straightforward to create in AWS, but retrieving them from inside a Kubernetes cluster is not. Pods frequently need to pull database credentials, API keys, and similar values at runtime. In this article we look at how to configure EKS to consume secrets and parameters from both Secrets Manager and SSM Parameter Store.

Kubernetes Secrets vs AWS Secrets

In Kubernetes, the native way to store a secret is the Secret kind:

apiVersion: v1
kind: Secret
metadata:
  name: test-secret
type: Opaque
data:
  password: JdkDSIGhhdZ

In a cloud context, however, sourcing secrets externally via a SecretProviderClass offers several advantages:

• A single configuration file lets you manage all secrets from one place.
• Keeping secrets outside the cluster eases integration with external tools, and makes it trivial to share a secret across namespaces and keep it in sync.
• Secret creation can be delegated to an Infrastructure-as-Code (IaC) tool — no need to reach the Kubernetes API (often in private subnets) to create and update secrets.
• Combining Secrets Manager / SSM with Kubernetes service accounts gives fine-grained control over which pods can access which secrets, and lets you define reusable groups of secrets.

Prerequisites

To use Secrets Manager or SSM Parameter Store from Kubernetes, some additional setup is required:

• An IAM policy with permission to retrieve the required secrets.
• Secrets created in Secrets Manager. The supported types are: a plain-text secret, a JSON-formatted secret (the whole JSON is the secret), and an SSM Parameter Store parameter.
• An OpenID Connect (OIDC) identity provider to associate IAM roles with Kubernetes service accounts.
• The Kubernetes Secrets Store CSI Driver installed.
• The AWS Secrets and Configuration Provider (ASCP) installed.
• The Reloader installed.

IAM Role and Policy

Next we create an IAM policy and an IAM role for the service account, using IAM Roles for Service Accounts (IRSA) to scope secret access to individual pods. With IRSA in place, the provider retrieves the pod's identity, exchanges it for an IAM role, and ASCP assumes that role to fetch only the secrets the pod is authorised to access — preventing one container from reading secrets meant for another.

The following policy allows retrieval of secrets from Secrets Manager and SSM Parameter Store, scoped to only the specific secrets that need loading (least privilege), plus use of a KMS key (required if the secrets are encrypted).

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:DescribeSecret",
        "secretsmanager:GetSecretValue"
      ],
      "Resource": [
        "arn:aws:secretsmanager:<AWS_REGION>:<AWS_ACCOUNT_ID>:secret:mySimpleSecret-s8Yb7Y",
        "arn:aws:secretsmanager:<AWS_REGION>:<AWS_ACCOUNT_ID>:secret:myJSONSecret-Uaauu1",
        "arn:aws:secretsmanager:<AWS_REGION>:<AWS_ACCOUNT_ID>:secret:/dev/msk/password-iUq8oR"
      ]
    }
  ]
}

To follow least privilege, we create an IAM role whose trust policy restricts it to a specific EKS cluster, namespace, and service account.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Principal": {
        "Federated": "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/oidc.eks.<AWS_REGION>.amazonaws.com/id/<OIDC_ID>"
      },
      "Condition": {
        "StringEquals": {
          "oidc.eks.<AWS_REGION>.amazonaws.com/id/<OIDC_ID>:aud": "sts.amazonaws.com",
          "oidc.eks.<AWS_REGION>.amazonaws.com/id/<OIDC_ID>:sub": "system:serviceaccount:<K8S_NAMESPACE>:<SERVICE_ACCOUNT_NAME>"
        }
      }
    }
  ]
}

If instead you want to allow all namespaces and service accounts in the cluster, use a trust policy scoped only to the EKS cluster:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Principal": {
        "Federated": "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/oidc.eks.<AWS_REGION>.amazonaws.com/id/<OIDC_ID>"
      },
      "Condition": {
        "StringLike": {
          "oidc.eks.<AWS_REGION>.amazonaws.com/id/<OIDC_ID>:aud": "sts.amazonaws.com",
          "oidc.eks.<AWS_REGION>.amazonaws.com/id/<OIDC_ID>:sub": "system:serviceaccount:*"
        }
      }
    }
  ]
}

Throughout the rest of this article we use test-app-secrets as the namespace and app-admin-account as the service account. Here is the same IAM setup expressed as Terraform:

data "aws_eks_cluster" "eks_cluster" {
  name = "${var.env}-${var.clustername}"
}

data "aws_iam_openid_connect_provider" "eks_oidc_provider" {
  url = data.aws_eks_cluster.eks_cluster.identity[0].oidc[0].issuer
}

data "aws_secretsmanager_secrets" "retrieve_secrets" {
  filter {
    name   = "tag-value"
    values = [var.sub_system]
  }
}

data "aws_iam_policy_document" "app_secret_eks_policy" {
  statement {
    actions = [
      "secretsmanager:GetSecretValue",
      "secretsmanager:DescribeSecret"
    ]
    resources = flatten([data.aws_secretsmanager_secrets.retrieve_secrets.arns])
    effect    = "Allow"
  }
  depends_on = [module.secret_manager_secret]
}

resource "aws_iam_policy" "app_secret_eks_policy" {
  name        = "${var.env}_${var.clustername}_api_readaccess_eks_Policy"
  description = "Custom policy for secret read access"
  policy      = data.aws_iam_policy_document.app_secret_eks_policy.json
}

resource "aws_iam_role" "app_secret_eks_role" {
  name               = "${var.env}-${var.clustername}-api-token-access"
  path               = "/"
  assume_role_policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "${data.aws_iam_openid_connect_provider.eks_oidc_provider.arn}"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringLike": {
        "${replace(data.aws_eks_cluster.eks_cluster.identity[0].oidc[0].issuer, "https://", "")}:sub": "system:serviceaccount:*",
        "${replace(data.aws_eks_cluster.eks_cluster.identity[0].oidc[0].issuer, "https://", "")}:aud": "sts.amazonaws.com"
      }
    }
  }]
}
POLICY
}

resource "aws_iam_policy_attachment" "app_secret_eks_policy_role_attachment" {
  name       = "${var.env}-${var.clustername}-app-secret-policy-attach"
  policy_arn = aws_iam_policy.app_secret_eks_policy.arn
  roles      = [aws_iam_role.app_secret_eks_role.name]
}

The final step is attaching the policy to the role — handled above by the aws_iam_policy_attachment resource.

ServiceAccount

Now we create a ServiceAccount so pods can assume the IAM role. Note that this service account is only available within the specified namespace.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-admin-account
  namespace: test-app-secrets
  annotations:
    eks.amazonaws.com/role-arn: <IAM_SERVICE_ACCOUNT_ROLE_ARN>

SecretProviderClass

To use the Secrets Store CSI Driver you create a SecretProviderClass custom resource. This supplies driver configuration and provider-specific parameters, and defines exactly which secrets a pod can access.

First, create the secrets and parameters we'll reference: a plain-text secret (mySimpleSecret), a JSON-formatted secret (myJSONSecret), and an SSM parameter (/dev/msk/password). Then define the manifest:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aws-secrets-providerclass
  namespace: test-app-secrets
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "mySimpleSecret"
        objectType: "secretsmanager"
      - objectName: "myJSONSecret"
        objectType: "secretsmanager"
      - objectName: "/dev/msk/password"
        objectType: "ssmparameter"

Here we pull two secrets from Secrets Manager (by secret name) and one from SSM Parameter Store (by parameter key). Again, these are only available inside the specified namespace.

Deploying the Solution

Update your Deployment to use the secrets-store.csi.k8s.io driver and reference the SecretProviderClass created above. On pod start (and restart), the CSI driver calls the provider binary, retrieves the secrets from Secrets Manager and Parameter Store, and mounts them into the container's file system.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-hello-world
  namespace: test-app-secrets
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      serviceAccountName: app-admin-account
      volumes:
      - name: mount-secrets-access
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: "aws-secrets-providerclass"
      containers:
      - name: demo-deployment
        image: nginx
        ports:
        - containerPort: 80
        volumeMounts:
        - name: mount-secrets-access
          mountPath: "/mnt/aws-secrets"
          readOnly: true

After deployment, exec into the pod and confirm the secrets are mounted:

$ ls -l /mnt/aws-secrets/
-rw-r--r-- 1 root root 11 Jan 31 23:10 mySQLsecret
-rw-r--r-- 1 root root 74 Jan 31 23:10 mySimpleSecret
-rw-r--r-- 1 root root 72 Jan 31 23:10 myJSONSecret

$ cat /mnt/aws-secrets/mySimpleSecret
this !s N0t P@ssw0rd

$ cat /mnt/aws-secrets/myJSONSecret
{ "username": "usernameSecretValue", "password": "passwordSecretValue" }

For a JSON secret, extracting a single property from the file requires an extra tool such as jq:

$ cat /mnt/aws-secrets/myJSONSecret | jq -r .username
usernameSecretValue

$ cat /mnt/aws-secrets/myJSONSecret | jq -r .password
passwordSecretValue

There's a cleaner way to get these into your app, which we'll cover next.

Secrets as Environment Variables

Reading secrets from a file has its limits — usually we want them as environment variables. To do that, extend the SecretProviderClass with a secretObjects section that syncs the mounted secrets into a native Kubernetes Secret:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aws-secrets-providerclass
  namespace: test-app-secrets
spec:
  provider: aws
  # --- new: sync mounted secrets into a Kubernetes Secret ---
  secretObjects:
    - secretName: eks-local-secrets
      type: Opaque
      data:
        - objectName: mySimpleSecret
          key: simpleSecret
        - objectName: myJSONSecret
          key: jsonSecret
        - objectName: parameterAlias
          key: myParameter
  parameters:
    objects: |
      - objectName: "mySimpleSecret"
        objectType: "secretsmanager"
      - objectName: "myJSONSecret"
        objectType: "secretsmanager"
      - objectName: "/dev/msk/password"
        objectType: "ssmparameter"
        objectAlias: parameterAlias

This creates a Kubernetes secret named eks-local-secrets with three keys: simpleSecret, jsonSecret and parameterAlias.

Then update the Deployment to expose those keys. You have two options.

containers:
- name: demo-deployment
  image: nginx
  env:
    - name: SIMPLE_SECRET_ENV_VAR
      valueFrom:
        secretKeyRef:
          name: eks-local-secrets
          key: simpleSecret
    - name: JSON_SECRET_ENV_VAR
      valueFrom:
        secretKeyRef:
          name: eks-local-secrets
          key: jsonSecret
    - name: MY_PARAMETER
      valueFrom:
        secretKeyRef:
          name: eks-local-secrets
          key: myParameter

containers:
- name: demo-deployment
  image: nginx
  envFrom:
    - secretRef:
        name: eks-local-secrets

With envFrom you don't map each variable individually — every key in the secret becomes an environment variable. Pick whichever suits your use case. Either way, re-deploy and verify:

$ echo $SIMPLE_SECRET_ENV_VAR
this !s N0t P@ssw0rd

$ echo $JSON_SECRET_ENV_VAR
{"username": "usernameSecretValue","password": "passwordSecretValue"}

$ echo $MY_PARAMETER
My parameter

Handling JSON Secrets

Storing raw JSON in an environment variable is rarely ideal. If you must, ensure the value in Secrets Manager is not pretty-printed — newlines, carriage returns and tabs will truncate the variable so it holds only part of the JSON.

Usually, though, you want the username and password from a JSON secret in two separate environment variables. The jmesPath field does exactly this — JMESPath (JSON Matching Expression paths) is a query language for JSON. Update the SecretProviderClass:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aws-secrets-providerclass
  namespace: test-app-secrets
spec:
  provider: aws
  secretObjects:
    - secretName: eks-local-secrets
      type: Opaque
      data:
        - objectName: usernameAlias
          key: username
        - objectName: passwordAlias
          key: password
  parameters:
    objects: |
      - objectName: "myJSONSecret"
        objectType: "secretsmanager"
        jmesPath:
          - path: username
            objectAlias: usernameAlias
          - path: password
            objectAlias: passwordAlias

The secretObjects section now defines two dedicated variables (username and password), whose keys are referenced in the Deployment and whose object names are referenced in the parameters section. Update the Deployment one last time:

env:
- name: USERNAME
  valueFrom:
    secretKeyRef:
      name: eks-local-secrets
      key: username
- name: PASSWORD
  valueFrom:
    secretKeyRef:
      name: eks-local-secrets
      key: password

Exec into the pod to confirm:

$ echo $USERNAME
usernameSecretValue

$ echo $PASSWORD
passwordSecretValue

Note: the pod filesystem now also contains two new files, username and password, holding the individual values.

Secret Rotation and Versioning

Rotating secrets is one of the most important security requirements — especially after data has been exposed. The real value of ASCP is that it keeps Kubernetes secrets synchronised with the AWS secrets automatically. Without it, changing an AWS secret would require recreating the pod to pick up the new value.

To enable rotation, add two properties when installing the Secrets Store CSI Driver:

Rotation is enabled in the driver via the --enable-secret-rotation flag on the secrets-store container (or the Helm parameters above). The polling interval controls how often mounted contents and Kubernetes secrets are refreshed to the newest version — rotation-poll-interval defaults to 2 minutes and can be changed with the rotationPollInterval property.

The driver also creates a SecretProviderClassPodStatus custom resource to track the binding between a pod and a SecretProviderClass, including which secret versions are currently loaded in the pod mount. View them with:

kubectl get secretproviderclasspodstatus <pod_name>

Secrets Reloader

Stakater Reloader watches ConfigMaps and Secrets for changes and, when it detects one, performs a rolling upgrade of the relevant Pods via their Deployment, DaemonSet or StatefulSet — closing the gap that rotation leaves for environment variables.

Annotation for the Secret

For a Deployment that consumes a secret, add a Reloader annotation naming the secret to watch:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-hello-world
  namespace: test-app-secrets
  labels:
    app: nginx
  annotations:
    reloader.stakater.com/auto: "true"
    secret.reloader.stakater.com/reload: eks-local-secrets   # secret to monitor

The default annotation can be changed with the --secret-annotation flag. The annotation also works for DaemonSets, StatefulSets and Rollouts.

Verifying Reloader is Working

There are four easy ways to confirm Reloader is doing its job.

1. From the logs. Check the Reloader logs — if you see entries like these, it's working:

Changes Detected in test-object of type 'SECRET' in namespace: test-reloader
Updated test-resource of type Deployment in namespace: test-reloader

2. By the pod's age. If you know a secret or ConfigMap just changed, check the relevant pod's age — it should have been created a few moments ago.

3. From the Kubernetes Dashboard. After a change, the dashboard should show the affected pod as newly created.

4. From the command line. After a change, confirm the pod was recreated:

kubectl get pods <pod_name> -n <namespace_name>

BootLabs builds secure, automated cloud platforms on AWS and Kubernetes for enterprises. If you're hardening secret management on EKS, our platform team can help you design it end to end.