Securely Manage EKS Secrets with AWS Secrets Manager

Introduction

In the world of container orchestration, managing sensitive information like database credentials, API keys, and tokens is a big challenge. Storing these secrets directly in container images or Kubernetes manifests is not secure. This article explores a strong solution: using AWS Secrets Manager with Amazon Elastic Kubernetes Service (EKS). We will look at how to use the AWS Secrets and Configuration Provider (ASCP) for the Kubernetes Secrets Store CSI Driver to safely bring secrets into EKS pods.

The Core Idea: AWS Secrets Manager and the CSI Driver

AWS Secrets Manager is a service that helps protect access to applications, services, and IT resources. It allows for easy rotation, management, and retrieval of secrets. When working with EKS, the best way to use these secrets is through the Secrets Store Container Storage Interface (CSI) driver.

The CSI driver allows Kubernetes to mount secrets from external stores, like AWS Secrets Manager, as volumes inside pods. The AWS Secrets and Configuration Provider (ASCP) is a component that works with the CSI driver to get secrets from AWS Secrets Manager and present them to the EKS pods. This method avoids storing secret data directly in Kubernetes.

How It Works: The Big Picture

Let's look at the process from a high level. An application running in an EKS pod needs a secret. Instead of that pod talking directly to AWS Secrets Manager, it uses a special volume. The Secrets Store CSI driver sees this and, with the help of the ASCP, contacts AWS Secrets Manager. To do this securely, it uses an IAM Role for Service Accounts (IRSA). This gives the pod a specific IAM role with permission to access only the secrets it needs. The secret is then mounted as a file in the pod's filesystem.

Here is a diagram to show this flow:

Step 1: Set Up the Secrets Store CSI Driver

First, we need to install the Secrets Store CSI driver in the EKS cluster. This can be done using Helm, a package manager for Kubernetes.

The following commands add the necessary Helm repository and install the driver:

helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver --namespace kube-system

Step 2: Install the AWS Secrets and Configuration Provider (ASCP)

After the CSI driver is installed, the ASCP must be deployed. This provider is what allows the CSI driver to communicate with AWS Secrets Manager.

A simple kubectl apply command can install the provider from a manifest file:

kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml

Step 3: Configure IAM Roles for Service Accounts (IRSA)

To allow pods to securely access AWS secrets, we use IAM Roles for Service Accounts (IRSA). This involves creating an IAM role with a policy that grants permission to access a specific secret. Then, this IAM role is associated with a Kubernetes service account.

First, create an IAM policy that allows access to the secret. For example: 

{    "Version": "2012-10-17",    "Statement": [        {            "Effect": "Allow",            "Action": "secretsmanager:GetSecretValue",            "Resource": "arn:aws:secretsmanager:REGION:ACCOUNT_ID:secret:SECRET_NAME-???????"        }    ]}

Next, create an IAM role and attach this policy. Then, create a Kubernetes service account and link it to the IAM role using an annotation.

Step 4: Create a SecretProviderClass

The SecretProviderClass is a custom Kubernetes resource that tells the CSI driver how to get the secrets. It specifies the provider (AWS) and which secrets to get.

Here is an example of a SecretProviderClass manifest:


apiVersion: secrets-store.csi.x-k8s.io/v1kind: SecretProviderClassmetadata:  name: my-app-secretsspec:  provider: aws  parameters:    objects: |      - objectName: "my-database-credentials"        objectType: "secretsmanager"

This manifest tells the driver to get the secret named my-database-credentials from AWS Secrets Manager.

Step 5: Mount the Secrets in a Pod

The final step is to update the application's deployment to use the secrets. This is done by mounting a volume that uses the CSI driver and references the SecretProviderClass.

Here is a part of a pod specification showing how to mount the volume:


spec:  serviceAccountName: my-app-service-account # The service account with the IAM role  containers:  - name: my-app-container    image: my-app-image    volumeMounts:    - name: secrets-volume      mountPath: "/mnt/secrets"      readOnly: true  volumes:  - name: secrets-volume    csi:      driver: secrets-store.csi.k8s.io      readOnly: true      volumeAttributes:        secretProviderClass: "my-app-secrets"

Once the pod starts, the secret values will be available as files in the /mnt/secrets directory inside the container.

Conclusion

Integrating AWS Secrets Manager with EKS using the Secrets Store CSI Driver provides a secure and manageable way to handle secrets. This approach keeps sensitive data out of code and Kubernetes manifests. By using IAM Roles for Service Accounts, access to secrets is tightly controlled, following the principle of least privilege. This method not only improves security but also simplifies secret management in a cloud-native environment.