Verifying images in a private Amazon ECR with Kyverno and IAM Roles for Service Accounts (IRSA)
When running workloads in Amazon Elastic Kubernetes Service (EKS), it is essential to ensure supply chain security by verifying container image signatures and other metadata. To achieve this, you can configure Kyverno, a CNCF policy engine designed for Kubernetes, to pull from ECR private registries for image verification. It’s possible to pass in the credentials via secrets, but that can get difficult to manage and automate across multiple clusters. In this blog post, we will explore an alternative method that simplifies the authentication process by leveraging Kyverno and IRSA (IAM Roles for Service Accounts) in EKS for image verification.
Applications, such as Kyverno, running within a Pod’s containers can utilize the AWS SDK to make API requests to AWS services by leveraging AWS Identity and Access Management (IAM) permissions. IAM roles for service accounts enable the management of credentials for these applications. Instead of manually creating and distributing AWS credentials to the containers, you can associate an IAM role with a Kubernetes service account and configure your Pods to utilize this service account. The detailed steps for this process can be found in the documentation. In this blog, we will guide you through the complete process of enabling IAM roles for the Kyverno service account and demonstrate how to verify this using the Kyverno verifyImages rule.
Setting up the EKS Cluster
First, you need to create an EKS cluster. You can then use the AWS CLI to update the kubeconfig file with the cluster details:
1$ aws eks update-kubeconfig --region us-west-2 --name kyverno-irsa
2Added new context arn:aws:eks:us-west-2:xxxxxxxxxxxx:cluster/kyverno-irsa to /Users/kyverno/.kube/config
Once the kubeconfig is updated, you can verify the cluster by running the following command:
1$ kubectl get node
2NAME STATUS ROLES AGE VERSION
3ip-172-31-56-181.us-west-2.compute.internal Ready <none> 1h v1.27.3-eks-a5565ad
Note: when you use IRSA, it updates the credential chain of the pod to use the IRSA token, however, the pod can still inherit the rights of the instance profile assigned to the worker node. You need to block access to instance metadata to prevent pods that do not use IRSA from inheriting the role assigned to the worker node.
You can follow this guidance to restrict access via the following command, for example:
1aws ec2 modify-instance-metadata-options --instance-id <instance-id> --http-tokens required --http-put-response-hop-limit 1
Installing Kyverno
Once you have the cluster set up, you can use Helm to install Kyverno into the cluster:
1helm upgrade --install kyverno kyverno/kyverno --namespace kyverno --create-namespace
Enabling IAM roles for service accounts
Creating an IAM OIDC Provider for the Cluster
To enable IRSA, you need to create an IAM OIDC provider for the EKS cluster. You can retrieve the OIDC issuer URL using the AWS CLI. The following command retrieves the provider for the cluster kyverno-irsa, replace it with your own cluster name:
1export cluster_name=kyverno-irsa
2oidc_id=$(aws eks describe-cluster --name $cluster_name --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5)
Determine whether an IAM OIDC provider with your cluster’s ID is already in your account.
1aws iam list-open-id-connect-providers | grep $oidc_id | cut -d "/" -f4
If output is returned, then you already have an IAM OIDC provider for your cluster. If no output is returned, then there’s no OIDC provider is associated with the cluster. You can create one using the eksctl command:
1$ eksctl utils associate-iam-oidc-provider --cluster $cluster_name --approve
22023-08-14 21:16:31 [ℹ] will create IAM Open ID Connect provider for cluster "kyverno-irsa" in "us-west-2"
32023-08-14 21:16:33 [✔] created IAM Open ID Connect provider for cluster "kyverno-irsa" in "us-west-2"
Configuring a Kubernetes Service Account to Assume an IAM Role
To associate an IAM role with a Kubernetes service account, you need to create an IAM policy for your IAM role. If you want to associate an existing IAM policy, you can skip this step.
Setup a custom policy with the following permissions, note that in production its best to not use a wildcard and specify resources:
1cat >notation-signer-policy.json <<EOF
2{
3 "Version": "2012-10-17",
4 "Statement": [
5 {
6 "Effect": "Allow",
7 "Action": [
8 "signer:GetSigningProfile",
9 "signer:ListSigningProfiles",
10 "signer:SignPayload",
11 "signer:GetRevocationStatus",
12 "signer:DescribeSigningJob",
13 "signer:ListSigningJobs"
14 ],
15 "Resource": "*"
16 }
17 ]
18}
19EOF
Create the IAM policy:
1aws iam create-policy --policy-name notation-signer-policy --policy-document file://notation-signer-policy.json
To configure a Kubernetes service account to assume an IAM role, you can use the eksctl command to create an IAM service account.
If your Kyverno is installed with default configurations, you can run the following command directly to create the IAM service account. Otherwise, replace the service account name and namespace with your custom values.
1$ eksctl create iamserviceaccount --override-existing-serviceaccounts kyverno-admission-controller --namespace kyverno --cluster kyverno-irsa --role-name kyverno-irsa --attach-policy-arn arn:aws:iam::xxxxxxxxxxxx:policy/notation-signer-policy --approve
22023-08-14 21:18:17 [ℹ] 1 iamserviceaccount (kyverno/kyverno-admission-controller) was included (based on the include/exclude rules)
32023-08-14 21:18:17 [!] metadata of serviceaccounts that exist in Kubernetes will be updated, as --override-existing-serviceaccounts was set
42023-08-14 21:18:17 [ℹ] 1 task: {
5 2 sequential sub-tasks: {
6 create IAM role for serviceaccount "kyverno/kyverno-admission-controller",
7 create serviceaccount "kyverno/kyverno-admission-controller",
8 } }2023-08-14 21:18:17 [ℹ] building iamserviceaccount stack "eksctl-kyverno-irsa-addon-iamserviceaccount-kyverno-kyverno-admission-controller"
92023-08-14 21:18:17 [ℹ] deploying stack "eksctl-kyverno-irsa-addon-iamserviceaccount-kyverno-kyverno-admission-controller"
102023-08-14 21:18:18 [ℹ] waiting for CloudFormation stack "eksctl-kyverno-irsa-addon-iamserviceaccount-kyverno-kyverno-admission-controller"
112023-08-14 21:18:54 [ℹ] waiting for CloudFormation stack "eksctl-kyverno-irsa-addon-iamserviceaccount-kyverno-kyverno-admission-controller"
122023-08-14 21:18:55 [ℹ] serviceaccount "kyverno/kyverno-admission-controller" already exists
132023-08-14 21:18:55 [ℹ] updated serviceaccount "kyverno/kyverno-admission-controller"
After creating the IAM service account, you can verify that the role and service account are configured correctly.
Confirm that the IAM role’s trust policy is configured correctly:
1$ aws iam get-role --role-name kyverno-irsa --query Role.AssumeRolePolicyDocument
2{
3 "Version": "2012-10-17",
4 "Statement": [
5 {
6 "Effect": "Allow",
7 "Principal": {
8 "Federated": "arn:aws:iam::xxxxxxxxxxxx:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/2EA2DE9A6C72778FA517C24D7BBE2916"
9 },
10 "Action": "sts:AssumeRoleWithWebIdentity",
11 "Condition": {
12 "StringEquals": {
13 "oidc.eks.us-west-2.amazonaws.com/id/2EA2DE9A6C72778FA517C24D7BBE2916:aud": "sts.amazonaws.com",
14 "oidc.eks.us-west-2.amazonaws.com/id/2EA2DE9A6C72778FA517C24D7BBE2916:sub": "system:serviceaccount:kyverno:kyverno-admission-controller"
15 }
16 }
17 }
18 ]
19}
Confirm that the policy that you attached to your role in a previous step is attached to the role:
1$ aws iam list-attached-role-policies --role-name kyverno-irsa --query AttachedPolicies --output text
2arn:aws:iam::xxxxxxxxxxxx:policy/notation-signer-policy notation-signer-policy
Confirm that the Kyverno service account is annotated with the role:
1$ kubectl describe serviceaccount kyverno-admission-controller -n kyverno
2
3Name: kyverno-admission-controller
4Namespace: kyverno
5Annotations: eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxxxx:role/kyverno-irsa
Confirm that the environment variables are injected to the admission controller:
1$ kubectl get pod -n kyverno -l app.kubernetes.io/component=admission-controller -o yaml | grep AWS -A2
2 - name: AWS_STS_REGIONAL_ENDPOINTS
3 value: regional
4 - name: AWS_DEFAULT_REGION
5 value: us-west-2
6 - name: AWS_REGION
7 value: us-west-2
8 - name: AWS_ROLE_ARN
9 value: arn:aws:iam::xxxxxxxxxxxx:role/kyverno-irsa
10 - name: AWS_WEB_IDENTITY_TOKEN_FILE
11 value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
If you do not see these environment variables, try restarting the pod to inject variables. If you still do not see these variables, follow this instruction to verify that your pod identity webhook configuration exists and is valid.
Verifying ECR private images using IRSA and Kyverno
To test IRSA works with Kyverno, you can create pods with signed and unsigned images respectively and verify container images signatures using the Kyverno policy. If the IAM role assumption is configured correctly, the pod should be deployed successfully. Otherwise, Kyverno will deny the request.
The test image used in this blog is signed by Notary. If you don’t have a signed images for testing, you can follow this guidance to sign a private ECR image using Notation.
You can inspect all signatures with Notation. The following is an inspection result of all signatures and signed artifacts for the test image xxxxxxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/test-shuting-notation:v1:
1✗ notation inspect xxxxxxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/test-shuting-notation:v1
2Inspecting all signatures for signed artifact
3xxxxxxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/test-shuting-notation@sha256:b31bfb4d0213f254d361e0079deaaebefa4f82ba7aa76ef82e90b4935ad5b105
4└── application/vnd.cncf.notary.signature
5 └── sha256:86abf8af48c152f871a5ea56a62a9302e145760089db926420e72c1bbd0de07d
6 ├── media type: application/jose+json
7 ├── signature algorithm: RSASSA-PSS-SHA-256
8 ├── signed attributes
9 │ ├── signingScheme: notary.x509
10 │ └── signingTime: Fri Aug 11 16:37:40 2023
11 ├── user defined attributes
12 │ └── (empty)
13 ├── unsigned attributes
14 │ └── signingAgent: Notation/1.0.0
15 ├── certificates
16 │ └── SHA256 fingerprint: da1f2d7d648dfacc7ebd59f98a9f35c753c331d80ca4280bb94060f4af4a5357
17 │ ├── issued to: CN=test,O=Notary,L=Seattle,ST=WA,C=US
18 │ ├── issued by: CN=test,O=Notary,L=Seattle,ST=WA,C=US
19 │ └── expiry: Thu May 19 21:15:18 2033
20 └── signed artifact
21 ├── media type: application/vnd.docker.distribution.manifest.v2+json
22 ├── digest: sha256:b31bfb4d0213f254d361e0079deaaebefa4f82ba7aa76ef82e90b4935ad5b105
23 └── size: 938
The following policy verifies the image signature for pods in test-shuting namespace, you can tune the policy to verify different images:
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 annotations:
5 pod-policies.kyverno.io/autogen-controllers: none
6 name: test-irsa
7spec:
8 background: true
9 rules:
10 - match:
11 resources:
12 kinds:
13 - Pod
14 namespaces:
15 - test-shuting
16 name: check-digest
17 verifyImages:
18 - attestors:
19 - count: 1
20 entries:
21 - certificates:
22 cert: |-
23 -----BEGIN CERTIFICATE-----
24 ...
25 ...
26 ...
27 -----END CERTIFICATE-----
28 imageReferences:
29 - xxxxxxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/test-shuting*
30 mutateDigest: true
31 required: true
32 type: Notary
33 verifyDigest: true
34 validationFailureAction: Enforce
35 webhookTimeoutSeconds: 30
Once the policy is installed in the cluster, you can create the pod using the signed image and check the creation passes through:
1$ kubectl -n test-shuting run test --image=xxxxxxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/test-shuting-notation:v1 --dry-run=server
2pod/test created (server dry run)
Then if you create the pod using an unsigned image, the pod creation is blocked by Kyverno as it does not have any signatures associated with it:
1$ kubectl -n test-shuting run test --image=xxxxxxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/test-shuting-notation:v1-unsigned --dry-run=server
2
3Error from server: admission webhook "mutate.kyverno.svc-fail" denied the request:
4
5resource Pod/test-shuting/test was blocked due to the following policies
6
7test-irsa:
8 check-digest: 'failed to verify image xxxxxxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/test-shuting-notation:v1-unsigned:
9 .attestors[0].entries[0]: failed to verify xxxxxxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/test-shuting-notation@sha256:74a98f0e4d750c9052f092a7f7a72de7b20f94f176a490088f7a744c76c53ea5:
10 no signature is associated with "xxxxxxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/test-shuting-notation@sha256:74a98f0e4d750c9052f092a7f7a72de7b20f94f176a490088f7a744c76c53ea5",
11 make sure the artifact was signed successfully'
Conclusion
By leveraging Kyverno and IRSA, you can simplify the configuration of IAM role assumptions for Kubernetes service accounts in EKS. This approach enhances the security of the cluster by ensuring fine-grained access control to AWS resources. With the steps outlined in this blog post, you can easily set up and test IRSA in your EKS cluster.