Security

Security Processes and Guidelines

Kyverno serves as an admission controller and is a critical component of the Kubernetes control plane. It is important to properly secure and monitor Kyverno. This section provides guidance on securing Kyverno and the security processes for the Kyverno project.

Disclosure Process

Security vulnerabilities are best handled swiftly and discretely with the goal of minimizing the total time users remain vulnerable to exploits.

If you find or suspect a vulnerability, please email the security group at kyverno-security@googlegroups.com with the following information:

  • description of the problem
  • precise and detailed steps (include screenshots) that created the problem
  • the affected version(s)
  • any known mitigations

The Kyverno security response team will send an initial acknowledgement of the disclosure in 3-5 working days. Once the vulnerability and mitigation are confirmed, the team will plan to release any necessary changes based on the severity and complexity. Additional details on the security policy and processes are available in the Kyverno git repo.

Security Audits

The Kyverno project participates in 3rd party security audits and reviews that help provide a comprehensive evaluation of the project’s security posture. These are listed below:

Kyverno Third-Party Security Audit 2023

A Kyverno Third-Party Security Audit was conducted by Ada Logics, in collaboration with the project maintainers and was funded by the Cloud Native Computing Foundation. The audit identified and addressed ten security issues, including six CVEs, with fixes in Kyverno v1.10.6 and v1.11.1. Notably, users of official releases were unaffected by CVEs in the Notary verifier. The audit did not reveal any policy by-pass situations, but did identify two security bugs with a third-party dependency. Kyverno was found to demonstrate strong compliance with SLSA, earning the highest score, and ensuring tamper-proof build artifacts. More information is available in this blog post. You can also review the complete Kyverno 2023 security audit report here.

Kyverno Fuzzing Security Audit 2023

The Kyverno Fuzzing Security Audit was conducted as part of the CNCF’s security initiative. Fuzz testing, or fuzzing, is an automated process that injects random inputs into the system to reveal defects and vulnerabilities. The audit, spanning July and August 2023, resulted in 15 fuzzers identifying three bugs. Post-audit, Kyverno continues to test for bugs and vulnerabilities using OSS-Fuzz. The audit’s findings prompted fixes and ongoing testing to ensure a secure and robust code base. You can read more about the fuzz testing in this blog post.

Contact Us

To communicate with the Kyverno team, for any questions or discussions, use Slack or GitHub.

Issues

All security related issues are labeled as security and can be viewed here.

Release Artifacts

The Kyverno container images are available here.

With each release, the following artifacts are uploaded (where CLI binaries include signature and PEM files):

  • checksums.txt
  • install.yaml
  • kyverno-cli-<version_number>.tar.gz
  • kyverno-cli_v<version_number>_darwin_arm64.tar.gz
  • kyverno-cli_v<version_number>_darwin_x86_64.tar.gz
  • kyverno-cli_v<version_number>_linux_arm64.tar.gz
  • kyverno-cli_v<version_number>_linux_s390x.tar.gz
  • kyverno-cli_v<version_number>_linux_x86_64.tar.gz
  • kyverno-cli_v<version_number>_windows_arm64.zip
  • kyverno-cli_v<version_number>_windows_x86_64.zip
  • Source code (zip)
  • Source code (tar.gz)

Verifying Kyverno Container Images

Kyverno container images are signed using Cosign and the keyless signing feature. The signatures are stored in a separate repository from the container image they reference located at ghcr.io/kyverno/signatures. To verify the container image using Cosign v1.x, follow the steps below.

  1. Install Cosign
  2. Configure the Kyverno signature repository:
1export COSIGN_REPOSITORY=ghcr.io/kyverno/signatures
  1. Verify the image:
1COSIGN_EXPERIMENTAL=1 cosign verify ghcr.io/kyverno/kyverno:<release_tag> | jq

For Cosign v2.x, use the following command instead.

1COSIGN_REPOSITORY=ghcr.io/kyverno/signatures cosign verify ghcr.io/kyverno/kyverno:<release_tag> \
2  --certificate-identity-regexp="https://github.com/kyverno/kyverno/.github/workflows/release.yaml@refs/tags/*" \
3  --certificate-oidc-issuer="https://token.actions.githubusercontent.com" | jq

If the container image was properly signed, the output should be similar to:

 1Verification for ghcr.io/kyverno/kyverno:v1.11.2 --
 2The following checks were performed on each of these signatures:
 3  - The cosign claims were validated
 4  - Existence of the claims in the transparency log was verified offline
 5  - The code-signing certificate was verified using trusted certificate authority certificates
 6[
 7  {
 8    "critical": {
 9      "identity": {
10        "docker-reference": "ghcr.io/kyverno/kyverno"
11      },
12      "image": {
13        "docker-manifest-digest": "sha256:c2d33cc05ca2c7bab7ca13f4ef24276f4f1a83687e13a971945475b0e931bde8"
14      },
15      "type": "cosign container image signature"
16    },
17    "optional": {
18      "1.3.6.1.4.1.57264.1.1": "https://token.actions.githubusercontent.com",
19      "1.3.6.1.4.1.57264.1.2": "push",
20      "1.3.6.1.4.1.57264.1.3": "5f9ed6f0f81e36a5f94a8bcece67ff94f7777a1a",
21      "1.3.6.1.4.1.57264.1.4": "releaser",
22      "1.3.6.1.4.1.57264.1.5": "kyverno/kyverno",
23      "1.3.6.1.4.1.57264.1.6": "refs/tags/v1.11.2",
24      "Bundle": {
25        "SignedEntryTimestamp": "MEQCIH+Nnu89Mzm9XEb/f8n868uaQAGd631+kkx9mjcdYU+gAiAVPMEfIBGT5A+QBRfGR+X/Majgt+Jh5tsVNvlyvUu99A==",
26        "Payload": {
27          "body": "eyJhcGlWZXJzaW9uIjoiMC4w<snip>",
28          "integratedTime": 1703770792,
29          "logIndex": 59820453,
30          "logID": "c0d23d6ad406973f9559f3ba2d1ca01f84147d8ffc5b8445c224f98b9591801d"
31        }
32      },
33      "Issuer": "https://token.actions.githubusercontent.com",
34      "Subject": "https://github.com/kyverno/kyverno/.github/workflows/release.yaml@refs/tags/v1.11.2",
35      "githubWorkflowName": "releaser",
36      "githubWorkflowRef": "refs/tags/v1.11.2",
37      "githubWorkflowRepository": "kyverno/kyverno",
38      "githubWorkflowSha": "5f9ed6f0f81e36a5f94a8bcece67ff94f7777a1a",
39      "githubWorkflowTrigger": "push",
40      "ref": "5f9ed6f0f81e36a5f94a8bcece67ff94f7777a1a",
41      "repo": "kyverno/kyverno",
42      "workflow": "releaser"
43    }
44  }
45]

Note that the important fields to verify in the output are optional.Issuer and optional.Subject. If Issuer and Subject do not match the values shown above, the image is not genuine.

All Kyverno images can be verified.

Verifying Provenance

Kyverno creates and attests to the provenance of its builds using the SLSA standard and meets the SLSA Level 3 specification. The attested provenance may be verified using the cosign tool.

For v1.x of Cosign, use the following command.

1COSIGN_EXPERIMENTAL=1 cosign verify-attestation \
2  --type slsaprovenance ghcr.io/kyverno/kyverno:<release_tag> | jq .payload -r | base64 --decode | jq

For v2.x of Cosign, use the following command.

1cosign verify-attestation --type slsaprovenance \
2  --certificate-identity-regexp="https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@refs/tags/*" \
3  --certificate-oidc-issuer="https://token.actions.githubusercontent.com" \
4  ghcr.io/kyverno/kyverno:<release_tag> | jq .payload -r | base64 --decode | jq

The output will look something similar to the below.

 1Verification for ghcr.io/kyverno/kyverno:v1.11.2 --
 2The following checks were performed on each of these signatures:
 3  - The cosign claims were validated
 4  - Existence of the claims in the transparency log was verified offline
 5  - The code-signing certificate was verified using trusted certificate authority certificates
 6Certificate subject: https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@refs/tags/v1.9.0
 7Certificate issuer URL: https://token.actions.githubusercontent.com
 8GitHub Workflow Trigger: push
 9GitHub Workflow SHA: 5f9ed6f0f81e36a5f94a8bcece67ff94f7777a1a
10GitHub Workflow Name: releaser
11GitHub Workflow Repository: kyverno/kyverno
12GitHub Workflow Ref: refs/tags/v1.11.2
13{
14  "_type": "https://in-toto.io/Statement/v0.1",
15  "predicateType": "https://slsa.dev/provenance/v0.2",
16  "subject": [
17    {
18      "name": "ghcr.io/kyverno/kyverno",
19      "digest": {
20        "sha256": "c2d33cc05ca2c7bab7ca13f4ef24276f4f1a83687e13a971945475b0e931bde8"
21      }
22    }
23  ],
24  "predicate": {
25    "builder": {
26      "id": "https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@refs/tags/v1.9.0"
27    },
28    "buildType": "https://github.com/slsa-framework/slsa-github-generator/container@v1",
29    "invocation": {
30      "configSource": {
31        "uri": "git+https://github.com/kyverno/kyverno@refs/tags/v1.11.2",
32        "digest": {
33          "sha1": "5f9ed6f0f81e36a5f94a8bcece67ff94f7777a1a"
34        },
35        "entryPoint": ".github/workflows/release.yaml"
36      },
37      "parameters": {},
38      "environment": {

Fetching the SBOM for Kyverno

An SBOM (Software Bill of Materials) in CycloneDX JSON format is published for each Kyverno release, including pre-releases. Like signatures, SBOMs are stored in a separate repository at ghcr.io/kyverno/sbom. To download and verify the SBOM for a specific version, install Cosign and run:

1COSIGN_REPOSITORY=ghcr.io/kyverno/sbom cosign download sbom ghcr.io/kyverno/kyverno:<release_tag>

To save the SBOM to a file, run the following command:

1COSIGN_REPOSITORY=ghcr.io/kyverno/sbom cosign download sbom ghcr.io/kyverno/kyverno:<release_tag> > kyverno.sbom.json

Security Scorecard

Kyverno uses Scorecards by OSSF to maintain repository-wide security standards. The current OSSF/scorecard score for Kyverno can be found in this tracker issue. The Kyverno team is committed to achieving and maintaining a high score. Contributions are welcome.

Vulnerability Scan Reports

The Kyverno Helm Chart is available via the Artifact Hub page along with an auto-generated Security Report generated by Artifact Hub for all the releases.

Security Best Practices

The following sections discuss related best practices for Kyverno:

Pod security

Kyverno Pods are configured to follow security best practices and conform to the Pod Security Standards restricted profile:

  • runAsNonRoot is set to true
  • privileged is set to false
  • allowPrivilegeEscalation is set to false
  • readOnlyRootFilesystem is set to true
  • all capabilities are dropped
  • limits and quotas are configured
  • liveness and readiness probes are configured

RBAC

The Kyverno RBAC configurations are described in the installation section.

Use the following command to view all Kyverno roles:

1kubectl get clusterroles,roles -A | grep kyverno

Networking

Kyverno network traffic is encrypted and should be restricted using NetworkPolicies or similar constructs.

By default, a Kyverno installation does not configure NetworkPolicies (see this issue). The Kyverno Helm chart has a networkPolicy.enabled option to enable a NetworkPolicy.

Kyverno requires the following network communications to be allowed:

  • ingress traffic to port 9443 from the API server
  • ingress traffic to port 9443 from the host for health checks
  • ingress traffic to port 8000 if metrics are collected by Prometheus or other metrics collectors
  • egress traffic to the API server if the API Call feature is used
  • egress (HTTPS) traffic to OCI registries if image verification policy rules are configured or if image registry context variables are used
  • egress (HTTP or HTTPS) traffic to external services if the external service call feature is used

Webhooks

Use the following command to view all Kyverno webhooks:

1kubectl get mutatingwebhookconfigurations,validatingwebhookconfigurations | grep kyverno

Kyverno creates the following mutating webhook configurations:

  • kyverno-policy-mutating-webhook-cfg: handles policy changes to index and cache policy sets.
  • kyverno-resource-mutating-webhook-cfg: handles resource admission requests to apply matching Kyverno mutate policy rules.
  • kyverno-verify-mutating-webhook-cfg: periodically tests Kyverno webhook configurations

Kyverno creates the following validating webhook configurations:

  • kyverno-policy-validating-webhook-cfg: validates Kyverno policies with checks that cannot be performed via schema validation
  • kyverno-resource-validating-webhook-cfg: handles resource resource admission requests to apply matching Kyverno validate policy rules.
  • kyverno-cleanup-validating-webhook-cfg: handles cleanup policies
  • kyverno-exception-validating-webhook-cfg: handles policy exceptions

Webhook Failure Mode

Kyverno policies are configured to fail-closed by default. This setting can be tuned on a per policy basis. Kyverno uses the configured policy set to automatically configure webhooks.

Webhook authentication and encryption

By default, Kyverno automatically generates and manage TLS certificates used for authentication with the API server and encryption of network traffic. To use a custom CA, please refer to the details in the installation section.

The Kyverno community manages a set of sample policies.

At a minimum, the Pod Security Standards and best practices policy sets are recommended for use.

Securing policies

Kyverno policies can be used to mutate and generate namespaced and cluster-wide resources. Hence, policies should be treated as critical resources and access to policies should be protected using RBAC. Note that some policies in these sets may have alternate versions. All policies should be inspected before being installed.

Threat Model

The Kubernetes SIG Security team has defined an Admission Control Threat Model. It is highly recommended that Kyverno administrators read and understand the threat model, and use it as a starting point to create their own threat model.

The sections below list each threat, mitigation, and provide Kyverno specific details.

Threat ID 1 - Attacker floods webhook with traffic preventing its operations

Threat Model Link

Mitigation:

Threat ID 2 - Attacker passes workloads which require complex processing causing timeouts

Threat Model Link

Mitigations:

Threat ID 3 - Attacker exploits misconfiguration of webhook to bypass

Threat Model Link

Mitigation:

Threat ID 4 - Attacker has rights to delete or modify the Kubernetes webhook object

Threat Model Link

Mitigation:

Threat ID 5 - Attacker gets access to valid credentials for the webhook

Threat Model Link

Mitigation:

Threat ID 6 - Attacker gains access to a cluster admin credential

Threat Model Link

Mitigation

N/A

Threat ID 7 - Attacker sniffs traffic on the container network

Threat Model Link

Mitigation

Threat ID 8 - Attacker carries out a MITM attack on the webhook

Threat Model Link

Mitigation

Threat ID 9 - Attacker steals traffic from the webhook via spoofing

Threat Model Link

Mitigation

Threat ID 10 - Abusing a mutation rule to create a privileged container

Threat Model Link

Mitigation

  • Mitigation ID 6 - All rules are reviewed and tested

    Kyverno rules are Kubernetes resources written in YAML and managed by an OpenAPIv3 schema. This approach makes it easy to understand policy definitions and to apply policy-as-code best practices, like code reviews, to Kyverno policies. The Kyverno CLI provides a test command for executing unit tests as part of a continuous delivery pipeline.

Threat ID 11 - Attacker deploys workloads to namespaces that are exempt from admission control

Threat Model Link

Mitigation

  • Mitigation ID 1 - RBAC rights are strictly controlled

    Kyverno RBAC configurations are described in the configuration section. The kyverno:admission-controller role is used by Kyverno to configure webhooks. It is important to limit Kyverno to the required permissions and audit changes in the RBAC roles and role bindings.

    Kyverno excludes certain critical system Namespaces by default including the Kyverno Namespace itself. These exclusions can be managed and configured via the ConfigMap.

Threat ID 12 - Block rule can be bypassed due to missing match (e.g. missing initContainers)

Threat Model Link

Mitigation

  • Mitigation ID 6 - All rules are reviewed and tested

    Kyverno rules are Kubernetes resources written in YAML and managed by an OpenAPIv3 schema. This approach makes it easy to understand policy definitions and to apply policy-as-code best practices, like code reviews, to Kyverno policies. The Kyverno CLI provides a test command for executing unit tests as part of a continuous delivery pipeline.

Threat ID 13 - Attacker exploits bad string matching on a blocklist to bypass rules

Threat Model Link

Mitigation

  • Mitigation ID 6 - All rules are reviewed and tested

    Kyverno rules are Kubernetes resources written in YAML and managed by an OpenAPIv3 schema. This approach makes it easy to understand policy definitions and to apply policy-as-code best practices, like code reviews, to Kyverno policies. The Kyverno CLI provides a test command for executing unit tests as part of a continuous delivery pipeline.

Threat ID 14 - Attacker uses new/old features of the Kubernetes API which have no rules

Threat Model Link

Mitigation

  • Mitigation ID 6 - All rules are reviewed and tested

    Kyverno rules are Kubernetes resources written in YAML and managed by an OpenAPIv3 schema. This approach makes it easy to understand policy definitions and to apply policy-as-code best practices, like code reviews, to Kyverno policies. The Kyverno CLI provides a test command for executing unit tests as part of a continuous delivery pipeline.

Threat ID 15 - Attacker deploys privileged container to node running Webhook controller

Threat Model Link

Mitigation

Threat ID 16 - Attacker mounts a privileged node hostPath allowing modification of Webhook controller configuration

Threat Model Link

Mitigation

Threat ID 17 - Attacker has privileged SSH access to cluster node running admission webhook

Threat Model Link

Mitigation

N/A

Threat ID 18 - Attacker uses policies to send confidential data from admission requests to external systems

Threat Model Link

Mitigation


Last modified January 23, 2024 at 7:31 AM PST: added fuzzing and 3rd party security audit links to the Security section of the docs (cherry-pick #1111) (#1115) (7cd87a5)