Policies and Rules Count
This metric can be used to track the number of policies as well as rules present in the cluster which are currently active and even the ones which are not currently active but were created in the past.
Monitoring
Monitor and observe the operation of Kyverno using metrics.
Introduction
As a cluster administrator, it may benefit you to have monitoring capabilities over both the state and execution of cluster-applied Kyverno policies. This includes monitoring over any applied changes to policies, any activity associated with incoming requests, and any results produced as an outcome. If enabled, monitoring will allow you to visualize and alert on applied policies, and is critical to overall cluster observability and compliance.
In addition, you can specify the scope of your monitoring targets to either the rule, policy, or cluster level, which enables you to extract more granular insights from collected metrics.
Installation and Setup
When you install Kyverno via Helm, additional services are created inside the kyverno Namespace which expose metrics on port 8000.
1$ values.yaml
2
3admissionController:
4 metricsService:
5 create: true
6 # ...
7
8backgroundController:
9 metricsService:
10 create: true
11 # ...
12
13cleanupController:
14 metricsService:
15 create: true
16 # ...
17
18reportsController:
19 metricsService:
20 create: true
21 # ...
By default, the service type is going to be ClusterIP meaning that metrics can only be scraped by a Prometheus server sitting inside the cluster.
In some cases, the Prometheus server may sit outside your workload cluster as a shared service. In these scenarios, you will want the kyverno-svc-metrics Service to be publicly exposed so as to expose the metrics (available at port 8000) to your external Prometheus server.
Services can be exposed to external clients via an Ingress, or using LoadBalancer or NodePort Service types.
To expose your kyverno-svc-metrics service publicly as NodePort at host’s/node’s port number 8000, you can configure your values.yaml before Helm installation as follows:
1admissionController:
2 metricsService:
3 create: true
4 type: NodePort
5 port: 8000
6 nodePort: 8000
7 # ...
8
9backgroundController:
10 metricsService:
11 create: true
12 type: NodePort
13 port: 8000
14 nodePort: 8000
15 # ...
16
17cleanupController:
18 metricsService:
19 create: true
20 type: NodePort
21 port: 8000
22 nodePort: 8000
23 # ...
24
25reportsController:
26 metricsService:
27 create: true
28 type: NodePort
29 port: 8000
30 nodePort: 8000
31 # ...
To expose the kyverno-svc-metrics service using a LoadBalancer type, you can configure your values.yaml before Helm installation as follows:
1admissionController:
2 metricsService:
3 create: true
4 type: LoadBalancer
5 port: 8000
6 nodePort:
7 # ...
8
9backgroundController:
10 metricsService:
11 create: true
12 type: LoadBalancer
13 port: 8000
14 nodePort:
15 # ...
16
17cleanupController:
18 metricsService:
19 create: true
20 type: LoadBalancer
21 port: 8000
22 nodePort:
23 # ...
24
25reportsController:
26 metricsService:
27 create: true
28 type: LoadBalancer
29 port: 8000
30 nodePort:
31 # ...
Configuring the metrics
While installing Kyverno via Helm, you also have the ability to configure which metrics to expose, this configuration will be stored in the kyverno-metrics ConfigMap.
You can configure which Namespaces you want to include and/or exclude for metric exportation when configuring your Helm chart. This configuration is useful in situations where you might want to exclude the exposure of Kyverno metrics for certain Namespaces like test or experimental Namespaces. Likewise, you can include certain Namespaces if you want to monitor Kyverno-related activity for only a set of certain critical Namespaces. Exporting the right set of Namespaces (as opposed to exposing all Namespaces) can end up substantially reducing the memory footprint of Kyverno’s metrics exporter. Moreover, you can also configure the exposure of specific metrics, disabling them completely or dropping some label dimensions. For Histograms, you can change the default bucket boundaries or configure it for a specific metric as well.
1...
2metricsConfig:
3 # 'namespaces.include': list of namespaces to capture metrics for. Default: all namespaces included.
4 # 'namespaces.exclude': list of namespaces to NOT capture metrics for. Default: [], none of the namespaces excluded.
5 # `exclude` takes precedence over `include` in cases when a Namespace is provided under both.
6 namespaces:
7 include: []
8 exclude: []
9
10 # Configures the bucket boundaries for all Histogram metrics, the value below is the default.
11 bucketBoundaries: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 15, 20, 25, 30]
12
13 # Per Metric configuration, allows disabling metrics, dropping labels and change the bucket boundaries.
14 metricsExposure:
15 kyverno_policy_execution_duration_seconds:
16 disabledLabelDimensions: ["resource_kind", "resource_namespace", "resource_request_operation"]
17 bucketBoundaries: [0.005, 0.01, 0.025]
18 kyverno_admission_review_duration_seconds:
19 enabled: false
20...
Disabling metrics
Some metrics may generate an excess amount of data which may be undesirable in situations where this incurs additional cost. Some monitoring products and solutions have the ability to selectively disable which metrics are sent to collectors while leaving others enabled.
Kyverno configuration side
As described above, Kyverno allows disabling metrics, dropping labels and changing the bucket boundaries by changing the kyverno-metrics ConfigMap, please refer to the example provided.
DataDog OpenMetrics side
Disabling select metrics with DataDog OpenMetrics can be done by annotating the Kyverno Pod(s) as shown below.
1apiVersion: v1
2kind: Pod
3metadata:
4 annotations:
5 ad.datadoghq.com/kyverno.checks: |
6 {
7 "openmetrics": {
8 "init_config": {},
9 "instances": [
10 {
11 "openmetrics_endpoint": "http://%%host%%:8000/metrics",
12 "namespace": "kyverno",
13 "metrics": [
14 {"kyverno_policy_rule_info_total": "policy_rule_info"},
15 {"kyverno_admission_requests": "admission_requests"},
16 {"kyverno_policy_changes": "policy_changes"}
17 ],
18 "exclude_labels": [
19 "resource_namespace"
20 ]
21 },
22 {
23 "openmetrics_endpoint": "http://%%host%%:8000/metrics",
24 "namespace": "kyverno",
25 "metrics": [
26 {"kyverno_policy_results": "policy_results"}
27 ]
28 }
29 ]
30 }
31 }
The Kyverno Helm chart supports including additional Pod annotations in the values file as shown in the below example.
1podAnnotations:
2 # https://github.com/DataDog/integrations-core/blob/master/openmetrics/datadog_checks/openmetrics/data/conf.yaml.example
3 # Note: To collect counter metrics with names ending in `_total`, specify the metric name without the `_total`
4 ad.datadoghq.com/kyverno.checks: |
5 {
6 "openmetrics": {
7 "init_config": {},
8 "instances": [
9 {
10 "openmetrics_endpoint": "http://%%host%%:8000/metrics",
11 "namespace": "kyverno",
12 "metrics": [
13 {"kyverno_policy_rule_info_total": "policy_rule_info"},
14 {"kyverno_admission_requests": "admission_requests"},
15 {"kyverno_policy_changes": "policy_changes"}
16 ],
17 "exclude_labels": [
18 "resource_namespace"
19 ]
20 },
21 {
22 "openmetrics_endpoint": "http://%%host%%:8000/metrics",
23 "namespace": "kyverno",
24 "metrics": [
25 {"kyverno_policy_results": "policy_results"}
26 ]
27 }
28 ]
29 }
30 }
Metrics and Dashboard
Policy and Rule Execution
This metric can be used to track the results associated with the rules executing as a part of incoming resource requests and even background scans. This metric can be further aggregated to track policy-level results as well.
HTTP Requests Latency
This metric can be used to track the latencies associated with HTTP requests.
Policy Rule Execution Latency
This metric can be used to track the latencies associated with the execution/processing of the individual rules whenever they evaluate incoming resource requests or execute background scans. This metric can be further aggregated to present latencies at the policy-level.
Admission Review Latency
This metric can be used to track the end-to-end latencies associated with the entire individual admission review, corresponding to the incoming resource request triggering a bunch of policies and rules.
Admission Requests Count
This metric can be used to track the number of admission requests which were triggered as a part of Kyverno.
Cleanup Controller Deleted Objects
This metric can be used to track the number of objects deleted by the cleanup controller.
Cleanup Controller Deleted Objects
This metric can be used to track the number of objects deleted by the cleanup TTL controller.
Cleanup Controller Errors Count
This metric can be used to track the number of errors encountered by the cleanup controller while trying to delete objects.
Cleanup Controller Errors Count
This metric can be used to track the number of errors encountered by the cleanup TTL controller while trying to delete objects.
Controller Drops Count
This metric can be used to track the number of times a controller drops elements. Dropping usually indicates an unrecoverable error, the controller retried to process an item a couple of times and after failing every try drop the item.
Controller Reconciliations Count
This metric can be used to track the number of reconciliations performed by various Kyverno controllers.
Controller Requeues Count
This metric can be used to track the number of times a controller requeues elements to be processed. Requeueing usually indicates that an error occured and that the controller enqueued the same item to retry processing it a bit later.
HTTP Requests Count
This metric can be used to track the number of http requests which were triggered as a part of Kyverno.
Policy Changes Count
This metric can be used to track the history of all Kyverno policy-related changes such as policy creations, updates, and deletions.
Client Queries
This metric can be used to track the number of queries per second (QPS) from Kyverno.
Grafana Dashboard
A ready-to-use dashboard for Kyverno metrics.
OpenTelemetry
OpenTelemetry integration in Kyverno.
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.
Last modified October 13, 2023 at 9:59 AM PST: fix: allow dropping metrics, labels and configuring histogram bucket boundaries to avoid high cardinality (#974) (4ee00e5)