Add a custom Prometheus alert
Abstract
Learn how to add a custom alert.
Prometheus is an open source systems monitoring system for which Grafana provides out-of-the-box support. You can extend your Managed Cloud Container solution with custom Prometheus alerts.
To create a custom Prometheus alert:
Go to the
alerts
folder and create acustom-alerts.yaml
file.Open the
custom-alerts.yaml
file and add your custom alert. For example, you might want to add an alert rule that fires when node memory usage exceeds 95%:apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: labels: prometheus: prometheus-service role: alert-rules name: prometheus-k8s-custom-alerting-rules namespace: monitoring spec: groups: - name: custom statistic rules: - alert: Memory percentage is >95% annotations: description: "Node memory utilization is > 95%" message: "Out of memory (instance {{ $labels.node }})" expr: label_replace(sum(kube_metrics_server_nodes_mem) by (exported_instance), "node", "$1", "exported_instance", "(.*)") / on(node) ((sum(kube_node_status_allocatable_memory_bytes) by (node)) / 1024) * 100 > 95 for: 10m labels: severity: P3 resource: '{{ $labels.node }}'
Run the Application pipeline.