Add a custom Prometheus alert

Abstract

Learn how to add a custom alert.

Prometheus is an open source systems monitoring system for which Grafana provides out-of-the-box support. You can extend your Managed Cloud Container solution with custom Prometheus alerts.

To create a custom Prometheus alert:

Go to the alerts folder and create a custom-alerts.yaml file.

Open the custom-alerts.yaml file and add your custom alert. For example, you might want to add an alert rule that fires when node memory usage exceeds 95%:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    prometheus: prometheus-service
    role: alert-rules
  name: prometheus-k8s-custom-alerting-rules
  namespace: monitoring
spec:
  groups:
  - name: custom statistic
    rules:
    - alert: Memory percentage is >95%
      annotations:
        description: "Node memory utilization is > 95%"
        message: "Out of memory (instance {{ $labels.node }})"
      expr: label_replace(sum(kube_metrics_server_nodes_mem) by (exported_instance), "node", "$1", "exported_instance", "(.*)")  / on(node) ((sum(kube_node_status_allocatable_memory_bytes) by (node)) / 1024) * 100 > 95
      for: 10m
      labels:
        severity: P3
        resource: '{{ $labels.node }}'

Run the Application pipeline.