Add a custom Prometheus alert

Abstract

Learn how to add a custom alert.

Prometheus is an open source systems monitoring system for which Grafana provides out-of-the-box support. You can extend your Managed Cloud Container solution with custom Prometheus alerts.

To create a custom Prometheus alert:

  1. Go to the alerts folder and create a custom-alerts.yaml file.

  2. Open the custom-alerts.yaml file and add your custom alert. For example, you might want to add an alert rule that fires when node memory usage exceeds 95%:

    apiVersion: monitoring.coreos.com/v1
    kind: PrometheusRule
    metadata:
      labels:
        prometheus: prometheus-service
        role: alert-rules
      name: prometheus-k8s-custom-alerting-rules
      namespace: monitoring
    spec:
      groups:
      - name: custom statistic
        rules:
        - alert: Memory percentage is >95%
          annotations:
            description: "Node memory utilization is > 95%"
            message: "Out of memory (instance {{ $labels.node }})"
          expr: label_replace(sum(kube_metrics_server_nodes_mem) by (exported_instance), "node", "$1", "exported_instance", "(.*)")  / on(node) ((sum(kube_node_status_allocatable_memory_bytes) by (node)) / 1024) * 100 > 95
          for: 10m
          labels:
            severity: P3
            resource: '{{ $labels.node }}'
  3. Run the Application pipeline.