AlertmanagerNotificationsFailing

# AlertmanagerNotificationsFailing

Larkspur Grocers is fictional. This note explains the alert. It is not an incident report. It does not record a page, a customer impact, or an MTTR.

## Signal

- Severity: `critical`
- Team: `platform`
- Pending duration (`for`): 15m

```promql
(
  sum by (service) (rate(alertmanager_notifications_failed_total[5m]))
  /
  sum by (service) (rate(alertmanager_notifications_total[5m]))
) > 0.10
and
sum by (service) (rate(alertmanager_notifications_total[5m])) > 0
```

## What the condition means

The 5 minute failure ratio of Alertmanager notifications has stayed above 0.10 for 15 minutes, and notifications are still being attempted.

## What an on-call would check

1. Read the expression above and the recording rules it uses.
2. Compare it with the promtool unit tests in `tests/rules/`. Every alert except Watchdog has a firing case and a non-firing case.
3. The annotation `runbook_url` is `https://runbooks.larkspur.example/AlertmanagerNotificationsFailing`. That host is a placeholder. This file is the copy in the repo.
4. This stack does not page anyone. Receivers use `*_file` credential paths and `example.invalid`.

返回告警目录