ObservaStack

runbooks/ServiceHighErrorRate.md

# ServiceHighErrorRate

Larkspur Grocers is fictional. This note explains the alert. It is not an incident report. It does not record a page, a customer impact, or an MTTR.

## Signal

- Severity: `warning`
- Team: `platform`
- Pending duration (`for`): 10m

```promql
service:http_requests:rate5m > 0
and
(
  service:http_requests_errors:rate5m
  /
  service:http_requests:rate5m
) > 0.05
```

## What the condition means

More than 5 percent of HTTP requests for a service are 5xx over 5 minutes, and that has held for 10 minutes. The service still has traffic.

## What an on-call would check

1. Read the expression above and the recording rules it uses.
2. Compare it with the promtool unit tests in `tests/rules/`. Every alert except Watchdog has a firing case and a non-firing case.
3. The annotation `runbook_url` is `https://runbooks.larkspur.example/ServiceHighErrorRate`. That host is a placeholder. This file is the copy in the repo.
4. This stack does not page anyone. Receivers use `*_file` credential paths and `example.invalid`.