ObservaStack

Prometheus, Alertmanager, and Grafana configuration for Larkspur Grocers, a fictional grocer. The public page is static. It does not run Prometheus or Grafana. Nothing here is a customer, a traffic figure, an incident, or an MTTR.

The page says the checks were recorded at build time. Numbers on the page come from results/checks.json. Screenshots use scripts/synth_openmetrics.py --seed 42 and are labeled synthetic.

What is real

• promtool check config, check rules, test rules, and check metrics run against the files in this directory.
• pint --offline lints rules/ with policies/.pint.hcl. Exceptions are in policies/pint-exceptions.md.
• amtool check-config, amtool config routes test, and amtool template render run against alertmanager/.
• During make screenshots, a local Grafana imports the five dashboard JSON files with POST /api/dashboards/import. The result is checks.json → grafana_import. Alertmanager is started with an empty --cluster.listen-address so it does not gossip on 0.0.0.0:9094. After Prometheus, Alertmanager, and Grafana are up, ss -ltnp must show every listening socket of those PIDs on 127.0.0.1. That result is checks.json → build_listeners. The processes are stopped when the script exits. Their files live under /tmp/observastack-build/ and are deleted afterwards.

Commands

make tools
      make check
      make screenshots
      make showcase
      make test

make check does not need the screenshots. make showcase needs results/checks.json, grafana_import, and build_listeners. Re-run make check after screenshots if you want a fresh timestamp; it keeps grafana_import and build_listeners. Then run make showcase again so the page matches.

Pinned tools, verified with the publisher's checksums, install into .tools/ (gitignored):

• Prometheus / promtool 3.15.0
• Alertmanager / amtool 0.34.1
• pint 0.89.0
• Grafana OSS 13.2.3 (build-time import and screenshots only)

Python dependencies are pinned in requirements.txt and installed into .venv/.

Adaptations the tools forced

• Metric names. promtool check metrics rejects node_exporter's camelCase node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes, and it rejects a non-histogram _count suffix (pg_stat_activity_count). The catalog uses node_memory_available_bytes, node_memory_total_bytes, and pg_stat_activity_connections. Rules match the catalog. demo-app emits that catalog, including exporter-shaped and engine-shaped series, so the linter sees every name. ALERTS in the demo output and in the screenshot fixture is not the rule engine.
• Alertmanager routes. A child route replaces its parent receiver. Critical alerts use two sibling routes, both continue: true, so PagerDuty and Slack both match.
• Time intervals. amtool config routes test in v0.34.1 does not evaluate active_time_intervals or mute_time_intervals. A platform warning resolves to slack-platform in amtool even outside business hours. The config still contains the intervals. faketime is not installed, and the clock is not changed.
• Credentials. Only _file fields. make check writes placeholder files under /tmp/observastack-secrets/. They are not secrets and are not committed. Hosts are .larkspur.example or example.invalid. The Prometheus alertmanager target is alertmanager.larkspur.example:9093 for that reason.
• Grafana datasource URL. The committed provisioning file uses ${PROMETHEUS_URL} for compose. The screenshot script writes a concrete http://127.0.0.1:19090 file under /tmp/observastack-build/ and does not keep Grafana's data/ directory or grafana.db in the repo. Grafana 13.2 ships the Prometheus datasource under data/plugins-bundled. make tools unpacks that tree to .tools/grafana/plugins-bundled/ (gitignored). The screenshot run links it into a /tmp homepath at data/plugins-bundled, which is where the server looks. Anonymous org name stays the default Main Org..
• promtool unit-test output. v3.15 prints SUCCESS per file, not per case. results/checks.json lists the case names from the YAML and marks them passed when that file exits 0.
• SLO unit-test names. Group names such as fast_burn_15x are the input burn of the last value sequence (error step / total step / budget), not the workbook thresholds 14.4 / 6 / 3 / 1. The thresholds use strict >. fast_burn_15x keeps a quiet prefix so Fast fires through the 1h+5m branch; the page prints that group's computed multiplier and eval times from the test file and checks.json.
• OpenMetrics timestamps for promtool tsdb create-blocks-from openmetrics are seconds. Rule backfill always passes --output-dir under /tmp/observastack-build/ruleblocks. The tool's default output directory is data/, which this repo must not contain.

Not run

• docker-compose.yml is digest-pinned and is not executed. This box has no Docker. Grafana's admin password is ${GF_SECURITY_ADMIN_PASSWORD} with no committed value. The Postgres exporter DSN in the default has no password.
• .github/workflows/observability.yml is not a repository-root workflow, so GitHub will not run it from this path. make test also expects the showcase, which expects screenshots.

Layout

site/observastack/ is generated by scripts/build_showcase.py. Do not hand-edit the numbers. The homepage of the portfolio is updated separately, after this demo is accepted.