Metrics, logs, dashboards and alert routing are restored as one reviewable
core change. Every route terminates at a null receiver on purpose: Matrix delivery is deferred to Wave 4, behind its own preflight and a digest pin.
13 commits, +22,240 / -827 lines, all inside the Task 10 changed-path allowlist.
Supersedes PR #46 without modifying, closing or force-pushing it; zero PR #46 commits are
ancestors. Matrix delivery is deferred to Ekho-Labs/infra-work #201 and plan
Tasks 15-17.
In this PR core
Alertmanager enabled - one replica, 120h retention, 5Gi
do-block-storage volume.
Every route resolves to null. Root and child routes point at the named
null receiver; Watchdog is explicitly null-routed.
Four inhibition rules preserved - critical over warning|info, warning over info,
InfoInhibitor over info, and the InfoInhibitor target rule.
Operator discovery matches nothing yet - alertmanagerConfigSelector
on ekholabs.io/alertmanager-config = observability, namespace selector on
kubernetes.io/metadata.name = kube-prometheus-stack,
alertmanagerConfigMatcherStrategy.type = None.
Deferred to Wave 4 out of scope
Task 15 stages the OnePasswordItem credential reference only.
Task 16 runs the in-cluster preflight.
Task 17 adds the digest-pinned receiver Deployment, Service, ServiceMonitor and
ConfigMap, plus the authenticated Matrix route.
Nothing leaks early. No receiver workload, Matrix route, Matrix string or
credential field exists in core.
02Data and trigger flow
Hover or focus a block to trace upstream and downstream; click to pin, Escape to release.
Metrics: solid
Logs: dotted
Alerts: dashed
01 Sources
02 Collection
03 Rules and routing
04 Surfaces
Pipeline detail
airflow-statsd ServiceMonitor. In kube-prometheus-stack, release: prometheus-stack-slim; namespace airflow; selector component=statsd + release=ekho-labs-airflow + tier=airflow; one endpoint, port statsd-scrape, path /metrics, 30s, http.
Alloy loki.process.stage.cri, then a guarded stage.match branch admitting only lines that already open a JSON object with a quoted key; drop_malformed = false passes plain Airflow text, loguru output and malformed JSON verbatim; it promotes exactly platform, scraper, proxy_configured, error_class, and stage.label_keep bounds indexed labels to those four plus namespace, pod, container and job.
Rule semantics. Presence-based state: a series exists with value 1 only while true, not bool-modified comparisons. {family=""} isolates family-level safety series from shared use-case batch rules, and kubeControllerManager rules are disabled for the managed DOKS control plane.
03Dashboard UID and folder map
Provisioned through ConfigMaps labelled grafana_dashboard: "1"; the Grafana
sidecar reads each ConfigMap's grafana_folder annotation, so folders are
provisioned state. Every UID is preserved from the pre-change state, none renamed.
No merge auto-applies. A push to main can only plan; applying takes a manual
dispatch bound to four verified identifiers. draft Terraform Plan on a PR is gated on the PR being non-draft and the security check
passing, so it is intentionally skipped while #86 is a draft.
01
Push to main plans only
The Terraform Main Plan job runs
terraform plan -input=false -out=tfplan, then projects
terraform show -json tfplan through jq -S with a checked-in
plan-review-manifest.jq into a deterministic
plan-review-manifest.json. The redacted manifest carries only format and
Terraform versions and, per resource, the address, module, mode, type, name, change actions,
replace paths and sensitivity booleans - no before, after, values,
outputs, variables or sensitive payloads. Raw plan JSON is never persisted, uploaded or
logged.
02
Exactly one private artifact
terraform-plan-${GITHUB_SHA}, containing only
tfplan and plan-review-manifest.json, retention 1 day. Both
files are SHA-256 hashed, and those digests are what the apply is later bound to.
03
Apply is manual and bound
Only workflow_dispatch on refs/heads/main, with
four required inputs: source_run_id, commit_sha,
tfplan_sha256, plan_manifest_sha256. It runs under
environment: production, concurrency group
terraform-production-apply with cancel-in-progress: false;
non-dispatch runs stay cancellable. It verifies that the source run belongs to this
repository, workflow path, event push, ref refs/heads/main, head
SHA commit_sha, conclusion success, and exactly one unexpired
artifact of that exact name; verifies both digests; rechecks that remote main
still equals the commit immediately before apply; and applies only the saved
tfplan.
05PR #46 keep, replace, drop
From the Task 2 reconciliation contract: 54 stable (ID, path, source symbol, diff anchor)
records, each with one disposition and one destination. Intent only, no private source
reproduced.
Showing all 3 dispositions.
Keep behavior ported
Both fleet dashboard UIDs stay stable.
All four Safety freshness stall alerts - hourly, six-hour, daily, weekly - keep their
source and family thresholds and warning severity.
The scraper fleet-state and batch-state named record groups keep presence semantics and
family isolation.
kubeControllerManager rules stay disabled, removing the permanent DOKS false
alert.
The Prometheus ingress backend service, re-verified against the rendered chart.
The family and use-case taxonomy, in rule selectors and documentation alike.
Persisted-document support context stays non-authoritative.
Replace re-authored
Alertmanager enablement, null noise routes and inhibition semantics move to the
core-only configuration, which has no Matrix value and no receiver.
The StatsD ServiceMonitor is re-verified against the live Service, port and a one-match
selector.
Every fleet panel and the platform variable are re-authored under the query,
manifest and viewport contracts; state and run panels lose their vector(0)
masking.
The fleet-alerts group gains concrete triage annotations and fixtures.
The remaining bool-valued Airflow failure expression is rewritten.
The OnePassword receiver Secret reference, the receiver ConfigMap, Deployment, Service
and ServiceMonitor, the Matrix delivery-failure rule and the authenticated Matrix route move
to Wave 4 with a digest pin.
Documentation claims of Matrix activation become core-accurate text.
Drop nothing transfers
The receiver missing-secret CreateContainerConfigError behavior: receiver
availability must precede routing.
The Matrix VPS provisioning commands: no credentials, no host mutation.
The stale fleet-alert Slack routing claim, incompatible with staged Matrix
activation.
06The one authorized delete
The plan must contain exactly one delete action, zero replace actions, and
no other delete.
This retires the obsolete tenth dashboard
scraper-batch-use-case-safety. Its ConfigMap resource is gone from the module
while the object still exists in state, which is exactly why the single delete appears.
1 delete0 replacements9 dashboards after
Six panels migrated one-to-one first
Source
Destination
Destination title
1
20
Safety Freshness Age by Family
2
21
Safety Persisted Records by Source
3
23
Safety Request Failures by Source
4
25
Safety Partial Coverage
5
24
Safety Location Resolution Rate
6
22
Safety Backfill Completion
All six landed in scraper-batch-use-case before retirement, with
destination IDs and titles verified in the dashboard JSON at PR head. Missing, duplicate,
unknown or nonexistent destinations are rejected by the Task 7 contract.
07Rollout, verification windows, rollback
Two reviewers, five frozen identifiers, three timed post-apply windows, and a rollback that reuses the reviewed path.
Task 11 Release gate
Freeze five identifiers
Base SHA, head SHA, MERGE_TREE_SHA, the tfplan
SHA-256 and the plan-review-manifest.json SHA-256. Momus and Oracle must
independently approve the identical five, including exactly the one authorized delete and
zero replacements. If base or head advances, recompute and rerun; only the approved pair and
tree may merge.
Task 12 Merge and apply
Record the known-good, then dispatch
The Task 11 base becomes KNOWN_GOOD_SHA, the approved pair
merges and the tree is verified. The merge triggers the plan-only main run. Download the exact
tfplan and redacted manifest, verify the trusted source run, commit, artifact and
both hashes, and have both reviewers approve the final run plus both digests. On drift or
rejection, never dispatch.
+5 / +10 / +20 min
Post-apply checks from APPLY_COMPLETED_AT
All existing scrape targets stay up.
Airflow StatsD is up and heartbeat age is at most 300 seconds.
Expected recording rule records are present.
The named false and disconnected alerts are absent.
Datasource, folder mapping, viewport, panel and LogQL checks pass.
The Loki sample spans at least 15 minutes inside the Task 8 cardinality limits:
platform ≤ 50, scraper ≤ 250,
proxy_configured ≤ 3, error_class ≤ 100.
Rollback
VCS revert, same reviewed path
A rollback is a revert followed by the same reviewed
plan-review-dispatch sequence, never an out-of-band apply. Baseline health must be restored
within 30 minutes or the runbook stops and escalates, blocking Task 13.
Task 13 Certified
A blocking 24-hour Loki window
From APPLY_COMPLETED_AT, the window must contain at least
23h55m of queryable post-apply samples within the same cardinality limits before
CORE_COMPLETE is recorded and PR #46 is closed as superseded.
08Evidence
Task 7 authoritative run 20260813T174749Z-b10f6315, dashboards rendered in a
disposable Grafana against branch artifacts.
9 / 18 / 0
dashboards, kiosk captures at 1440x900 desktop and 390x844 mobile, failed captures.
70 / 6
variable checks and data-link checks, 0 failures in each.
18 / 32
horizontal-reach and scope-selection checks, 0 failures in each.
974 / 0
datasource queries and failures, and 0 page errors.
91
query-harness targets, 86 PromQL and 5 LogQL, over from=now-6h, to=now, step 60s: 84 pass with data, 7 pass empty and individually justified, 0 failed.
10 / 10
promtool churn cases passed inside a digest-pinned prom/prometheus image run with --network=none.
27 / 27
Safety family and source pairs resolved to exactly one latest group, so stale Pushgateway outcome groups cannot contaminate current readings.
Cleanup
Container and workdir removed and verified absent, port-forwards 19090 and 13100 killed, no production mutation.
Task 10 gates passing locally: portable temporary-root tests; the Task 3, 5,
6, 8 and 9 contracts; the Task 8 pipeline and failure harnesses; Terraform fmt and validate; dashboard JSON parsing; diff check; changed-path and forbidden generated-path checks; and the static resource preflight, which found only the allowed kubernetes_manifest.airflow_statsd_service_monitor
addition.
09Residual risk
From the Task 10 receipt and Task 7 validation. Severity is the reviewer's calibration of
what could still go wrong before apply.
Showing all 6 risks.
Detail
Only a static Terraform resource preflight ran locally. The single authorized Safety ConfigMap delete and the zero-replacement requirement are asserted against the plan only at the CI plan gate and at Task 11.
Blast radius
An unexpected delete or replace would not surface until the gate.
Mitigation
Task 11 blocks merge until both reviewers approve the frozen tfplan and manifest digests.
Detail
The gated Terraform Plan job is skipped while #86 is a draft.
Mitigation
Task 10 acceptance requires the required checks, including Terraform Plan, to pass.
Detail
No Trivy, Gitleaks or TruffleHog executable or image existed in the environment, so the diff secret-pattern scan was reviewed manually.
Findings
Its only matches were checked-in credential-sanitization fixtures and environment-variable names.
Detail
The safety_hcl check used an incorrect module-relative path before HCL parsing. The failure-fixture suite plus two targeted runtime checks were blocked by an absent temporary root and an absent Playwright Chromium headless shell.
Unaffected
The 18 captures and the query, churn and render checks.
Detail
Standalone terraform -chdir=tf/modules/prometheus-stack test reported unavailable module provider plugins, and a broad basedpyright run reported pre-existing unresolved optional imports and strictness diagnostics.
Unaffected
Neither affected the focused portable contract suite.
Interim
PR #46 stays open until Task 13 closes it as superseded, so two competing observability descriptions coexist in the interim.