Ekho-Labs / infrastructure-tf-kubernetes · draft

PR #86 - Core observability recovery

Metrics, logs, dashboards and alert routing are restored as one reviewable core change. Every route terminates at a null receiver on purpose: Matrix delivery is deferred to Wave 4, behind its own preflight and a digest pin.

Base
main @ 82e909af
Head
work/observability-recovery-core-replacement-20260813 @ d9ceb7ad
Supersedes
PR #46 @ 863ddef3 - open, unmodified

01The core / Matrix-deferred boundary

13 commits, +22,240 / -827 lines, all inside the Task 10 changed-path allowlist. Supersedes PR #46 without modifying, closing or force-pushing it; zero PR #46 commits are ancestors. Matrix delivery is deferred to Ekho-Labs/infra-work #201 and plan Tasks 15-17.

In this PR core

  • Alertmanager enabled - one replica, 120h retention, 5Gi do-block-storage volume.
  • Every route resolves to null. Root and child routes point at the named null receiver; Watchdog is explicitly null-routed.
  • Four inhibition rules preserved - critical over warning|info, warning over info, InfoInhibitor over info, and the InfoInhibitor target rule.
  • Operator discovery matches nothing yet - alertmanagerConfigSelector on ekholabs.io/alertmanager-config = observability, namespace selector on kubernetes.io/metadata.name = kube-prometheus-stack, alertmanagerConfigMatcherStrategy.type = None.

Deferred to Wave 4 out of scope

  • Task 15 stages the OnePasswordItem credential reference only.
  • Task 16 runs the in-cluster preflight.
  • Task 17 adds the digest-pinned receiver Deployment, Service, ServiceMonitor and ConfigMap, plus the authenticated Matrix route.
  • Nothing leaks early. No receiver workload, Matrix route, Matrix string or credential field exists in core.

02Data and trigger flow

Hover or focus a block to trace upstream and downstream; click to pin, Escape to release.

01 Sources

02 Collection

03 Rules and routing

04 Surfaces

Pipeline detail

airflow-statsd ServiceMonitor. In kube-prometheus-stack, release: prometheus-stack-slim; namespace airflow; selector component=statsd + release=ekho-labs-airflow + tier=airflow; one endpoint, port statsd-scrape, path /metrics, 30s, http.

Alloy loki.process. stage.cri, then a guarded stage.match branch admitting only lines that already open a JSON object with a quoted key; drop_malformed = false passes plain Airflow text, loguru output and malformed JSON verbatim; it promotes exactly platform, scraper, proxy_configured, error_class, and stage.label_keep bounds indexed labels to those four plus namespace, pod, container and job.

Rule semantics. Presence-based state: a series exists with value 1 only while true, not bool-modified comparisons. {family=""} isolates family-level safety series from shared use-case batch rules, and kubeControllerManager rules are disabled for the managed DOKS control plane.

03Dashboard UID and folder map

Provisioned through ConfigMaps labelled grafana_dashboard: "1"; the Grafana sidecar reads each ConfigMap's grafana_folder annotation, so folders are provisioned state. Every UID is preserved from the pre-change state, none renamed.

DashboardUIDTemplate variables
Kubernetes
Kubernetes Cluster Overviewk8s-cluster-overviewnone
Kubernetes Node Metricsk8s-node-metrics$node
Kubernetes Pod Metricsk8s-pod-metrics$namespace, $pod
Scraper Observability
Scraper Fleet Overviewfleet-overview$platform
Platform Health Detailfleet-health-detail$platform
Scraper Investigationscraper-investigation$platform, $scraper, $use_case, $canonical_collection
Airflow Scraper Stateairflow-run-health$platform, $scraper, $use_case, $canonical_collection
Airflow Scraper Output and Qualityairflow-output-quality$platform, $scraper, $use_case, $canonical_collection
Scraper Batch Use-Case Healthscraper-batch-use-case$platform, $scraper, $use_case, $canonical_collection

04Gated plan, private artifact, bound apply

No merge auto-applies. A push to main can only plan; applying takes a manual dispatch bound to four verified identifiers. draft Terraform Plan on a PR is gated on the PR being non-draft and the security check passing, so it is intentionally skipped while #86 is a draft.

01

Push to main plans only

The Terraform Main Plan job runs terraform plan -input=false -out=tfplan, then projects terraform show -json tfplan through jq -S with a checked-in plan-review-manifest.jq into a deterministic plan-review-manifest.json. The redacted manifest carries only format and Terraform versions and, per resource, the address, module, mode, type, name, change actions, replace paths and sensitivity booleans - no before, after, values, outputs, variables or sensitive payloads. Raw plan JSON is never persisted, uploaded or logged.

02

Exactly one private artifact

terraform-plan-${GITHUB_SHA}, containing only tfplan and plan-review-manifest.json, retention 1 day. Both files are SHA-256 hashed, and those digests are what the apply is later bound to.

03

Apply is manual and bound

Only workflow_dispatch on refs/heads/main, with four required inputs: source_run_id, commit_sha, tfplan_sha256, plan_manifest_sha256. It runs under environment: production, concurrency group terraform-production-apply with cancel-in-progress: false; non-dispatch runs stay cancellable. It verifies that the source run belongs to this repository, workflow path, event push, ref refs/heads/main, head SHA commit_sha, conclusion success, and exactly one unexpired artifact of that exact name; verifies both digests; rechecks that remote main still equals the commit immediately before apply; and applies only the saved tfplan.

05PR #46 keep, replace, drop

From the Task 2 reconciliation contract: 54 stable (ID, path, source symbol, diff anchor) records, each with one disposition and one destination. Intent only, no private source reproduced.

Showing all 3 dispositions.

Keep behavior ported

  • Both fleet dashboard UIDs stay stable.
  • All four Safety freshness stall alerts - hourly, six-hour, daily, weekly - keep their source and family thresholds and warning severity.
  • The scraper fleet-state and batch-state named record groups keep presence semantics and family isolation.
  • kubeControllerManager rules stay disabled, removing the permanent DOKS false alert.
  • The Prometheus ingress backend service, re-verified against the rendered chart.
  • The family and use-case taxonomy, in rule selectors and documentation alike.
  • Persisted-document support context stays non-authoritative.

Replace re-authored

  • Alertmanager enablement, null noise routes and inhibition semantics move to the core-only configuration, which has no Matrix value and no receiver.
  • The StatsD ServiceMonitor is re-verified against the live Service, port and a one-match selector.
  • Every fleet panel and the platform variable are re-authored under the query, manifest and viewport contracts; state and run panels lose their vector(0) masking.
  • The fleet-alerts group gains concrete triage annotations and fixtures.
  • The remaining bool-valued Airflow failure expression is rewritten.
  • The OnePassword receiver Secret reference, the receiver ConfigMap, Deployment, Service and ServiceMonitor, the Matrix delivery-failure rule and the authenticated Matrix route move to Wave 4 with a digest pin.
  • Documentation claims of Matrix activation become core-accurate text.

Drop nothing transfers

  • The receiver missing-secret CreateContainerConfigError behavior: receiver availability must precede routing.
  • The Matrix VPS provisioning commands: no credentials, no host mutation.
  • The stale fleet-alert Slack routing claim, incompatible with staged Matrix activation.

06The one authorized delete

The plan must contain exactly one delete action, zero replace actions, and no other delete.

Single destructive action

module.prometheus_stack.kubernetes_config_map_v1.grafana_dashboard_safety_ingestion

This retires the obsolete tenth dashboard scraper-batch-use-case-safety. Its ConfigMap resource is gone from the module while the object still exists in state, which is exactly why the single delete appears.

1 delete0 replacements9 dashboards after

Six panels migrated one-to-one first

SourceDestinationDestination title
120Safety Freshness Age by Family
221Safety Persisted Records by Source
323Safety Request Failures by Source
425Safety Partial Coverage
524Safety Location Resolution Rate
622Safety Backfill Completion

All six landed in scraper-batch-use-case before retirement, with destination IDs and titles verified in the dashboard JSON at PR head. Missing, duplicate, unknown or nonexistent destinations are rejected by the Task 7 contract.

07Rollout, verification windows, rollback

Two reviewers, five frozen identifiers, three timed post-apply windows, and a rollback that reuses the reviewed path.

  1. Task 11
    Release gate

    Freeze five identifiers

    Base SHA, head SHA, MERGE_TREE_SHA, the tfplan SHA-256 and the plan-review-manifest.json SHA-256. Momus and Oracle must independently approve the identical five, including exactly the one authorized delete and zero replacements. If base or head advances, recompute and rerun; only the approved pair and tree may merge.

  2. Task 12
    Merge and apply

    Record the known-good, then dispatch

    The Task 11 base becomes KNOWN_GOOD_SHA, the approved pair merges and the tree is verified. The merge triggers the plan-only main run. Download the exact tfplan and redacted manifest, verify the trusted source run, commit, artifact and both hashes, and have both reviewers approve the final run plus both digests. On drift or rejection, never dispatch.

  3. +5 / +10
    / +20 min

    Post-apply checks from APPLY_COMPLETED_AT

    • All existing scrape targets stay up.
    • Airflow StatsD is up and heartbeat age is at most 300 seconds.
    • Expected recording rule records are present.
    • The named false and disconnected alerts are absent.
    • Datasource, folder mapping, viewport, panel and LogQL checks pass.
    • The Loki sample spans at least 15 minutes inside the Task 8 cardinality limits: platform ≤ 50, scraper ≤ 250, proxy_configured ≤ 3, error_class ≤ 100.
  4. Rollback

    VCS revert, same reviewed path

    A rollback is a revert followed by the same reviewed plan-review-dispatch sequence, never an out-of-band apply. Baseline health must be restored within 30 minutes or the runbook stops and escalates, blocking Task 13.

  5. Task 13
    Certified

    A blocking 24-hour Loki window

    From APPLY_COMPLETED_AT, the window must contain at least 23h55m of queryable post-apply samples within the same cardinality limits before CORE_COMPLETE is recorded and PR #46 is closed as superseded.

08Evidence

Task 7 authoritative run 20260813T174749Z-b10f6315, dashboards rendered in a disposable Grafana against branch artifacts.

9 / 18 / 0
dashboards, kiosk captures at 1440x900 desktop and 390x844 mobile, failed captures.
70 / 6
variable checks and data-link checks, 0 failures in each.
18 / 32
horizontal-reach and scope-selection checks, 0 failures in each.
974 / 0
datasource queries and failures, and 0 page errors.
91
query-harness targets, 86 PromQL and 5 LogQL, over from=now-6h, to=now, step 60s: 84 pass with data, 7 pass empty and individually justified, 0 failed.
10 / 10
promtool churn cases passed inside a digest-pinned prom/prometheus image run with --network=none.
27 / 27
Safety family and source pairs resolved to exactly one latest group, so stale Pushgateway outcome groups cannot contaminate current readings.
Cleanup
Container and workdir removed and verified absent, port-forwards 19090 and 13100 killed, no production mutation.

Task 10 gates passing locally: portable temporary-root tests; the Task 3, 5, 6, 8 and 9 contracts; the Task 8 pipeline and failure harnesses; Terraform fmt and validate; dashboard JSON parsing; diff check; changed-path and forbidden generated-path checks; and the static resource preflight, which found only the allowed kubernetes_manifest.airflow_statsd_service_monitor addition.

09Residual risk

From the Task 10 receipt and Task 7 validation. Severity is the reviewer's calibration of what could still go wrong before apply.

Showing all 6 risks.

10Recorded verdicts

Reproduced as recorded, not re-graded.

AgentModelRoleVerdict
sisyphusopenai/gpt-5.6-terraTask 10 implementation and local QAapproved-with-notes
Momusunknownplan amendment reviewapproved
oracleunknownplan amendment reviewapproved