ragleap-observability

Install

Like the other infrastructure packages, this ships no functional Python code — the real deliverable is the helm/ chart. The package exists so it can be versioned and released through the same PyPI pipeline as every other RagLeap package.

Chart in the wheel: from 0.8.1, pip install ragleap-observability includes helm/. Earlier wheels contained only the Python module, so use a repository checkout for those.

pip install ragleap-observability
# or, with uv
uv add ragleap-observability

What's inside

Step 1 of a deliberately sequenced build order: Prometheus and exporters first, since alerting or dashboards on data that doesn't exist yet is pointless.

Live verification — 8 real bugs found and fixed

Deployed against a real kind cluster, into the same namespace as a live ragleap-ops db deployment — not a mock. None of these were caught by helm lint/helm template alone.

• runAsNonRoot rejected the exporter image — a different root cause than the usual root-default-image bug. The image sets its user by name (nobody) rather than a numeric UID, which Kubernetes cannot verify against runAsNonRoot without an explicit number. Fixed with runAsUser: 65534, confirmed via docker run --rm --entrypoint id ... postgres-exporter:v0.15.0 before assuming.
• The ragleap-ops monitoring-role dependency, previously just a documented gap, is now closed and live-verified. ragleap_monitor confirmed to genuinely exist in the live database via psql, not assumed from the schema file.
• Cross-namespace DNS lookup failure in Promtail's Loki client, initially misread as a CoreDNS health problem. The client URL used a bare service name (ragleap-loki), which only resolves via CoreDNS within the querying pod's own namespace. A CoreDNS restart did not fix it — the real signal the root cause was elsewhere. Fixed by switching to the fully-qualified ragleap-loki.ragleap-core.svc.cluster.local.
• kubernetes_sd_configs produced 0 active targets despite correct service discovery — root-caused through two layers. Promtail auto-injects a node-scoping field selector built from $HOSTNAME, which without an explicit override defaults to the pod's own name, not the real node name. Fixed via the Downward API. A hand-built multi-capture regex for __path__ then matched real on-disk paths exactly by hand, yet Promtail's own /ready endpoint still reported no tailable logs — replaced with Grafana's own canonical production pattern.
• A missing newline silently corrupted a ConfigMap's YAML document boundary. target_label: container--- with no line break before the --- separator merged the ConfigMap into the next resource. Caused a cascade of confusing symptoms across several helm upgrade cycles: a silent "success" that never created the real object, an "unknown field" warning, and a downstream crash-loop. Root-caused by comparing helm get manifest output byte-for-byte against kubectl get configmap.
• Loki's default ingestion rate limit (4 MB/s) rejected the historical-replay burst generated every time Promtail's positions.yaml reset. Fixed by raising ingestion_rate_mb/ingestion_burst_size_mb in Loki's limits_config.
• A temporary diagnostic scrape job silently starved the real job of every file it touched. positions.yaml keys purely by file path, not by job, so once both jobs existed the older job's tailer permanently occupied every position — discovery and relabeling looked correct, but the real job had nothing left to tail. Removing the diagnostic job was a necessary fix, not cleanup.
• Prometheus's Deployment had no explicit rollout strategy, defaulting to Kubernetes' standard RollingUpdate — and its ReadWriteOnce PVC holds a single-writer TSDB directory. The first real rollout since this Deployment was created (triggered by adding AlertManager) tried to start the new pod alongside the still-running old one and failed immediately with opening storage failed: lock DB directory: resource temporarily unavailable, crash-looping until the old pod was manually removed. A latent bug from the original deployment, never exercised until this rollout. Fixed with strategy: type: Recreate, which fully terminates the old pod before starting the new one.

After both fixes: postgres-exporter reached 1/1 Running, 0 restarts. /metrics port-forwarded and curled directly, returning pg_up 1 plus real non-zero pg_stat_database statistics for ragleap_core — genuinely scraped data, not placeholder zeros.

Optional Alloy log shipper (0.7.0)

Grafana has ended Promtail and replaced it with Grafana Alloy. The chart can run Alloy instead, off by default; Promtail stays the default. To switch, set alloy.enabled=true and promtail.enabled=false. Enabling both fails at render time, because every log line would be shipped twice.

• Verified end to end in Docker with the real Alloy and Loki images against the chart's own config: labels and CRI parsing are correct, a restart resumes from its saved position without repeats or losses, and Loki stores each line once even when a fresh Alloy replays lines that Promtail already shipped.
• Verified live on a single kind node. A marker pod printing a numbered line every second stayed in order, with no gaps and no repeats, in one Loki stream across a switch to Alloy and back. The Alloy ClusterRole lists only pods, and its service account cannot read the pod log subresource or secrets.
• Trade-offs. Alloy runs as root, because the node's pod log files are root-owned. It keeps its read positions in alloy.dataHostPath on each node, which is the chart's only writable host path, and removing Alloy leaves that directory behind. A fresh Alloy replays existing files, and Loki rejects entries that are too old, which Alloy logs as entry too far behind.
• Not tested: multiple nodes, sustained load, very short-lived pods, and any hardening beyond the default (security context, resource limits, a read-only root filesystem). Trivy for Alloy v1.20.1 showed 6 rows and 4 unique CVEs against 98 rows and 84 for Promtail 3.3.0 (October 2026, with a fix available).

Availability SLO alerts (0.8.0, opt-in)

Set slo.enabled=true to add fast-burn (1h and 5m, 14.4x) and slow-burn (6h and 30m, 6x) alerts on the Prometheus up metric for slo.availability.jobs. The 99.5% target (slo.availability.errorBudget: 0.005) is a placeholder, not a measured figure.

• Verified: the rule logic, with promtool unit tests (healthy, sustained outage, one-minute blip, one-hour outage). Breaking each threshold made the tests fail. With defaults the rendered chart is unchanged.
• Fix: rule_files now loads even with AlertManager off. Before, PostgresExporterDown was silently inactive in that case.
• Evaluated on a real Prometheus: v3.13.4 in a single container, without Kubernetes, at a 15 second scrape interval, using the 0.8.2 rules. A stopped target made fast burn and PostgresExporterDown fire, and both cleared after recovery. A target removed from the scrape config made RagleapTargetAbsent fire (PostgresExporterDown did not, as expected), and it cleared when the target returned. Evaluated in Kubernetes: on a throwaway kind cluster on a GitHub Actions runner (chart 0.8.3, a stand-in Postgres so the real postgres-exporter ran, and a stand-in Slack-compatible receiver). The same alerts fired and cleared, and AlertManager delivered the firing and resolved notifications to the stand-in. The manual test is tests/k8s/obs-k8s-test.sh. Not verified: slow-burn firing, a real Slack or email send, Loki and Grafana, and multi-node behaviour.
• Limits: this measures target reachability, not user experience, because the app exports no request metrics yet. A target that vanishes entirely does not trigger the burn-rate alerts. From 0.8.2 a separate RagleapTargetAbsent alert fires when a listed job has no up series for 5 minutes (rule logic tested only). If you list a job this chart does not scrape, that alert fires permanently.

Known limitations

Stated plainly, not hidden — same discipline as every other package here.

• Neo4j Prometheus support is unverified and conflicting across sources. Neo4j's own KB and a working xk6-neo4j example show it configured successfully against Community 4.4.x; a separate monitoring vendor's compatibility notes claim Community Edition is unsupported for their collector. neo4jExporter.enabled stays false until independently live-verified against the real neo4j:5-community image.
• AlertManager is wired to Prometheus and has optional Slack and email receivers (off by default); a real Slack/email send is not yet confirmed. Firing and resolved delivery of PostgresExporterDown were verified live against a Slack-compatible stand-in, with credentials read from a mounted Secret. Until a receiver is enabled, alerts go to a placeholder webhook. PagerDuty is not supported. Loki's retention (7 days) is also unverified for real storage-sizing needs, flagged as a placeholder.
• Grafana's admin password defaults to a placeholder when no option is set. The password is now configurable (grafana.admin.password or grafana.admin.existingSecret) and the install prints a warning, but upgrading an existing release will not change it: use grafana cli admin reset-admin-password inside the pod (verified live), which leaves the chart's Secret out of sync.