ragleap-observability

Install

Like the other infrastructure packages, this ships no functional Python code — the real deliverable is the helm/ chart. The package exists so it can be versioned and released through the same PyPI pipeline as every other RagLeap package.

pip install ragleap-observability
# or, with uv
uv add ragleap-observability

What's inside

Step 1 of a deliberately sequenced build order: Prometheus and exporters first, since alerting or dashboards on data that doesn't exist yet is pointless.

Live verification — 7 real bugs found and fixed

Deployed against a real kind cluster, into the same namespace as a live ragleap-ops db deployment — not a mock. None of these were caught by helm lint/helm template alone.

• runAsNonRoot rejected the exporter image — a different root cause than the usual root-default-image bug. The image sets its user by name (nobody) rather than a numeric UID, which Kubernetes cannot verify against runAsNonRoot without an explicit number. Fixed with runAsUser: 65534, confirmed via docker run --rm --entrypoint id ... postgres-exporter:v0.15.0 before assuming.
• The ragleap-ops monitoring-role dependency, previously just a documented gap, is now closed and live-verified. ragleap_monitor confirmed to genuinely exist in the live database via psql, not assumed from the schema file.
• Cross-namespace DNS lookup failure in Promtail's Loki client, initially misread as a CoreDNS health problem. The client URL used a bare service name (ragleap-loki), which only resolves via CoreDNS within the querying pod's own namespace. A CoreDNS restart did not fix it — the real signal the root cause was elsewhere. Fixed by switching to the fully-qualified ragleap-loki.ragleap-core.svc.cluster.local.
• kubernetes_sd_configs produced 0 active targets despite correct service discovery — root-caused through two layers. Promtail auto-injects a node-scoping field selector built from $HOSTNAME, which without an explicit override defaults to the pod's own name, not the real node name. Fixed via the Downward API. A hand-built multi-capture regex for __path__ then matched real on-disk paths exactly by hand, yet Promtail's own /ready endpoint still reported no tailable logs — replaced with Grafana's own canonical production pattern.
• A missing newline silently corrupted a ConfigMap's YAML document boundary. target_label: container--- with no line break before the --- separator merged the ConfigMap into the next resource. Caused a cascade of confusing symptoms across several helm upgrade cycles: a silent "success" that never created the real object, an "unknown field" warning, and a downstream crash-loop. Root-caused by comparing helm get manifest output byte-for-byte against kubectl get configmap.
• Loki's default ingestion rate limit (4 MB/s) rejected the historical-replay burst generated every time Promtail's positions.yaml reset. Fixed by raising ingestion_rate_mb/ingestion_burst_size_mb in Loki's limits_config.
• A temporary diagnostic scrape job silently starved the real job of every file it touched. positions.yaml keys purely by file path, not by job, so once both jobs existed the older job's tailer permanently occupied every position — discovery and relabeling looked correct, but the real job had nothing left to tail. Removing the diagnostic job was a necessary fix, not cleanup.

After both fixes: postgres-exporter reached 1/1 Running, 0 restarts. /metrics port-forwarded and curled directly, returning pg_up 1 plus real non-zero pg_stat_database statistics for ragleap_core — genuinely scraped data, not placeholder zeros.

Known limitations

Stated plainly, not hidden — same discipline as every other package here.

• Neo4j Prometheus support is unverified and conflicting across sources. Neo4j's own KB and a working xk6-neo4j example show it configured successfully against Community 4.4.x; a separate monitoring vendor's compatibility notes claim Community Edition is unsupported for their collector. neo4jExporter.enabled stays false until independently live-verified against the real neo4j:5-community image.
• Grafana and Loki are now built and live-verified; AlertManager is not. Correctly sequenced last — alerting on data that doesn't exist yet is pointless. Loki's retention (7 days) is also unverified for real storage-sizing needs, flagged as a placeholder.