59 of 59 traces were the same 250ms heartbeat. The cron polling its own queue.
When we deployed our new observability stack at Tacavar, I pulled up the three dashboards I’d built over the previous week and felt that rare, quiet pride of an operator watching a system hum. Green panels. Smooth heatmaps. Traces scrolling in real time. It was the kind of dashboard you leave open on a second monitor because it proves the system is alive.
## The Three Dashboards That Looked Alive
The stack was textbook: Tempo for tracing, Prometheus for metrics, Grafana tying them together, all instrumented with OpenTelemetry SDKs across our agents. We had three dashboards rotating on the big screen in the Tacavar Ops channel.
**Bailian Team Overview** gave us request rates, token consumption, error budgets for each agent service. **Paperclip Swarm Observability** zoomed into the multi‑agent pipeline that moves paperclip tasks from intent to execution. **Tacavar Ops** covered the monitoring system itself—collector throughput, span drops, query latencies.
Every panel rendered something. Latency heatmaps showed tight distributions around 200–300ms. Request counters trended up and to the right. Even the panels where numbers sat at zero were styled with clean axes and subtle fill gradients, as if they were breathing. Nothing looked broken. And that was the problem.
## Probing the Traces: The Heartbeat Illusion
A weekend on‑call shift turned into a forensic exercise. I wanted to understand a single slow trace that appeared in the 99th percentile. So I went to Tempo, opened the trace view, and hit the search query that fed the main traces panel: `{service="paperclip-worker"}`.
In the last hour there were exactly 59 traces. Not thousands. Not hundreds. Fifty-nine.
Every one of them was the same operation: `paperclip_handle_heartbeat`. A 60‑second cron job polling its own work queue. 250ms of connectivity check, an empty response, a 200. No agent runs. No LLM calls. No tool invocation. No task outcomes. The swarm had been sitting idle—or maybe completely down—and our beautiful Grafana boards had turned a single heartbeat into what looked like a fully operational system.
The Bailian Team Overview dashboard was worse. Its metrics panels queried `agent_calls_total`, `agent_tokens_total`, `agent_cost_usd_total`—metrics that didn’t exist anywhere in our exports. The Prometheus datasource returned an empty result set. Grafana rendered that absence as an unbroken time series line at zero, identical to a service running at genuinely low but real throughput. I’d been staring at painted voids for days.
## Why Missing Data Looks Identical to Healthy Data
This is a property of the toolchain, not a configuration mistake. When a PromQL query returns no data, Grafana shows an empty graph. When a trace query matches only infrastructure noise, Tempo returns what looks like sample activity. The rendering layer has no concept of whether missing data means a dead system or a quiet one.
For founders and operators, this is catastrophic. Observability exists to reduce uncertainty. If the same visual output can mean “everything is fine” or “nothing is running,” the system is not providing observability—it’s providing decoration. You cannot tell the difference without interrogating the data directly, and nobody does that on a Tuesday morning when the dashboards are green.
This gap is especially dangerous with OpenTelemetry auto‑instrumentation. It gives you spans for HTTP calls, gRPC handlers, health checks. Those spans will keep arriving even if your business logic is a ghost town. The infrastructure is alive. The product is dead. Grafana will show you a living system.
## The Grafana Anti-Pattern: Graceful Degradation
Call it the **dashboard anti‑pattern**: systems that fail silently, rendering emptiness as elegance. Grafana’s graceful degradation is a design philosophy that works well for general‑purpose charting. In observability, it’s a liability. A broken dashboard forces investigation; a beautifully empty one trains you to stop looking.
I’ve seen teams that install the same dashboard for a new service and immediately feel they’re done. The panels populate with zeros, the legends appear, the axes scale. No one asks whether the queries are valid or the underlying data exists. Weeks pass before someone notices the service isn’t emitting the metric at all.
At Tacavar, we started referring to this as the “heartbeat trap.” A single cron job can animate an entire dashboard—cron interval bar charts, duration heatmaps, error ratios that show 0% failure on a sample size of 59. It’s statistically perfect and operationally meaningless.
## How We Rebuilt the Stack With Query Validation
We didn’t throw out Tempo or Grafana. We rebuilt the layer that sits between dashboards and data—the queries themselves—so that emptiness is never mistaken for health.
The first fix: query‑validation sidecars that run alongside every dashboard refresh. For metrics panels, the sidecar counts rows. If `agent_calls_total` returns zero time series or an empty instant vector, the dashboard gets a red overlay and an explicit `NO_DATA` annotation. Not a green zero. A red flag. For trace panels, we track distinct operation names per service. If the set is `heartbeat` plus one or two infrastructure spans with no business spans, the panel shows a warning state.
Second fix: we removed zero‑baseline charts wherever zero is not a valid operating state. A latency heatmap with all values at zero is not a healthy system—it’s a system producing no latency because nothing is happening. Those panels now require a minimum data threshold before they render as normal.
Third fix: every Tacavar dashboard exposes a small “query diagnosis” row at the bottom—raw row counts, last sample timestamp, distinct operation names from Tempo. You can collapse it, but it’s always there. It turns a rendered dashboard into a piece of evidence, not a painting.
## One Rule for Every Observability Setup
Whether you’re on a team of two or two hundred, the rule is the same: **never trust rendered output alone**. Before you ship a dashboard, run the underlying queries manually. Count the rows. Verify the metric names exist. Know what zero means. If you can’t distinguish between a healthy idle system and a broken one, your observability stack is lying to you.
This isn’t a tool problem you can solve by switching from Tempo to something else or adding another panel. It’s a discipline problem. Grafana, Tempo, and OpenTelemetry give you incredibly fast paths to beautiful visualizations. The faster the path, the more diligent you have to be about validation, because the beauty hides the gaps.
We learned this by staring at 59 identical heartbeats and a dashboard that never once told us our agents weren’t running. Don’t wait for your own version of that moment. Validate your queries. Count your traces. Make zero mean something.
Tacavar builds monitoring stacks that fail loudly, not beautifully. If your dashboards need a second look, we can audit them at tacavar.com.