I had a beautiful Grafana stack. 59 of 59 traces were the same heartbeat cron. Zero actual work. That sentence should have been the first thing I saw when I opened my Grafana dashboard. Instead, it took me a week and a direct query into Tempo to find it. The dashboards looked perfect. Every panel rendered, every legend populated, every timeseries was smooth and flat. The problem wasn't that my observability stack was broken. The problem was that it was too good at looking fine. This is the story of how I learned to count rows instead of admire panels. ## The Setup: Tempo, Grafana, OpenTelemetry, and Three Dashboards The stack was as standard as it gets for a modern agent platform. OpenTelemetry traces from every service were sent to Tempo for trace storage. Tempo tracing gave me distributed trace context, but only for the operations that existed. Grafana dashboards gave the whole team a single window into latency, errors, and volume. We had three dashboards: Bailian Team Overview, Paperclip Swarm Observability, and Tacavar Ops. Each one was tuned for a specific audience—engineers, operations, and leadership. I wanted agent monitoring that would catch every phase of an agent run: context assembly, model invocation, tool calls, and final output. To support that, I instrumented the services with OpenTelemetry and exported Prometheus-style metrics for cumulative agent calls, token counts, and cost. The dashboards consumed both Tempo and Prometheus, so they showed traces and counters side by side. It was a textbook observability architecture. The first week, the panels moved. Latency histograms appeared. Trace lists populated. I remember watching the bars for `agent_calls_total` slowly climb. I assumed the numbers were real because they looked real. The axis ticks were there. The legend colors matched. I had no reason to doubt the rendering. And that was exactly the problem. ## What the Panels Showed Me Take the Tacavar Ops dashboard as an example. The top row showed request rate and error rate. The middle row showed p50/p95/p99 latency. The bottom row was a heatmap of trace spans by operation. Every panel had the same pleasing shape: a left-to-right line hugging the bottom, occasionally punctuated with modest spikes. The error rate was exactly zero. The p95 was stable. The heatmap was mostly green. This is what we all want to see: a healthy, boring production system. The dashboards were green, and green is success. I showed them to the rest of the team during our Friday review and no one asked a single difficult question. Why would they? The dashboards were beautiful. ## Probing the Queries: 59 of 59 Traces Were a Heartbeat But something kept niggling at me. The trace list always showed exactly a few dozen traces in the last hour, and they all had the same friendly operation name. I didn't recognize it from the core agent flow. So I opened Tempo's query UI and asked for all traces in the past 60 minutes. Tempo returned 59 traces. I grouped them by operation name. Every single one was `paperclip_handle_heartbeat`. The heartbeat cron fires every 60 seconds and polls a work queue. It doesn't execute an agent. It doesn't call an LLM. It doesn't invoke a tool. It checks whether there is work, sees none, and returns. That's not a production workflow. That's an idle loop. I expanded one trace to be sure. The whole span took 250 milliseconds. The parent span was the cron trigger; the child span was the queue poll. There were no nested agent spans, no token accounting, no tool invocation. The trace was real but the operation was noise. Zero agent runs. Zero LLM calls. Zero tool calls. Zero task outcomes. My entire observability stack was monitoring a single cron process that spends its life checking an empty queue. ## The Missing Metrics That Rendered as Empty Styled Timeseries I then clicked over to the Bailian Team Overview dashboard, the one with the agent call counters. The panels looked identical to the ones I'd been watching for days. `agent_calls_total` was a flat line at zero. `agent_tokens_total` was a flat line at zero. `agent_cost_usd_total` was a flat line at zero. The legend showed the metric names. The color was a solid, confident blue. The zero baseline was inside the panel with plenty of headroom. I opened Grafana's Explore tab and ran `count(agent_calls_total)` against Prometheus. The query returned "no data". The metric had never been ingested. No agent call had ever been recorded. Same for the token and cost metrics. They didn't exist in the Prometheus store. The panels weren't lying in the traditional sense. They were displaying the output of a query that returned no rows. Grafana styled that empty result as a clean zero-value timeseries, identical in every visual way to a metric that had genuinely been zero over the same period. That's not a rendering bug. It's a design choice that turns missing data into a silent lie. ## Why Graceful Degradation Is an Anti-Feature Observability tools have a bias toward being useful even when data is missing. If a PromQL query fails, Grafana still renders the panel with an empty frame rather than an error. If a trace query returns zero spans, Tempo returns an empty list instead of a loud "there is no data." This is graceful degradation, and in most software systems it's a virtue. In observability, it is an active threat. A broken dashboard makes people investigate. A beautifully empty one makes people think their system is working. This is one of the most dangerous observability anti-patterns, because it inverts the feedback loop. Monitoring exists to tell you when reality differs from expectation. If missing data is rendered as a healthy zero, expectation collapses. You never know when you're blind. I'm not saying dashboards should crash on a missing metric. I'm saying they should mark absence as absence—use a distinct pattern, an "N/A" label, a gray dashed line. Anything that makes "I see no data" visually different from "I see zero data." ## How to Audit Your Own Observability Stack You can find these traps without waiting for a week-long coincidence. Here is the audit I run on every new dashboard now: 1. **Run every query in your dashboards directly.** Don't trust the panel. Open the query editor, copy the PromQL or Tempo query, and run it in Explore. Count the rows in the raw result. If you see fewer than you expect, you have a data problem, not a visualization problem. 2. **Check for empty metrics.** For each metric name used in a dashboard, query `count(metric_name)` or use a label query to confirm the metric exists. If it returns nothing, the dashboard is synthesizing a quiet zero. 3. **Look for the same operation repeated.** In Tempo, group traces by operation name. If one operation dominates and it's a heartbeat or health check, your real workflows are not being traced. 4. **Instrument a known operation.** Trigger a real agent run or an LLM call. Confirm a new trace appears with the expected spans. If not, your instrumentation has a gap. 5. **Set alerts on absence.** In Prometheus, use `absent()` or a recording rule that fires when a series stops existing. The alert should say "metric missing", not just "metric high." Your pager should see absence as an outage. 6. **Make missing data visually loud.** Use a custom value mapping in Grafana to display `N/A` for null, or set thresholds so an empty series renders as a distinct gray. Make absence impossible to confuse with zero. ## The Generalizable Lesson: Count Rows, Not Panels The panels are decorations. Queries are ground truth. I had three dashboards with dozens of panels and not a single one told me the system was idle. Everything rendered beautifully because the rendering engine was too graceful to complain. The only way to catch it was to stop looking at the rendered output and start looking at the raw query results. Count rows, not panels. When you set up a monitoring stack, the first question is not "does this look good?" It's "how many records does this query actually return?" If the answer is zero or one—that heartbeat—you have an observability problem, not a data problem. A dashboard that renders cleanly from dirty data is worse than no dashboard at all, because it gives you the confidence to ignore what is actually happening. Tacavar's own infrastructure went through this exact audit. The stack was real, but the signal was empty. Now we treat empty dashboards as incidents. Put your own infrastructure under Tacavar's monitoring microscope at tacavar.com.