📈 Episode 10: The Observability Moonshot 📈

MONDAY, 9:00 AM

Wolfy loved observability. Logs, metrics, traces, dashboards, golden signals, cardinality budgets, burn rates. He loved the quiet poetry of knowing exactly why production was on fire before anybody opened Slack.

Unfortunately, Wolfy also loved instrumenting his personal life.

His smartwatch measured heart rate. His chair measured posture. His espresso machine exported Prometheus metrics. His animatronic tail had a Grafana dashboard called Tail Latency and Wag Throughput.

This was fine until the company announced a new executive dashboard.

"We need one pane of glass," said the VP of Engineering. "All important production health in one place."

Wolfy nodded solemnly. One pane of glass. Simple. Elegant. Executive-friendly. He created a folder called exec-prod-overview, connected the approved data sources, and definitely did not notice that one personal Prometheus endpoint was still checked in his local Grafana config.

10:13 AM - The Dashboard Goes Live

Grafana - Executive Production Overview

API P95 Latency

184ms

Checkout Error Rate

0.02%

Queue Depth

Tail Wag Throughput

412 wags/min

The dashboard appeared on the giant office TV.

Four executives, two directors, and one very tired finance person stared at Tail Wag Throughput.

"Is... 412 good?" asked the CFO.

"Context-dependent," said Wolfy, already sweating through his hoodie.

10:16 AM - The Alert

🚨 CRITICAL: TAIL_WAG_SLO_BURN_RATE_ABOVE_THRESHOLD 🚨

ALERT TailWagBudgetBurning severity="critical" summary="Tail wag error budget will be exhausted in 38 minutes" labels: service="executive-prod-overview" source="wolfy-home-lab" moon_phase="waxing_gibbous" emotional_state="trying_to_act_normal"

Slack detonated.

@sre-oncall 10:17 AM
Why is PagerDuty calling me about a tail?

@finance 10:18 AM
Is tail wag throughput billable infrastructure?

@jake-from-marketing 10:18 AM
I knew Q3 morale looked fluffy but this is a lot

@wolfy 10:19 AM
Everyone remain calm. The tail has exceeded its error budget but customer traffic is fine.

That was the exact sentence that caused the CTO to walk into the SRE room.

"Wolfy," said the CTO, "why does your tail have an SLO?"

"Because without an SLO," Wolfy said, "how would I know whether it is meeting user expectations?"

The CTO closed his eyes in the ancient posture of engineering leadership.

10:24 AM - The Actual Incident

Then the checkout error rate jumped from 0.02% to 7.8%.

Everyone stopped laughing.

Grafana - Executive Production Overview

API P95 Latency

4.8s

Checkout Error Rate

7.8%

Queue Depth

18,044

Tail Wag Throughput

0 wags/min

Wolfy's ears flattened.

"The tail stopped wagging before the checkout graph spiked," he whispered.

"That is not a root cause analysis," said the CTO.

"No," said Wolfy, "but it is a correlation with excellent timestamp alignment."

He opened the trace waterfall. Every failing checkout request waited on one dependency: the recommendations service. The recommendations service waited on feature flags. The feature flag client waited on DNS. DNS waited on a misconfigured sidecar.

The sidecar had been deployed at 10:22 AM.

The tail had stopped wagging at 10:22 AM and eight seconds.

$ kubectl rollout undo deploy/feature-flag-sidecar -n prod deployment.apps/feature-flag-sidecar rolled back $ promql 'rate(tail_wag_total[1m])' 412 $ promql 'checkout_error_rate' 0.01

The office TV updated. Checkout recovered. Queue depth drained. Wolfy's tail, visible just behind his chair despite all reasonable HR policies, resumed a gentle operational wag.

11:05 AM - The Postmortem

@cto 11:06 AM
Action item: remove personal metrics from executive dashboards.

@sre-lead 11:07 AM
Counterpoint: tail wag throughput had better leading signal than three of our synthetic checks.

@finance 11:08 AM
I am willing to approve one dashboard panel if it reduces incident cost.

@jake-from-marketing 11:09 AM
Please let me brand this as Emotional Observability

The final postmortem was professional, blameless, and only slightly cursed. It identified the sidecar regression, the missing DNS readiness check, and the fact that Wolfy's "emotionally derived telemetry" was not an approved production signal.

Nevertheless, the SRE team added a new class of alerts the next day:

# Human Signals Policy - Dashboards must use approved data sources - Personal telemetry must never be mixed with production telemetry - Leading indicators are valuable, even when discovered accidentally - No business-critical alert may depend on vibes - Exception process: ask Security, Legal, and apparently Wolfy

The executive dashboard became clean again.

But deep in Grafana, inside a restricted folder named experimental-signals-do-not-present-to-board, one small panel remained.

Experimental Signals - SRE Only

Production Calm Index

99.8%

Wolfy Tail Wag Rate

nominal

📈 THE OBSERVABILITY MOONSHOT 📈