Another miniseries I am starting in this newsletter is “Engineering fundamentals”, the parts of software engineering that I believe remain true no matter what the circumstances are. Today it’s observability, which is critical to how a system runs and easy to leave as an afterthought.
Most engineering fundamentals get treated like table stakes. They get no attention while everything is working, and observability is the one that is easiest to postpone and most expensive to retrofit.
What observability is
It’s your ability to tell what is going on inside your system at any point in time. Three things make that possible, and you need all three: something recorded, something connected, and something aggregated. Good-quality observability means you can infer what is happening inside the software when you need to look.
There are 3 core parts of observability: each one answers a different question. Logs tell you what happened. Traces tell you the path it happened along. Metrics tell you how often, and whether that is getting worse.
Logs are timestamped messages emitted by your services. They’re not tied to any particular request and they rarely tell you where in the code they came from, but correlated with a trace you can find the request they belong to.
Traces record the path a single request took as it moved across your services, made up of spans, one per unit of work. They’re how you find the root cause of a performance problem in something you can’t reproduce locally.
Metrics are numeric aggregations over time: error rate, request rate, CPU, latency. Good for spotting trends, and close to useless for explaining one specific event.
I’m not going deep on any of these here, they each deserve their own post and you wouldn’t want to read a Substack post that long. Instead I want to explain why it matters and how I would go about designing it.
Why you need it
When you have a compiled software artifact running continuously on a server in the cloud, you can’t attach a debugger to it and work out what a user is experiencing. Nor can you get a customer to reproduce the scenario for you, you might get that far depending on how mature your support function is, but it’s still a prohibitively expensive way of finding out what’s going on.
You need logs so you can dig into customer problems when they happen, traces to connect those logs into a single story, and metrics to find out what is broken before a customer tells you.
The spectrum of systems with observability runs from a no observability, which requires a high-touch model, where your operations team spends one-on-one time with customers finding and debugging what is broken, through to a fully automated system firing telemetry 24/7, detecting problems and self-healing without a human ever getting involved.
Observability lets your team debug the system without having to spend cycles reproducing the issue. That’s why you need it.
That is the goal for most systems. In practice, the volume of telemetry required makes it unreachable. The goal you can actually hit is enough observability that your operations team can trivially detect and fix any issue.
This is the part that makes observability non-negotiable for me, and it’s the part that decides how I would design it.
Designing your observability
You can’t instrument your entire system. You can’t add a debug log to every line of code. That’s far too much strain on any ingestion system, especially at the scale of millions of users. So instrumentation has to be designed, thoughtfully and tactically.
You start with what is useful. With what has to work. Which means you have to understand your customers, what they use your system for, and what they get out of it.
Let’s take a deliberately ridiculous example, because I want something concrete to design against. Let’s say we run a national network of instrumented beehives rented out as pollination services, Pollination-as-a-Service. Farmers pay per hectare, we place hives on their land, the hives forage, and at the end of the season the farmer gets a pollination score for every block plus a bill.
The journey that matters is short: sign up, hives get placed, the hives report their own condition, the season closes with a report and an invoice. The farmer cares about exactly one thing, did the crop get pollinated, and nothing else in this system matters unless that number moves.
Now we can say which parts matter. Which parts have to be reliable, and which parts have to be fast?
Ingest: telemetry ingest and retention have to be reliable. A season is 180 days: 1,200 hives, each reporting 1,440 readings a day, 311 million readings. Lose the data for 2% of the hives and 24 of them effectively have no season data, which means we send a number we can’t stand behind.
Placement: placement and haulage have to be reliable. They depend on road transport and weather, both outside our control, and a missed placement window is a missed crop.
Alerting: it has to be reliable, because the failure is silent. Nobody notices an alert that never fired until the hive is empty and the season is gone.
Dashboard: the farmer dashboard and pricing quotes have to be fast. Farmers check their hives first thing in the morning. Anything slower than a second and they go back to driving past the hives and looking through the mesh, which tells them nothing.
Now match each one to the signal it needs.
Ingest gets telemetry completeness: one metric, the percentage of hives that reported today, with an alert when it drops. That single number catches dead devices, broken collectors and bad firmware in one query.
Ingest and Placement share traces across the hive-to-cloud hop, so when a reading goes missing you can see which hop ate it.
Placement gets structured logs recording the decision inputs: which farm, which bloom window, which truck, because this is the component where you’ll be asked why you did it that way six months later.
Dashboard gets request rate, p95 latency, error rate. Standard, boring, correct.
Alerting gets metrics for alert precision, because an alert nobody trusts is an alert nobody reads.
Hive agents report once a minute: 20 readings a second fleet-wide across 1,200 hives, roughly 800 bytes each, so 16 KB/s and 1.4 GB a day. At roughly 10 cents a gigabyte on a managed backend that is about $50 a year, and that is before anyone runs a query.
Now the version that gets proposed in a design review: let us also stream the hive audio, then we can do acoustic swarm detection. 48 kHz stereo at 16-bit is 192 KB/s per hive. Across 1,200 hives that is 230 MB/s, roughly 20 TB a day, or about $60,000 a month. Same idea, and 14,000 times the ingest bill, for a stream nobody has shown they need.
The version that survives review is on demand. A six-second clip is 1.15 MB, so 20,000 of them a month comes to 23 GB, about $2,300. You only pay when there is a reason to listen.
Detail is never free, and neither is its absence. The trade-off between the usefulness of the detail and what it costs has to be made on purpose, because at a certain scale the bill points you to the right architecture.
Iterating on observability
Once it’s implemented, the important thing isn’t stopping. Observability has to evolve with the system or it becomes useless. Staying with the ridiculous example, because the numbers are what make the point. These are the triggers I watch for. None of them has a fix you apply once, because in every case the next step is the same: adapt the design to what changed:
New requirements: we add a consumer tier so backyard beekeepers can rent a single hive. That takes us from 1,200 hives to a projected 60,000. Fleet-wide ingest goes from 20 readings a second to 1,000 readings a second, and every component that assumed a known and bounded fleet now has to handle auth and billing for a customer base 50 times larger. The original design wasn’t wrong, the world moved underneath it. Our next step is to adapt the design: the scale numbers have changed, so re-do the maths and figure out any new bottlenecks.
Requirements we missed: on-demand clips made an acoustic swarm detector cheap enough to ship, so in the ridiculous version we shipped one, and it’s wrong 4% of the time. Across the 20,000 clips we pull a month, that is 800 false swarm alerts, and inside a quarter the beekeepers are muting the channel, most of what it sends ignored within 30 days. The alert is perfectly reliable and completely useless. A better notification channel wouldn’t have fixed it. Our next step is to adapt the design: add a
swarm_confidencemetric and alert on low confidence instead of on every detection, so the channel stays trustworthy.Internal bugs: a firmware release across 5% of the fleet silently changes weight sampling from every 60 seconds to every 120. Every dashboard for those hives still looks fine, because half of their data is still data. Nothing surfaces it for 11 days, and when the season reports go out, 60 hives have numbers that are quietly wrong. Your monitoring needs monitoring. Our next step is to adapt the design: break the completeness gauge down by firmware version.
Growth: nothing about the design changed, the fleet just got bigger, and what mattered at 1,200 hives stops mattering at 60,000. At 1,200 hives you can alert on every individual event and pull up a dashboard per hive. At 60,000 you alert on fleet-level indicators like the percentage of hives reporting today, then go and look at the individual hive when the fleet number moves. Our next step is to adapt the design: move the default aggregation level up with the fleet. The right level of aggregation is a function of your scale, not of your taste.
Observability in a world of AI
AI writes a lot of our code now, and the further we go the less of any system lives inside an operations team’s head. You can’t debug what you have never read. In that world observability is even more important.
If you want to go deeper, start with OpenTelemetry (OTEL). It’s the standard framework for observability and the main thing it gives you is vocabulary, so that “span” means the same thing in your design docs, in your SRE runbooks, and in your vendor contract. It’s a twenty-minute read: OpenTelemetry observability primer
One line to take from it: an application is properly instrumented when the people debugging it never have to go and add instrumentation first.
Observability is table stakes. Design it early, on purpose, and expect to keep paying attention to it.