Vantage: finding the cause of an outage in under a minute
For the person on call at 3 a.m., whose first question is where did it start?
When a large software system breaks, one fault sets off hundreds of alerts in the services that depend on it, and the person on call has to dig through logs under pressure to find where it started. Vantage is a design concept that does the digging. It cuts the flood down to the alert that started it, draws the one path through the system that matters, and points at the slowest step. An AI reads the evidence and suggests a fix, and a person approves it before anything runs.
- Context
- Concept exploration
- My role
- Designed and built it alone, end to end
- What I built
- An incident console that traces hundreds of alerts back to the one that started them
- Status
- A working prototype on made-up data. Not tested with users
Working prototype: Vantage, running on made-up data. Open in a new tab
01 · The problem
slow to fixHundreds of alerts, and the cause buried under them
When a large system fails, one fault sets off alerts in every service that depends on it. The person on call, usually woken for it, has to find the one that matters before customers notice.
Every minute to resolution costs money and trust, and the tools often make it slower. An alert that fires on every symptom teaches people to ignore it, and a wall of dashboards leaves the person to connect the dots alone. The job was to cut the flood to the one thing worth looking at first, and to get there in seconds.
Where this comes from
Alert fatigue is well documented in on-call guidance, PagerDuty's among others, as a main cause of slow response and burnout. Netflix has described its engineers' troubleshooting as detective work across scattered logs and dashboards, which is what its tracing tool Edgar was built to shorten.
Live example · Basalt
made-up data412 alerts cut down to the one that started them. Drawn live in Basalt, the dark design system the console uses.
02 · Constraints
four limitsWhat the design had to work within
An on-call tool is read by a tired person who did not choose the moment. Each of these shaped every screen.
Read at 3 a.m.
The reader is tired, under pressure and often on a laptop in bed. Every screen shows the few facts that move the investigation and keeps the rest one step away.
The AI only reads
Vantage can rank causes and suggest a fix. Anything that changes the live system runs through Oversee, a separate control screen, after a person approves it.
No charting library
Every chart is hand-built SVG that works from the keyboard and reads aloud to a screen reader. An on-call tool has to meet accessibility rules like any other.
Made-up data
There are no production traces to design against, so the console runs on one scripted incident, built to behave the way a real one does.
03 · Decisions
three callsThree decisions that shaped it
Each one bought speed at the moment of the incident and paid for it somewhere else.
Decision 1
Measure time to insight, not time to resolution
- What I chose
- One measure for every screen: the time from an incident starting to a person understanding why. In the scripted incident it is 41 seconds.
- What else I considered
- Mean time to resolution, the number teams already track, and the raw alert count.
- Why
- Resolution depends on the fix, the team and the deploy pipeline. Insight is the part a console can shorten, so it is the part the design answers for.
- What I gave up
- A number teams can compare with their current tools on day one. Time to insight has to be measured in a study before it means anything.
Decision 2
Open on the cause, not on a dashboard
- What I chose
- The console opens on the service that looks wrong, lights one path through the system and shows the slowest step. Detail is one click away, never on the first screen.
- What else I considered
- A chart-heavy console with golden-signal cards, an annotated timeline and bars for where the time went. I built it, and it is kept at the foot of this page.
- Why
- On its own the dashboard was strong. In an on-call console it added reading time to a tool whose whole job is speed.
- What I gave up
- Richer charts at the moment of the incident. They moved to the postmortem, where there is time to read them, and to the examples on this page.
Decision 3
The AI suggests, and a person approves in a separate tool
- What I chose
- Vantage reads the evidence, ranks likely causes with a confidence level and the metrics, traces and deploys behind each, and suggests a fix that can be undone. The fix runs only after a person approves it in Oversee.
- What else I considered
- Letting the console apply a reversible fix itself, with an undo.
- Why
- At 3 a.m. the cost of a wrong automated change is higher than the seconds a handoff costs, and an undo is only as good as the person awake to use it.
- What I gave up
- Speed, one extra screen, and a handoff between two tools that have to agree about the incident.
Example · timeline
recoveredThe whole incident on one line: a spike at 3:02 a.m., the cause found 41 seconds later, and the impact stopped in under two minutes. That 41 seconds is the number every screen is designed to shorten.
Example · golden signals
four small chartsThe four golden signals (latency, traffic, errors, saturation), each as its own small chart. Status shows as a shape and a word, never color alone.
04 · One incident, start to finish
four screens, in orderOne incident, from the 3 a.m. alert to the postmortem
Here is the scripted incident as four screens, in the order the person on call sees them: the page on a phone's lock screen, the flood of alerts with no clear source, the console that finds one cause and suggests one fix, and the postmortem, drafted as soon as a person approves the fix. Each screen is the working prototype, so you can click through it.
Screen 1 of 4 · lock screen
Sev-2 alertIt starts on the lock screen at 3:01 a.m.: 412 alerts across six services, no clear source, and one thing to tap.
Screen 2 of 4 · alert flood
412 alertsThis is what you wake up to. Every service is alerting, and none of them is obviously the cause.
Screen 3 of 4 · console
time to insightVantage cuts the flood down to one source, lights up the path through the system, and stops at a fix a person has to approve. The cause is on screen 41 seconds after the spike.
Screen 4 of 4 · postmortem
resolvedClosed in under two minutes, with the postmortem already drafted from what happened.
05 · The visuals
main focusDetailed visuals a tired person can still read
This is the core of the project: detailed visuals a tired person can still read at a glance. I drew them by hand in accessible SVG, with no charting library.
Two views do most of the work. A service topology is a map of how the services depend on each other. It lights up one traced path through them, so your eye goes to the cause instead of counting boxes. A flame graph shows each step of a request as a bar whose width is the time it took, so the widest bar is the answer. Both work fully from the keyboard: every service and step can be reached with the arrow keys, and a screen reader announces what's selected. Packed information and accessibility can go together if you build the visual with care.
The example incident · made-up data
key numbersExample · service map
one path lit upThe service map: each shape is a service, and one lit path shows where the problem runs.
Example · flame graph
width is timeFlame graph: each bar's width is time, so the widest bar is the cause.
Example · where the time went
same answer as barsThe same answer as a bar chart: 79% of the request's time is spent waiting for one database connection.
Proven chart types
Both chart types are well proven. Brendan Gregg created the flame graph to make performance profiles readable at a glance, and service maps built on distributed tracing (following one request through many services) are standard through OpenTelemetry. What Vantage adds is making them work with a keyboard and a screen reader, without a charting library.
06 · The first screen
easy path firstThe console opens on the problem
Open-ended exploration tools are powerful but offer no guidance: endless ways to slice the data, and no view on which one matters right now.
Vantage adds guidance on top. It opens on the service that looks wrong, not a home page. The first screen shows only the few facts that move the investigation, with more detail one step away. Some teams call this the "paved road": the common path is the easy one, and the console steers you to the signal that explains the others instead of handing you a blank query builder.
Where this already exists
Honeycomb's BubbleUp shows which attributes differ inside a problem, and Netflix's Atlas and Telltale do related work across a large fleet. Vantage borrows the idea of guidance, not their machinery.
07 · Reliability targets
error budgetHow much room is left
Up or down is the wrong question during an incident. The useful one is how much room is left before the team stops shipping features and works on reliability.
A service-level objective (SLO) is a reliability target a team sets. The error budget is how much failure that target still allows. Together they turn reliability from a feeling into a number a team can act on. Vantage shows the budget being used up against a safe pace, plus a gauge for what's left, so when an incident eats into it, the trade-off is a clear, shared decision instead of an argument.
Example · error budget
38% leftThe error budget being used up over time, and a gauge of what's left.
From SRE practice
SLOs and error budgets are central to Google's site reliability engineering (SRE) practice: set a target, measure against it, and let the remaining budget decide how fast a team ships. Vantage just makes that budget visible when the decision is being made.
08 · The AI suggests, a person decides
suggests, never actsAn AI that suggests a fix, and a person who approves it
The AI panel reads the evidence and suggests a cause. It never changes anything in the live system. Finding the cause and acting on it are kept apart on purpose.
You ask a question in plain language. Vantage points out what looks unusual, ranks the likely causes with an honest confidence level, and shows the evidence behind each: metrics, traces (records of a request's path) and recent deploys. Then it suggests a fix that can be undone, and stops. Anything that changes the live system goes to a separate control screen, where a person has to approve it.
Suggest → approve
a person approvesThe suggested fix doesn't run here. A person previews and approves it in Oversee, a control screen I designed as a separate project. It has four states: set the scope, preview with a dry run, stop it partway, and roll it back.
See Oversee, where a person approves the fix09 · One answer everywhere
UI · CLI · APIThe same answer in the console, the terminal and the API
Engineers don't work in one window. They use terminals, code editors and scripts, and the answer has to read the same wherever it shows up.
Ask "what's the slowest span in checkout?" (a span is one timed step in a request) and you get the same answer as a console panel, a command line (CLI) result, and a JSON response from the API. That fits how engineers already work with Git and the command line, lets the answer feed into scripts and automation, and means nothing gets lost moving between them.
Example · three places
same answerOne answer in three places: a console row, a CLI line and a JSON response.
Why three places
Meeting engineers in the terminal and the API, not only in a browser tab, is what makes a tool part of how they already work.
10 · Outcome
not yet measuredWhat success would mean: a shorter time to insight
Vantage is a concept with a working prototype on made-up data, so there are no numbers. The measure that would count is time to insight, taken with on-call engineers on a replayed real incident, against the tools they use today.
What exists: a console, a phone flow and a postmortem that run end to end on one scripted incident, where the cause is on screen 41 seconds after the spike; hand-built charts that work from the keyboard and read aloud; a handoff into Oversee for the fix, built as its own project; and the chart-heavy version I set aside, kept as a prototype so the trade-off can be seen rather than described.
11 · Reflection
what I would test firstWhat I would do next
Put it in front of three on-call engineers with a recording of one of their own incidents, and time them to the cause with and without Vantage. I would also ship the command line answer first. It is where they already are at 3 a.m., and it is the cheapest way to find out whether the ranking is right.
Set-aside version · dashboard
not usedThe set-aside version, kept as its own prototype. Good charts, but too much to read on a screen where seconds count.
Where this draws from
The signals come from Google's site reliability engineering practice (the four golden signals, service-level objectives and error budgets), Brendan Gregg's USE method and flame graphs, and the RED method popularized by Tom Wilkie. Alert fatigue as a cause of slow response is documented in on-call guidance such as PagerDuty's. Distributed tracing and service maps are standard through OpenTelemetry, and Netflix has written about Edgar, Atlas and Telltale, the tools its engineers use for the same problems. Vantage is a portfolio concept that learns from that practice; it is not a claim to have built any of it.
More case studies
Three more case studies about busy screens people read under pressure: four earthquakes in Japan told through data, interview practice with scores you can question, and a supervisor watching an AI voice agent during a call.