The status page is entirely green. /health responds instantly, availability remains at 99.92%, and there have been almost no alerts. Yet users cannot open a report, and search takes three seconds to respond.
The monitoring system was not wrong. It really checked /health, received 200 OK, and honestly recorded a successful probe.
The problem is elsewhere: the state of one convenient endpoint became a conclusion about the state of the entire product.
There is a less obvious trap too. Even a correctly chosen check sees the service only at isolated moments. A short outage can fit entirely between two probes, trigger no alert, and never appear in the incident history. The final monthly uptime can still be almost exact.
We will test both effects experimentally. First, we will create a month of service operation in which every second of downtime is known in advance. Then we will look at it through monitoring systems with different probe intervals. Finally, we will run a real HTTP service whose health check remains green while its user-facing features do not work. The complete example source is included in the appendix.
The result is that a polished percentage can be correct while the picture of the system is not.
A Month in Which Every Outage Is Known
In production, nobody knows the service's true availability.
We have results from external probes, request metrics, logs, and user reports. Each source shows only part of what happened. If a short outage did not overlap a probe and left no suitable metric, it cannot be reconstructed afterward.
In an experiment, we can define the complete timeline in advance.
Take thirty days and place twenty-six incidents within them:
#[derive(Clone, Copy, Debug)]
pub struct Outage {
pub start: u64,
pub seconds: u64,
}Each Outage stores the starting second and the outage duration.
Twenty-four incidents last less than a minute. Two more last considerably longer. The shortest takes ten seconds, and the longest takes seventeen minutes.
Because we know the service's state at every second, we can calculate its true availability directly:
$ cargo run --release --quiet
month: 30 days (2592000 seconds)
incidents: 26 total — 24 under a minute, 2 longer
shortest: 10 s, longest: 1020 s
downtime: 2077 s (34.6 min)
TRUE uptime: 99.9199 %The service was unavailable for 2,077 seconds during the month, about thirty-four and a half minutes. Its true uptime was 99.9199%.
That is a good result, but the percentage hides the shape of the failures. The service was not down in one continuous block. It entered an unavailable state twenty-six times, and most failures lasted less than a minute.
The difference barely matters to a monthly total. It matters greatly to users and the on-call team.
One long incident is usually noticed quickly, investigated, and tied to a specific cause. Short recurring failures can appear as random errors for months without combining into one obvious outage.
Let us see how many of them monitoring detects at all.
A Five-Minute Probe Does Not Observe Five Minutes
A typical prober does not watch a service continuously. It makes a request, stores the result, and sleeps until the next check.
At the end of the month, availability is calculated as the share of successful probes:
successful probes / all probesRun our known timeline through several intervals:
$ cargo run --release --quiet
TRUE uptime: 99.9199 % (2077 s of downtime, 26 incidents)
interval probes failed reported error incidents seen
300 s 8640 7 99.9190 % -0.0009 pp 4 of 26
60 s 43200 32 99.9259 % +0.0061 pp 11 of 26
30 s 86400 67 99.9225 % +0.0026 pp 20 of 26
10 s 259200 207 99.9201 % +0.0003 pp 26 of 26
1 s 2592000 2077 99.9199 % ±0.0000 pp 26 of 26A check every five minutes reports 99.9190%. The true value is 99.9199%.
Both numbers round to the same 99.92% on a status page. Judging by the final percentage, monitoring reconstructed the month almost perfectly.
But the final column shows a different picture.
The five-minute probe noticed only four of the twenty-six incidents. The other twenty-two occurred entirely between checks.
Users received errors. The service really was unavailable. But monitoring raised no alert and did not preserve the outage as an event.
Its percentage was still almost correct.
How Monitoring Can Miss Failures and Guess Uptime
There is no contradiction. Availability percentage and incident count describe different properties of a system.
A short outage can begin and end between probes. Monitoring then underestimates downtime.
Elsewhere, a check may land inside a ten-second failure. That one failed point receives the same weight as every other five-minute interval even though the service was unavailable for only a small part of it. Here, downtime is overestimated.
Over a long timeline, errors occur in both directions and partly cancel each other out. The average share of successful probes can therefore be close to the true share of available time.
There is no such compensation for incidents.
A particular outage either overlaps at least one check or disappears completely from the observed history. A correctly rounded monthly percentage cannot reconstruct twenty-two missed events.
A periodic check sees only selected moments. If an outage is shorter than the interval between probes, it can occur entirely between them and remain invisible.
Monitoring answered the question of what share of its probes succeeded.
But a stronger conclusion was drawn from the result: the month was quiet, and the service almost never failed.
The probes did not measure that.
The Interval Defines the Size of Invisible Events
As the interval gets shorter, brief incidents gradually appear in the observed picture.
The five-minute probe sees four outages. The one-minute probe sees eleven. A check every thirty seconds finds twenty, while the ten-second check finds all twenty-six.
This does not mean that every service should be probed once per second. Frequent synthetic requests create load, cost money, and can increase noisy alerts.
But the interval cannot be treated as a neutral default setting. It determines the scale of events that monitoring is capable of knowing about.
If a thirty-second outage matters to the product, a check every five minutes is unsuitable for detecting it. Not because the monthly percentage will necessarily be wrong, but because many such events will never enter the sample.
Sometimes that is acceptable. A ten-second disruption may genuinely have no effect on important user workflows.
That should be a deliberate decision: such events are not considered significant, so the monitoring system does not need to record every one of them.
The problem begins when sparse probes create the impression of complete knowledge about the service.
99.92% looks like a detailed account of the month. In our experiment, twenty-two unknown incidents disappeared behind that number.
A Frequent Check Can Still Look in the Wrong Place
Suppose the interval is reduced to one second. Monitoring no longer misses short gaps in time.
One question remains: which path does it check?
Run a small HTTP service on the loopback interface:
match path {
"/health" => respond(&mut stream, "200 OK", "ok"),
"/api/report" => respond(
&mut stream,
"500 Internal Server Error",
"report backend unavailable",
),
"/api/search" => {
sleep(Duration::from_millis(slow_ms));
respond(&mut stream, "200 OK", "results")
}
_ => respond(&mut stream, "404 Not Found", "no such path"),
}The process is running, accepting connections, and responding over HTTP. But the three endpoints are in entirely different states.
/health instantly returns the predefined response ok. It does not access a database, queue, or external API.
/api/report depends on an unavailable backend and returns an error every time.
/api/search returns a correct result, but only after three seconds.
Probe each path twenty times:
$ ./blindspots/run.sh
service on port 46095
path answered median within SLA
/health 20/20 0 ms 20/20
/api/report 0/20 0 ms 0/20
/api/search 20/20 3000 ms 0/20All three rows refer to the same running process.
If the status page uses only the first row, the service looks completely healthy. To a user, the last two mean that the product does not work.
What /health Actually Proves
A health check is usually kept simple on purpose.
It should respond quickly, create almost no load, and avoid failing because of every temporary problem in a secondary dependency. Such an endpoint is useful to an orchestrator: it can determine that the process is not stuck, is listening on a port, and can handle a minimal HTTP request.
But that is not yet a user workflow check.
In our service, /health responded twenty times out of twenty. It really was one hundred percent available.
At the same time, /api/report did not succeed once.
If users come to the product to build reports, user-facing availability is zero even though the process and health check remain fully operational.
Monitoring did not receive a false answer. It asked a narrow technical question and published the result as the state of the entire product.
A green /health proves only that /health is green.
Why 200 OK Is Not Enough Either
The situation with /api/search is more subtle.
All twenty requests completed with 200 OK. If monitoring counts only response codes, availability is one hundred percent.
But the median latency was three seconds. Not one request met the SLA.
If the product promises to show search results within half a second, the feature was unavailable from the user's perspective throughout the test.
A user operation succeeds only when the correct response arrives on time. A
200code after the allowed deadline can be a technical success and a product failure at the same time.
The same endpoint now has two honest results:
HTTP 200 received: 20 of 20
within SLA: 0 of 20The first describes the completion of an HTTP request. The second describes fulfillment of the promise made to the user.
Publishing the first as search availability is valid only if response time does not matter to the product.
One Service Can Have Several Uptimes
It is now difficult to speak of "service uptime" as one natural number.
We can check whether the process exists. We can establish a TCP connection. We can call /health. We can complete a real user workflow. We can count only correct responses as successful, or only responses that arrived before a deadline.
Each check produces its own result:
process accepts connections: 100 %
health endpoint responds: 100 %
report API succeeds: 0 %
search returns HTTP 200: 100 %
search completes within SLA: 0 %All these numbers are simultaneously correct.
Infrastructure needs to know whether the process is alive. A load balancer needs to know whether it can route traffic to that process. The product team needs to know whether the user operation works. The user needs to know whether the correct response arrived quickly enough.
The error begins when one technical signal is called the overall availability of the service.
A Percentage Does Not Tell the Incident Story
Even a correctly chosen SLI does not show how downtime is distributed.
One thirty-minute incident and ninety twenty-second failures produce the same total downtime. But for users, the on-call team, and the architecture, those are completely different months.
A long incident will almost certainly overlap external probes. Short recurring failures can repeatedly fit between them, affect individual users, and never combine into an obvious outage.
That is why a percentage needs an event history beside it: how many incidents occurred, how long each one lasted, which features failed, and how many requests were affected.
In our experiment, two statements describe the same timeline:
uptime: 99.92 %and:
26 incidents occurred,
24 of them lasted less than a minuteThe first is useful for an error-budget report. The second reveals a recurring problem worth investigating.
A percentage aggregates time. It does not replace incident history.
A Check Should Begin with the User Promise
First, define what the product promises the user.
If the user needs to build a report, a synthetic check should traverse the critical report-building path, including important dependencies.
If search must respond within 500 ms, a response after that deadline should count as a failure regardless of its HTTP code.
If short outages matter, checks must be frequent enough, or the system needs other data sources: server-side metrics for all requests, error logs, and client telemetry.
A simple /health endpoint does not become useless. It still answers whether the process is alive.
A sparse external probe is useful too. It looks at the system from the outside and can detect problems invisible to internal telemetry.
But neither signal should be presented as the complete picture.
Good monitoring combines several limited observations and understands the blind spot of each one.
Monitoring Did Not Lie
Return to the green status page.
The five-minute probes calculated monthly uptime almost perfectly and missed twenty-two of twenty-six incidents.
/health reported one hundred percent availability while /api/report did not work at all.
/api/search returned 200 OK twenty times but never met the SLA.
None of these results was calculated incorrectly.
The error was in interpretation. Success among sparse probes became a history of every incident. Process health became product health. A successful HTTP code became a successful user experience.
Monitoring sees only the moments and paths we choose to check.
Every green health check therefore needs two questions beside it.
What exactly does it confirm?
And what can be broken while it continues to return 200 OK?
Appendix: Full Source Files
timeline/src/lib.rs — 59 lines
//! Scene 01: the ground truth a monitoring system never sees.
//!
//! A deterministic month of a service: 30 days at one-second
//! resolution with a fixed set of outages — many short ones, a couple
//! of long ones. This timeline is the reality; every later scene
//! samples it and reports what it believes.
/// One outage: when it began and how long it lasted, in seconds.
#[derive(Clone, Copy, Debug)]
pub struct Outage {
pub start: u64,
pub seconds: u64,
}
pub const MONTH: u64 = 30 * 24 * 3600;
/// The month's incidents. Durations are what real services actually
/// look like: a long tail of brief blips, a few substantial outages.
pub fn outages() -> Vec<Outage> {
let mut outages = Vec::new();
// 24 brief blips, one every ~30 hours, 8 to 55 seconds each.
let mut state: u64 = 0x5EED_1313;
for i in 0..24 {
state ^= state << 13;
state ^= state >> 7;
state ^= state << 17;
let start = 3600 + i * 107_000 + state % 5_000;
let seconds = 8 + state % 48;
outages.push(Outage { start, seconds });
}
// Two outages nobody could miss.
outages.push(Outage {
start: 11 * 24 * 3600 + 4200,
seconds: 17 * 60,
});
outages.push(Outage {
start: 23 * 24 * 3600 + 61_000,
seconds: 6 * 60,
});
outages.sort_by_key(|outage| outage.start);
outages
}
/// Is the service up at this second?
pub fn up_at(outages: &[Outage], second: u64) -> bool {
!outages
.iter()
.any(|outage| second >= outage.start && second < outage.start + outage.seconds)
}
/// Total downtime in seconds.
pub fn downtime(outages: &[Outage]) -> u64 {
outages.iter().map(|outage| outage.seconds).sum()
}
/// Availability as a percentage of the month.
pub fn availability(outages: &[Outage]) -> f64 {
(MONTH - downtime(outages)) as f64 / MONTH as f64 * 100.0
}timeline/src/main.rs — 29 lines
//! Prints the ground truth: how many incidents there were, how long
//! they lasted, and the real availability figure for the month.
use anyhow::Result;
use timeline::{MONTH, availability, downtime, outages};
fn main() -> Result<()> {
let outages = outages();
let short = outages.iter().filter(|o| o.seconds < 60).count();
let long = outages.len() - short;
println!("month: {} days ({MONTH} seconds)", MONTH / 86_400);
println!(
"incidents: {} total — {short} under a minute, {long} longer",
outages.len()
);
println!(
"shortest: {} s, longest: {} s",
outages.iter().map(|o| o.seconds).min().unwrap_or(0),
outages.iter().map(|o| o.seconds).max().unwrap_or(0)
);
println!(
"downtime: {} s ({:.1} min)",
downtime(&outages),
downtime(&outages) as f64 / 60.0
);
println!("TRUE uptime: {:.4} %", availability(&outages));
Ok(())
}sampling/src/main.rs — 68 lines
//! Scene 02: the same month, seen through probes.
//!
//! A prober wakes every N seconds, asks "are you up?", and computes
//! availability as successful probes over total probes. That number is
//! what the status page shows. Here it is placed next to the truth for
//! several intervals — including the one most services actually use.
use anyhow::Result;
use timeline::{MONTH, Outage, availability, downtime, outages, up_at};
struct Report {
interval: u64,
probes: u64,
failed: u64,
seen_incidents: usize,
}
fn probe(outages: &[Outage], interval: u64) -> Report {
let mut probes = 0;
let mut failed = 0;
let mut seen = vec![false; outages.len()];
let mut second = 0;
while second < MONTH {
probes += 1;
if !up_at(outages, second) {
failed += 1;
for (index, outage) in outages.iter().enumerate() {
if second >= outage.start && second < outage.start + outage.seconds {
seen[index] = true;
}
}
}
second += interval;
}
Report {
interval,
probes,
failed,
seen_incidents: seen.iter().filter(|hit| **hit).count(),
}
}
fn main() -> Result<()> {
let outages = outages();
let truth = availability(&outages);
println!(
"TRUE uptime: {truth:.4} % ({} s of downtime, {} incidents)",
downtime(&outages),
outages.len()
);
println!();
println!("interval probes failed reported error incidents seen");
for interval in [300, 60, 30, 10, 1] {
let report = probe(&outages, interval);
let reported = (report.probes - report.failed) as f64 / report.probes as f64 * 100.0;
println!(
"{:>5} s {:>6} {:>6} {:>7.4} % {:+.4} pp {:>2} of {}",
report.interval,
report.probes,
report.failed,
reported,
reported - truth,
report.seen_incidents,
outages.len()
);
}
Ok(())
}blindspots/src/service.rs — 78 lines
//! Scene 03: a real service over loopback, healthy by every measure a
//! health check can take — and broken for the people using it.
//!
//! `/health` does what health endpoints usually do: it answers. It
//! touches no database, renders no page, and therefore cannot fail for
//! any reason that would affect a user. `/api/report` is the endpoint
//! users actually call; here it is broken. `/api/search` answers
//! correctly but takes seconds.
use anyhow::{Context, Result};
use clap::Parser;
use std::io::{BufRead, BufReader, Write};
use std::net::{TcpListener, TcpStream};
use std::thread::sleep;
use std::time::Duration;
/// The demo service: a healthy health check over a broken product.
#[derive(Parser)]
struct Args {
/// Address to bind; port 0 lets the OS choose
#[arg(long, default_value = "127.0.0.1:0")]
addr: String,
/// Milliseconds /api/search takes to answer correctly
#[arg(long, default_value_t = 3000)]
slow_ms: u64,
}
fn respond(stream: &mut TcpStream, status: &str, body: &str) -> Result<()> {
let response = format!(
"HTTP/1.1 {status}\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}",
body.len()
);
stream
.write_all(response.as_bytes())
.context("write response")?;
Ok(())
}
fn handle(mut stream: TcpStream, slow_ms: u64) -> Result<()> {
let mut request = String::new();
BufReader::new(stream.try_clone()?)
.read_line(&mut request)
.context("read request line")?;
let path = request.split_whitespace().nth(1).unwrap_or("/");
match path {
// The health check: no dependencies, so nothing can break it.
"/health" => respond(&mut stream, "200 OK", "ok"),
// The endpoint users call. Its dependency is down.
"/api/report" => respond(
&mut stream,
"500 Internal Server Error",
"report backend unavailable",
),
// Correct, and unusable: the answer arrives after seconds.
"/api/search" => {
sleep(Duration::from_millis(slow_ms));
respond(&mut stream, "200 OK", "results")
}
_ => respond(&mut stream, "404 Not Found", "no such path"),
}
}
fn main() -> Result<()> {
let args = Args::parse();
let listener = TcpListener::bind(&args.addr).context("bind")?;
println!("service on {}", listener.local_addr()?);
for stream in listener.incoming() {
let stream = stream.context("accept")?;
let slow_ms = args.slow_ms;
std::thread::spawn(move || {
if let Err(error) = handle(stream, slow_ms) {
eprintln!("service: {error}");
}
});
}
Ok(())
}blindspots/src/probe.rs — 84 lines
//! The prober, and the users it claims to represent.
//!
//! Three campaigns against the same live service: the health check
//! everyone configures, the endpoint users actually call, and the slow
//! one — judged both the way a monitor judges it (did it answer?) and
//! the way a user judges it (did it answer in time?).
use anyhow::{Context, Result};
use clap::Parser;
use std::io::{Read, Write};
use std::net::TcpStream;
use std::time::{Duration, Instant};
/// Probe a path N times and report what a monitor would conclude.
#[derive(Parser)]
struct Args {
/// Service port
#[arg(long)]
port: u16,
/// Number of requests per campaign
#[arg(long, default_value_t = 20)]
requests: u32,
/// The latency a user is willing to wait, in milliseconds
#[arg(long, default_value_t = 1000)]
sla_ms: u128,
}
struct Verdict {
ok: u32,
within_sla: u32,
median_ms: u128,
}
fn request(port: u16, path: &str) -> Result<(bool, u128)> {
let started = Instant::now();
let mut stream = TcpStream::connect(("127.0.0.1", port)).context("connect")?;
stream.set_read_timeout(Some(Duration::from_secs(10)))?;
write!(
stream,
"GET {path} HTTP/1.1\r\nHost: localhost\r\nConnection: close\r\n\r\n"
)?;
let mut response = String::new();
stream.read_to_string(&mut response).context("read")?;
let ok = response.starts_with("HTTP/1.1 200");
Ok((ok, started.elapsed().as_millis()))
}
fn campaign(port: u16, path: &str, requests: u32, sla_ms: u128) -> Result<Verdict> {
let mut ok = 0;
let mut within = 0;
let mut times = Vec::new();
for _ in 0..requests {
let (success, ms) = request(port, path)?;
if success {
ok += 1;
if ms <= sla_ms {
within += 1;
}
}
times.push(ms);
}
times.sort_unstable();
Ok(Verdict {
ok,
within_sla: within,
median_ms: times[times.len() / 2],
})
}
fn main() -> Result<()> {
let args = Args::parse();
println!(
"{:<14} {:>10} {:>12} {:>14}",
"path", "answered", "median", "within SLA"
);
for path in ["/health", "/api/report", "/api/search"] {
let verdict = campaign(args.port, path, args.requests, args.sla_ms)?;
println!(
"{:<14} {:>6}/{:<3} {:>9} ms {:>10}/{:<3}",
path, verdict.ok, args.requests, verdict.median_ms, verdict.within_sla, args.requests
);
}
Ok(())
}