Every operations team we talk to has some version of the same folder: the email rule that quietly files hundreds of alerts a day somewhere nobody looks. Ask when it was created and you'll hear the same story — the alerts got noisy, people got tired, and the folder was easier than fixing the system.

The usual response is to blame discipline: "we should triage better." But alert fatigue is not a human failing. It is the predictable output of how most alerting systems are designed.

How threshold sprawl happens

Static thresholds are the original sin. CPU > 80%, link > 90%, temperature > 75°F. Each rule is reasonable in isolation. The problems compound:

  • Context-blindness. 90% utilization on a backup circuit during a failover drill is fine; 60% on a voice-carrying WAN link at 9am may be an emergency. A static number can't tell the difference.
  • The tuning treadmill. Every false page prompts a threshold bump, until the threshold is high enough that it only fires during a genuine disaster — at which point it fires after users noticed.
  • Multiplication. One faulty device breaches ten thresholds. One WAN brownout breaches a hundred. The operator gets a hundred pages describing one problem.

Principles that survive contact with production

Alert on symptoms, page on impact. A rising error rate is a symptom worth recording. A site's application traffic dropping off a cliff is impact worth a page. Most signals belong on a dashboard or in a daily digest, not a phone.

Severity must mean something. If "critical" alerts routinely sit unacknowledged overnight, they aren't critical — relabel them until the top tier is small enough that people trust it.

Compare to history, not to constants. "Interface errors are 40x their weekly baseline" survives environments that "errors > 100/min" cannot. This is where anomaly detection earns its keep — not as magic, but as thresholds that recompute themselves per-device, per-hour.

Correlate before you page. The hundred alerts from one brownout should reach a human as one incident with a hundred pieces of supporting evidence. Grouping by site, by device, and by time window is unglamorous and does more for on-call quality of life than any other feature.

The test

Here's the metric we hold ourselves to when designing alerting in Ntrospect: every page should change what the on-call person does next. If a page doesn't alter anyone's behavior, it should have been a log line. Audit last week's pages against that bar and you'll know exactly how much of your alerting is design debt.