A mid-sized network produces telemetry at a scale that surprises people: a few hundred devices exporting flows, logs, and metrics can generate tens of millions of records per hour. Every design decision in a telemetry platform flows downstream from one question — where do you put all of that?

The shape of the data decides

Network telemetry has a very particular shape:

  • Write-heavy, append-only. Records arrive by the thousand per second and are never updated.
  • Timestamped and time-queried. Nearly every question is "…in this window, compared to that window."
  • Extremely repetitive. A million flow records from one site share a handful of source subnets, destination ports, and protocols.
  • Queried in aggregates. Operators ask for top talkers, sums, percentiles, and time buckets — not individual rows.

Row-oriented databases are built for the opposite workload: transactional updates and point lookups. Ask one for "total bytes by application across last week's flows" and it dutifully reads every column of every row off disk to sum one field.

What columns buy you

A columnar engine — we use ClickHouse under Ntrospect — stores each column contiguously. Three consequences matter enormously for telemetry:

Compression that borders on unfair. A column holding a million port numbers, most of them 443, compresses spectacularly. In practice we see 10–20x versus row storage, which converts directly into longer retention on the same disk budget.

Queries touch only what they need. Top-talkers-by-bytes reads the bytes, address, and timestamp columns and skips the rest of the record entirely. Combined with time-ordered layout, week-long aggregations over billions of rows come back in interactive time — fast enough to iterate a hypothesis during an incident instead of scheduling a report.

Retention becomes a policy, not a crisis. Cheap storage plus native TTLs means keeping ninety days of full-fidelity flow history is a configuration choice rather than a capacity project.

Why this matters to an operator

None of this is academic. The practical difference is behavioral: when the "what changed at Branch-West this week?" query takes two seconds instead of two minutes, engineers actually run it — repeatedly, conversationally, mid-incident. And a baseline-hungry anomaly engine gets to learn from months of history instead of days.

Fast queries aren't a luxury feature in observability. They're the difference between a tool people consult and a tool people trust.