Batch is the wrong clock
A warehouse job that lands every few hours is fine for finance close. It is the wrong clock for a plant, a device fleet, or an operations team that pages on symptoms.
When ingest falls behind, the dashboard still looks calm. The incident is already old.
Buffer at the edge, stream the rest
We put a buffer next to the source so a network blip does not drop the last hour. A streaming path carries events into a time-series or columnar store operators can query without waiting for the warehouse model to finish.
The warehouse can still exist. It just stops being the only place the truth shows up.
// Pattern: Edge-buffered streaming with queryable time-series store
interface TelemetryStreamTopology {
readonly edgeBuffer: 'Disk-backed ring buffer with automatic catch-up';
readonly streamBus: 'Kafka partitioned by deviceId / plantZone';
readonly queryStore: 'ClickHouse columnar table (sub-20ms query latency)';
readonly alertPipeline: 'P99 detection latency < 1.5s on symptom thresholds';
}Dashboards that match how the team works
Alerts should page on symptoms the on-call already understands, not on every noisy gauge. Dashboards follow the same runbook language: what is broken, what is delayed, what can wait until morning.
That is the operations and telemetry pattern — pipelines, a store you can query, and views that match the work — not a promised SLA or a fabricated before-and-after.
Have a similar architectural challenge?
Talk directly with a senior engineer about tradeoffs, cutovers, and sizing.