Steve HutchinsonBig Pines

Telemetry topic tier

The telemetry.* Kafka namespace. Carries raw infrastructure telemetry: telemetry.metrics.raw, telemetry.logs.raw, telemetry.traces.raw, and the normalized derivative telemetry.events.normalized. All topics use Diskless storage because scrape intervals are 15 seconds or longer.

The telemetry topic tier is the intake layer of the cognitive bus - where raw infrastructure signals enter the pipeline before being processed into operational primitives, patterns, and recommendations. The four topics serve distinct roles: telemetry.metrics.raw carries scraped metric samples (CPU, memory, latency histograms, Kafka consumer lag); telemetry.logs.raw carries structured log events; telemetry.traces.raw carries distributed trace data; and telemetry.events.normalized is the normalized derivative that the pattern detection worker consumes, where each raw event has been translated from vendor-specific format into the operational primitive vocabulary by the system mapping. All topics in the telemetry tier use Diskless (object-storage-backed) configuration because their scrape intervals - typically 15 seconds or longer - make the added object-storage latency invisible, while the long retention requirements (90+ days for trend analysis) make Diskless economically compelling compared to broker-local disks at that volume. The telemetry tier is also the highest-volume tier by raw event count, which is another reason to keep it on cheap, high-capacity object storage.

This site collects anonymous usage data to understand how people read and navigate the blog. Accepting enables persistent reader preferences across visits.