Kafka Primer
Distributed commit log for high-throughput event streaming. Understand brokers, topics, partitions, and consumer groups before wiring Meridian pipelines.
Core Concepts
Broker
A single Kafka server. Clusters run 3+ for quorum.
Topic
Named log stream. Append-only, partitioned.
Partition
Ordered, immutable sequence of records within a topic.
Offset
Monotonic position of a record in a partition.
Producers & Consumers
Producer
Publishes records to a topic. Can specify partition key for deterministic routing. Acknowledgment modes: none, leader, all.
Consumer Group
Horizontally scales consumption. Each partition is assigned to exactly one member. Group coordinator handles rebalancing.
Retention & Compaction
Kafka retains records by time or size. Log compaction keeps the latest value per key — ideal for changelog-style topics. Meridian uses compacted topics for license state snapshots.