Kafka topic + partition design

Step 1 — Bound your throughput

Measure peak produce and consume rates in MB/s per partition. A single partition on decent hardware handles ~10-50 MB/s before latency spikes. Multiply by replication factor to account for broker write amplification.

Step 2 — Partition count formula

partitions = max(target_throughput / per_partition_limit, consumer_concurrency)

Partitions are the unit of parallelism. One consumer per partition within a group. Over-partitioning increases controller overhead and ISR replication chatter.

Step 3 — Key strategy

Choose a partition key that distributes evenly. Use a composite key or salted hash if your natural key is skewed. Null keys enable round-robin but break ordering guarantees.

Step 4 — Retention and compaction

Set retention.bytes and retention.ms based on your replay window. Enable log compaction only when you need the latest value per key — it trades CPU for storage.