Build the right mental model for Kafka as a durable distributed event log, streaming platform and decoupling layer.

Operating model

PRODUCER → TOPIC → PARTITIONED LOG → CONSUMER GROUPS / STREAM PROCESSORS → DOWNSTREAM SYSTEMS

Learning outcomes

• Explain Kafka beyond the phrase message queue
• Distinguish topics, partitions, records and offsets
• Explain why consumption does not delete records
• Choose Kafka only when durable event streaming fits the problem

Deep dive

Durable event log

Kafka appends immutable records to partition logs. Consumers track their own position, so several independent applications can replay or process the same retained history without copying the stored event for each subscriber.

Queue and pub/sub behaviors

Traditional consumer groups divide partitions among members, creating work-sharing behavior. Separate consumer groups independently read the same topic, creating publish/subscribe fan-out. Modern share groups add another queue-like cooperative model later in the course.

Durability is configured

Replication, acknowledgements, in-sync replicas, retention, disk health and recovery design determine real durability. Kafka does not turn a weak one-replica topic into a strongly durable system by itself.

When not to use Kafka

Kafka adds partitions, schemas, lag, rebalancing, brokers/controllers and lifecycle work. A small synchronous workflow or simple low-volume job queue may be clearer with a simpler tool.

Production checklist

• Name the event and its owner
• Define producer and consumer groups
• Define retention/replay need
• State ordering key
• State why Kafka is better than a simpler alternative

Commands / example

bin/kafka-topics.sh --bootstrap-server localhost:9092 --list
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic orders --partitions 3 --replication-factor 1
bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic orders
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders --from-beginning

Common failure modes

• Treating Kafka as a database replacement
• Assuming records disappear after consumption
• Using one giant topic for unrelated domains
• Choosing Kafka before defining the event contract