Kafka is a distributed, fault-tolerant, high-throughput event streaming platform. Think of it as a durable commit log where producers write events and consumers read them — at their own pace, without losing data.
KAFKA CLUSTER
┌────────┐ ┌────────────────────────────────────┐ ┌────────┐
│Producer│────────►│ │────────►│Consumer│
│ A │ write │ Broker 1 Broker 2 Broker 3 │ read │ Group 1│
└────────┘ │ ┌──────┐ ┌──────┐ ┌──────┐ │ └────────┘
┌────────┐ │ │ P0 │ │ P1 │ │ P2 │ │ ┌────────┐
│Producer│────────►│ │(Lead)│ │(Lead)│ │(Lead)│ │────────►│Consumer│
│ B │ │ │ P1 │ │ P2 │ │ P0 │ │ │ Group 2│
└────────┘ │ │(Rep) │ │(Rep) │ │(Rep) │ │ └────────┘
│ └──────┘ └──────┘ └──────┘ │
│ │
│ ZooKeeper / KRaft │
│ (metadata & coordination) │
└────────────────────────────────────┘
P0, P1, P2 = Partitions of a Topic
(Lead) = Leader partition (Rep) = Replica/Follower
Publishes messages (events) to a Kafka topic. Decides which partition to send to — via round-robin, key-based hashing, or custom partitioner. Supports batching and compression.
Reads messages from topic partitions. Tracks its position using offsets. Part of a Consumer Group for parallel processing. Can auto-commit or manually commit offsets.
A single Kafka server that stores data and serves clients. A cluster has multiple brokers for fault-tolerance. Each broker handles read/write for the partitions it leads.
A logical category/feed name to which messages are published. Think of it as a "table" in a database. Messages in a topic are immutable and append-only.
A topic is split into partitions for parallelism. Each partition is an ordered, immutable log. Messages within a partition have a sequential offset. Partitions are distributed across brokers.
A set of consumers that cooperatively read from a topic. Each partition is assigned to exactly one consumer in the group. Enables horizontal scaling of consumption.
A unique sequential ID for each message within a partition. Consumers track their offset to know where they left off. Stored in an internal topic: __consumer_offsets.
Manages cluster metadata: broker registration, topic configs, partition leaders. KRaft (Kafka Raft) is the new built-in replacement for ZooKeeper, removing the external dependency.
Key + Value are serialized (JSON, Avro, Protobuf). If a key is present, it's hashed to determine the target partition. No key → round-robin across partitions.
Producer's metadata cache knows which broker leads each partition. Message is sent directly to that broker. Batching and compression happen here for throughput.
The message is appended to the partition's on-disk log. It gets assigned a unique offset (e.g., offset 42). The write is sequential I/O — this is why Kafka is fast.
Follower brokers fetch the new message and write it to their local log. Once enough replicas confirm (based on acks config), the message is "committed."
acks=0: fire-and-forget (no ack). acks=1: leader confirms. acks=all: all ISR replicas confirm. Tradeoff: durability vs latency.
Consumer sends a fetch request with its current offset. Broker returns messages from that offset onwards. Consumer processes them and commits the new offset.
┌─────────────── PARTITION 0 ────────────────────────────────┐ │ offset: 0 │ 1 │ 2 │ 3 │ 4 │ 5 │ 6 │ 7 │ 8 │ ... │ │ │ │ │ │ │ │ │ │ │ │ │ ◄─── old messages ──── │ ──── new messages ───► │ │ │ │ │ committed │ │ offset │ │ (Consumer │ │ position) │ └────────────────────────────────────────────────────────────┘ CONSUMER GROUP ASSIGNMENT: ┌─────────────────────────────────────────────┐ │ Topic: "orders" (3 partitions) │ │ │ │ Consumer Group "payment-service": │ │ Consumer A ← reads P0 │ │ Consumer B ← reads P1 │ │ Consumer C ← reads P2 │ │ │ │ Consumer Group "analytics-service": │ │ Consumer X ← reads P0, P1 │ │ Consumer Y ← reads P2 │ │ │ │ ⚠ If consumers > partitions: │ │ Extra consumers sit IDLE │ └─────────────────────────────────────────────┘ ISR (In-Sync Replicas): ┌─────────────────────────────────────────────┐ │ Broker 1 (Leader) → offset 100 ✓ ISR │ │ Broker 2 (Follower)→ offset 99 ✓ ISR │ │ Broker 3 (Follower)→ offset 85 ✗ OUT │ │ │ │ replica.lag.max.messages controls │ │ when a follower is kicked from ISR │ └─────────────────────────────────────────────┘
| Config | What it controls | Trade-off |
|---|---|---|
acks | How many replicas must confirm a write | Durability vs latency |
replication.factor | Number of copies of each partition | Fault-tolerance vs storage |
min.insync.replicas | Minimum ISR count to accept a write | Availability vs consistency |
retention.ms | How long messages are kept | Storage vs replay ability |
num.partitions | Parallelism of a topic | Throughput vs ordering |
enable.auto.commit | Auto vs manual offset commits | Convenience vs exactly-once |
max.poll.records | Max messages per consumer poll() | Throughput vs processing time |
Follow this order — from most common to least common causes. In an interview, walk through this methodically.
Don't assume the producer is working. Check producer logs for errors. Verify with the console consumer or by checking topic offsets.
kafka-console-consumer --bootstrap-server localhost:9092 --topic orders --from-beginning kafka-run-class kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic ordersA very common issue — the consumer is subscribed to a different group ID, so it's reading from a different set of offsets, or it was already consumed and committed those offsets before.
kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group payment-serviceIf LAG = 0 but consumer sees no messages, it means all messages were already consumed (offset is at the end). If LAG > 0 but consumer isn't reading, the consumer is likely stuck or disconnected.
kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group payment-service # Output columns: TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID # If LAG keeps growing → consumer is slow or dead # If CURRENT-OFFSET = LOG-END-OFFSET → nothing new to readIs the consumer assigned to partitions? If the group has more consumers than partitions, some consumers will be idle. Also check if the consumer is subscribed to the correct topic name (typo!).
# In consumer logs, look for: # "Assigned partitions: [orders-0, orders-1]" # If "Assigned partitions: []" → consumer is idle or rebalancingIf a consumer group is NEW (no committed offsets) and auto.offset.reset=latest, it will only see messages published AFTER it started. Old messages are invisible. Set to earliest to consume from the beginning.
If processing takes too long, the consumer fails to call poll() within max.poll.interval.ms, gets kicked from the group, rejoins, gets reassigned → infinite loop. Look for "Member X has been removed" in logs.
Producer writes Avro, consumer expects JSON → deserialization fails silently or throws. Consumer might be crashing on every message without making progress.
# Check consumer logs for: # "SerializationException" or "DeserializationException" # Ensure producer serializer matches consumer deserializerConsumer might not have permission to read from the topic (ACL issue), or there's a firewall between consumer and broker. Also check if SSL/SASL configs match.
# Check broker logs for: # "Authorization failed" or "Authentication failed" # kafka-acls --list --topic orders --bootstrap-server localhost:9092If retention.ms is short and consumer was offline for too long, messages were deleted before the consumer could read them. The consumer's committed offset now points to a deleted segment.
| Symptom | Likely Cause | Fix |
|---|---|---|
| LAG = 0, no messages | Already consumed or auto.offset.reset=latest | Reset offsets or use earliest |
| LAG growing, consumer active | Slow processing or rebalancing | Increase poll interval, reduce batch size |
| LAG growing, no active consumer | Consumer crashed or disconnected | Check consumer logs, restart |
| Consumer assigned 0 partitions | More consumers than partitions, or wrong topic | Scale down consumers or add partitions |
| Constant rebalancing | Processing exceeds max.poll.interval.ms | Increase timeout or optimize processing |
| Deserialization errors in logs | Schema mismatch between producer/consumer | Align serializers, use Schema Registry |
| Consumer starts but reads nothing | ACL denied, wrong bootstrap server, or wrong group | Check permissions, config, network |
THE PROBLEM SCENARIO:
═══════════════════════════════════════════════════════════════
Client Payment Service Database
│ │ │
│──── POST /pay $100 ────────►│ │
│ │── BEGIN TX ──────────────►│
│ │ UPDATE balance -= 100 │
│ │── COMMIT ────────────────►│
│ │ │
│ ✗ TIMEOUT (network) │◄── 200 OK ──────────────│
│ (client never got │ (but response lost │
│ the response) │ in network) │
│ │ │
│──── POST /pay $100 ────────►│ │
│ (RETRY! same request) │── BEGIN TX ──────────────►│
│ │ UPDATE balance -= 100 │
│ │── COMMIT ────────────────►│ ⚠ DOUBLE
│◄── 200 OK ──────────────────│◄── OK ───────────────────│ DEDUCTION!
│ │ │
Balance was 500.
After 1st request: 400 ✓
After retry: 300 ✗ ← User lost $100 extra!
The core idea: every payment request carries a unique idempotency_key. Before processing, we check if we've already processed this key. If yes, return the cached response. If no, process and store the result.
THE SOLUTION:
═══════════════════════════════════════════════════════════════
Client Payment Service Database
│ │ │
│ POST /pay $100 │ │
│ Idempotency-Key: "abc-123" │ │
│─────────────────────────────►│ │
│ │── Check: "abc-123" │
│ │ exists in │
│ │ idempotency_store? ───►│
│ │◄── NO ──────────────────│
│ │ │
│ │── BEGIN TX ──────────────►│
│ │ 1. INSERT into │
│ │ idempotency_store │
│ │ (key, status,result) │
│ │ 2. UPDATE balance -=100 │
│ │── COMMIT ────────────────►│
│ │ │
│ ✗ TIMEOUT │◄── OK ──────────────────│
│ │ │
│ POST /pay $100 │ │
│ Idempotency-Key: "abc-123" │ │
│─────────────────────────────►│ │
│ │── Check: "abc-123" │
│ │ exists? ──────────────►│
│ │◄── YES, return cached ──│
│ │ │
│◄── 200 OK (cached) ─────────│ ✅ NO DOUBLE │
│ (same response as 1st) │ DEDUCTION! │
Balance: 400 ✓ (deducted only once)
-- Stores the result of each unique payment request CREATE TABLE idempotency_store ( idempotency_key VARCHAR(255) PRIMARY KEY, status VARCHAR(20), -- PROCESSING, SUCCESS, FAILED response_code INT, response_body TEXT, -- cached JSON response created_at TIMESTAMP DEFAULT NOW(), expires_at TIMESTAMP -- TTL: auto-cleanup old keys ); -- UNIQUE constraint on key ensures no duplicates even -- under race conditions (DB-level safety net)PaymentService.java — Idempotent Handler
public class PaymentService { @Transactional public PaymentResponse processPayment(PaymentRequest req) { String key = req.getIdempotencyKey(); // ── Step 1: Check if already processed ── Optional<IdempotencyRecord> existing = idempotencyRepo.findByKey(key); if (existing.isPresent()) { IdempotencyRecord record = existing.get(); if (record.getStatus() == Status.PROCESSING) { // Another thread is processing this right now throw new ConflictException( "Payment is being processed. Retry later."); } // Already completed — return cached response return record.getCachedResponse(); } // ── Step 2: Claim this key (with PROCESSING status) ── try { idempotencyRepo.insert(new IdempotencyRecord( key, Status.PROCESSING, null )); } catch (DuplicateKeyException e) { // Race condition: another thread inserted first throw new ConflictException("Duplicate request"); } // ── Step 3: Process the actual payment ── try { validateBalance(req.getUserId(), req.getAmount()); deductBalance(req.getUserId(), req.getAmount()); createTransaction(req); PaymentResponse response = new PaymentResponse( Status.SUCCESS, "Payment processed"); // ── Step 4: Save result & mark as SUCCESS ── idempotencyRepo.updateStatus( key, Status.SUCCESS, response); return response; } catch (Exception e) { // Payment failed — mark as FAILED so retries can reattempt idempotencyRepo.updateStatus( key, Status.FAILED, null); throw e; } } }
(user_id, request_id)version columnUPDATE accounts SET balance = balance - 100, version = version + 1 WHERE id = ? AND version = ?enable.idempotence=true in Kafka producer config
RACE CONDITION: Two retries arrive simultaneously
═══════════════════════════════════════════════════════════
Thread A Thread B
│ │
│── Check key "abc-123" ──► │── Check key "abc-123" ──►
│◄── NOT FOUND ─────────── │◄── NOT FOUND ───────────
│ │
│── INSERT key "abc-123" ──► │── INSERT key "abc-123" ──►
│◄── OK ────────────────── │◄── DuplicateKeyException! ✓
│ │ (DB UNIQUE constraint
│── Deduct balance ──────► │ saves us)
│◄── OK ────────────────── │
│ │── Return "conflict" ────►
│── Mark SUCCESS ────────► │
│◄── OK ────────────────── │
SOLUTION: The UNIQUE constraint on idempotency_key in the DB
acts as the final safety net, even if the application-level
check misses a race condition.
ADDITIONAL SAFEGUARDS:
┌─────────────────────────────────────────────────────────┐
│ 1. SELECT ... FOR UPDATE (pessimistic locking) │
│ Lock the row when checking, preventing concurrent │
│ reads until the transaction completes. │
│ │
│ 2. Redis SETNX (distributed lock for microservices) │
│ SETNX idempotency:abc-123 "processing" EX 30 │
│ Only one thread wins. Others get "already locked." │
│ │
│ 3. Database advisory locks │
│ pg_advisory_xact_lock(hash(idempotency_key)) │
│ Lightweight, auto-released on transaction end. │
└─────────────────────────────────────────────────────────┘
FULL ARCHITECTURE: Event-Driven Payment with Idempotency
═══════════════════════════════════════════════════════════
┌──────────┐ ┌────────────┐ ┌─────────────────┐ ┌──────────┐
│ Client │────►│ API │────►│ Kafka Topic │────►│ Payment │
│ (App) │ │ Gateway │ │ "payment- │ │ Consumer │
│ │ │ │ │ requests" │ │ Service │
└──────────┘ └────────────┘ └─────────────────┘ └────┬─────┘
│ │
│ Idempotency-Key │
│ in HTTP header │
│ ┌───────────────────┐ │
│ │ Database │◄───┘
│ │ ┌───────────────┐ │
│ │ │idempotency_ │ │
│ │ │store │ │
│ │ ├───────────────┤ │
│ │ │accounts │ │
│ │ ├───────────────┤ │
│ │ │transactions │ │
│ │ └───────────────┘ │
│ └───────────────────┘
│
│ THREE LAYERS OF IDEMPOTENCY:
│
│ Layer 1: API Gateway deduplicates by Idempotency-Key
│ (Redis cache, TTL = 24h)
│
│ Layer 2: Kafka producer idempotence (enable.idempotence=true)
│ Prevents duplicate messages in topic
│
│ Layer 3: Payment Consumer checks idempotency_store in DB
│ (UNIQUE constraint = final safety net)
Idempotency-Key header.expires_at = created_at + 24h. A background job or database TTL mechanism (Cassandra TTL, Redis EXPIRE, pg_cron) deletes expired records. Stripe keeps keys for 24 hours.enable.idempotence=true, the broker assigns each producer a PID (Producer ID) and tracks sequence numbers per partition. If a retry sends the same (PID, sequence) pair, the broker silently discards the duplicate. Consumer-side: Use isolation.level=read_committed with transactions to get exactly-once delivery. But this only covers the Kafka layer — your application still needs its own idempotency logic for database writes.┌──────────────────────────────────────────────────────────────────┐ │ INTERVIEW ANSWER STRUCTURE │ ├──────────────────────────────────────────────────────────────────┤ │ │ │ Q1: KAFKA BASICS │ │ → "Kafka is a distributed commit log with producers, consumers, │ │ brokers, topics, partitions, and consumer groups." │ │ → Draw the architecture diagram │ │ → Explain message flow (6 steps) │ │ → Mention: acks, ISR, replication, retention, offsets │ │ │ │ Q2: DEBUG CONSUMER NOT CONSUMING │ │ → "I'd approach this systematically..." │ │ → Verify producer → check consumer group → check lag │ │ → Check auto.offset.reset → rebalancing → deserialization │ │ → Show you know the CLI commands │ │ │ │ Q3: PAYMENT IDEMPOTENCY │ │ → "Every request gets a unique idempotency key" │ │ → Check-before-process + SAME transaction + UNIQUE constraint │ │ → Draw the before/after diagram │ │ → Mention: Redis cache layer, TTL cleanup, outbox pattern │ │ → Handle race conditions: SETNX, SELECT FOR UPDATE, DB unique │ │ │ └──────────────────────────────────────────────────────────────────┘