Queueing at Meesho: Engines - Part 1
On one of our highest-traffic days this year, Kafka at Meesho moved an enormous volume of data: trillions of messages flowed through the platform in a single day, with tens of millions of messages arriving every second at the busiest moment, and far more data served out than taken in.
The platform was held. Every partition stayed online and above min ISR all day, billions of produce requests completed without a single failure, produce latency barely moved from a normal day.
A significant portion of this ran on open-source Apache Kafka: 33 clusters, 594 brokers and about 108,000 partitions, operated by our own team. For a long time an enterprise Kafka service carried this load for us. So we asked a simple question: could we run open-source Kafka at the same scale, for far less money, without giving up latency?
The answer turned out to be yes, but it took more than one step. Every fix exposed the next problem, and each one taught us something about how Kafka really works. This post is that journey, told through the following chapters; each learning brought us closer to our goal. Join us.
Chapter 1: First things first, deploy a basic setup
Every journey starts with a first step, and ours was getting a cluster running. We wanted standard, proven building blocks, so we picked Strimzi, a widely used way to run Kafka on Kubernetes.
That left a bigger question: what runs the control plane? For years the answer was ZooKeeper, but that means operating a second distributed system, with slower controller failover as partitions grow and metadata that can drift between components. So we did what Kafka itself did and chose KRaft. The controllers form a Raft quorum and keep cluster metadata in a log of their own, with no external system involved. That gave us one system instead of two, quick controller failover, and brokers that pull their metadata and start fast.
With the foundation in place, it was time to put a real load on it.
Chapter 2: get the most out of the infra
We pointed our workload at the new cluster: three brokers and 45 MB/s of production. Kafka request latency came back at around 100 ms. That was far too slow, and so the investigation began.
Where did the memory go?
Digging in, we found the brokers reading from disk far more than they should. That was odd, because Kafka is powered by the page cache: recent data should be served from memory, not disk. Was the page cache the problem? It was. The JVM heap had been set to 26 GB on a 32 GB machine, leaving almost nothing for the page cache.
How much heap does a Kafka broker need?
Kafka rarely needs more than 5 to 6 GB of heap, even at 150,000 messages and 200 megabits per second. So we cut the heap to 5 GB and handed the rest back to the OS. Request latency dropped from about 100 ms to about 19 ms.
Should I use G1GC or ZGC for Kafka?
While we were there, we also chose G1GC, which did better than ZGC on CPU for our workloads. Because Kafka keeps its heap small, around 5 GB of short-lived buffers, and G1GC is built for exactly this. It keeps pauses within a few milliseconds with almost no tuning, and it’s the collector Kafka ships with by default. ZGC shines on heaps of hundreds of gigabytes. On ours, it used more CPU for its concurrent collection and gave us longer GC waits, so G1GC won.

Can Kafka really run on HDDs?
The other half of Kafka is storage. The SSDs we were using were expensive, and HDDs were far cheaper but slow. Then again, Kafka writes data sequentially and was built to run well on spinning disks, so we gave it a shot. The first result hurt: app latency shot up to 5 seconds at 45 MB/s. Exploring further, we found the filesystem matters a lot. For our workload, XFS handled Kafka's batched writes better than ext4 with minimal tuning. We switched, and disk latency fell from about 30 ms to about 20 ms.
That was better, but still too high. Could we cut the I/O itself? Linux updates a file's access time (atime) on every read, and Kafka never uses it. So we mounted the disks with noatime. Disk latency dropped to about 10 ms, and write operations fell from 4.8k/s to 3.5k/s.


What it bought us.
With these changes, a cluster with 3 brokers 4 cores 32 GB each handled 60 MB/s of production and 180 MB/s of consumption(without disc reads), matching our earlier benchmark at half the instance cost and 60% less disk cost. Things were looking great, until our consumers started reading old data.
Up to now we tuned the machine around Kafka. The next problems lived inside the broker, and we couldn't fix what we didn't understand. So let's follow a single produce request with acks=all from arrival to acknowledgement.

Arrival. The request lands in the socket receive buffer, sized by socket.receive.buffer.bytes, with socket.request.max.bytes capping how large a single request can be.
The network threads. A network thread from a small shared pool picks the request up, parses it, and drops it into the request queue. That same thread looks after the connection for the request's whole life, including sending the response at the end. The pool size is num.network.threads, and the default is just 3. Keep that number in mind.
The request queue and I/O threads. Requests wait for an I/O thread (queued.max.requests caps the queue). An I/O thread takes the request off the queue, validates the data with a CRC checksum, then writes it. The pool size is num.io.threads, default 8.
Page cache, then disk. Here's the part that makes Kafka fast. The write goes to the OS page cache, not straight to disk, and the OS flushes it later. Kafka relies on replication, not fsync, for durability. This is why our memory work in Chapter 2 paid off so well.
Purgatory. Data in the page cache isn't safe until other brokers have copied it. With acks=all, the request is parked in purgatory, a timer-based waiting room, until follower replicas fetch the new data. Followers ask the leader for data on a schedule set by replica.fetch.wait.max.ms (500 ms by default), with num.replica.fetchers threads doing the fetching. Consumer fetches wait here too. Remember purgatory; it comes back to haunt us in Chapter 5 in Part 2 of the blog.
The way back. Once the acks requirement is met, the broker builds a response, and a network thread writes it into the socket send buffer for the producer to read.
Two things in this path matter for the rest of our story. Network threads are a small pool shared by every client, and every request passes through them twice, once on the way in and once on the way out. Now let's see what happens when one of them gets stuck.
Chapter 3: tune the broker for how it really works
Why do producers pay for slow consumers?
Everything was calm as long as consumers kept up. Then one fell behind, or a team kicked off a backfill, and it read old data from disk. The moment that happened, producer latency jumped and production dipped. Why should a producer suffer because some consumer is behind?
Our first guess was memory [not enough page cache]. We moved to 8-core, 64 GB brokers, and the cache now held about 55 minutes of data instead of 30, but anything older still hurt producers. We tried the usual knobs next: bigger socket buffers, larger replica fetches, different flush intervals. None of them made a difference. The thread metrics didn't help either; network and I/O threads showed about 99% idle. It took us a while to realise those were averages, and averages were hiding the spikes.
So we went back to the request path from the interlude. When a consumer reads data that isn't in the page cache, Kafka's zero-copy sendfile call must first load it from disk, and it does so synchronously. The network thread just waits. Producers share that same small pool of network threads, so they end up queued behind a slow disk read they have nothing to do with.
How many network and I/O threads should a Kafka broker have?
Once we saw that, the fix was clear: give the broker enough threads that one slow read can't block everyone. We swept the settings while pushing 60 MB/s in and 180 MB/s out, with every read coming from disk. There was a clear sweet spot, and going past it made things worse again.

Wait, shouldn’t threads match cores?
The usual rule says to run about one thread per CPU core. On our 8-core brokers, 30 network threads and 60 I/O threads looked like far too many. But that rule is for threads that spend their time computing. Ours spent their time waiting. On a cold read, sendfile blocks the network thread until the HDD returns the data, which takes ~10ms. A thread that’s waiting on disk doesn’t use a core, it just sits there. With Kafka’s default 3 network threads, three slow consumer reads were enough to block every thread, and producers queued behind them while the CPU sat mostly idle. The numbers show it: at the default we used only 2.39 cores, yet latency was about 90 ms. So the question isn’t how many cores you have. It’s how many threads can be stuck waiting on disk at once, and how many you need left free for everyone else. With 30 network threads, a handful of slow reads no longer block the pool, and producers always find a free thread. Past 30/60, extra threads stopped paying off. More threads meant more context switching and more parallel random reads on a spinning disk, so each read got slower, and latency crept back up to 11 ms and then 18 ms.
What about restarts?
Threads mattered in one more place. When a broker starts after an unclean shutdown, it must check and truncate every partition's log segments, by default with a single thread per log directory. Raising num.recovery.threads.per.data.dir lets recovery run in parallel, so a crashed broker rejoins the cluster in minutes rather than hours.
At 30 network threads and 60 I/O threads, cold reads stopped hurting producers. The clusters were fast and stable. Now there were a lot of them to run.
Where we landed
The scoreboard. Part 1 started with a basic cluster and a simple goal: get the most out of every machine before reaching for a bigger one. Here’s what each step bought us.
Put together, a cluster sustained 60 MB/s of production and 180 MB/s of consumption on cheap HDDs. That matched the benchmark we started from, at half the instance cost and 60% less disk cost. Producers stopped paying for slow consumers.
What we learned.
Start with boring, standard building blocks, and let Kafka’s own choices guide yours. KRaft over ZooKeeper and G1GC are both what Kafka itself picked. Give memory to the page cache, not the heap. Make cheap disks fast before you pay for expensive ones, because the filesystem and mount options matter more than they look. Understand how the broker really works before you tune it. Cold reads are a thread problem as much as a disk problem. The “one thread per core” rule is for threads that compute, not threads that wait. And always distrust an average.
What’s next?
By the end of Part 1 now we know how to set up a MVP Kafka Cluster. We had fast, stable clusters, and a growing number of them. A single fast cluster is an engineering win; dozens of them carrying production traffic are an operations problem. In Part 2, we’ll show how we made our clusters production-ready:
- scaling and rebalancing without pain
- protecting leadership and data when brokers fail
- the 500 ms “phantom” commit latency that taught us to tune for each workload
- cutting the storage bill with tiered storage
- watching the watcher: observability the soul of the platform
Stay tuned.
The team
This journey was a team effort. Thanks to everyone who built, tuned and ran these clusters with us:
- Avinash Devadhar, Senior Software Engineer
- Shivam Gupta, Senior Software Engineer
- Ujjwal Kumar Pandey, Software Engineer II
- Harshit Karwal, Software Engineer II
- Harshdeep Shrivastava, Software Engineer I
- Ujjwal Verma, Software Engineer I
References. Kafka: The Definitive Guide (2nd edition, O’Reilly, 2021) · Apache Kafka documentation · KIP-500 and KIP-631 · Strimzi documentation · AutoMQ on disk reads and producers.

