Queueing at Meesho: Abstractions - Part 1
At Meesho, almost everything that happens after a tap goes through a queue. An order placed, a product viewed, a notification sent: each one is an event on Kafka, picked up by every service that needs to know. At peak, our clusters took in ~2.75 million records a second and delivered ~4.3 million a second to consumers.
Running Kafka at that scale taught us where the real pain lives: in the clients, not the brokers. Hundreds of services tuned their own settings, batch jobs decided cluster sizes, and every migration meant coordinated redeployments across teams. Meanwhile the platform underneath kept changing: over three years, our Kafka moved through various enterprise offerings, and today we run our own Open Source Kafka.
During the biggest sale events since, that fleet has absorbed many times that scale: a festival-day flood of orders, page views and notifications pouring through dozens of clusters at once. And the hundreds of services producing and consuming those messages rode through every one of those moves without changing a single line of code
What made that possible is MQ (Meesho Queue), our queuing platform. A service talks to an ID, not a cluster. MQ tunes every client with tested defaults, monitors every producer and consumer out of the box, and turns migrations into a zero-downtime config change, taking the pain away from every Kafka user in the org. This blog covers the client tuning that bought us headroom, why tuning one service at a time stopped scaling, how MQ turned a cluster into an ID, and the migration recipe that let us switch from managed Kafka offerings to our own open-source Kafka swiftly, while live traffic kept flowing.
Life on managed Kafka: when the defaults bit back
Our first Kafka home was a cloud managed one. It served us well and got us to scale. As traffic grew, the cracks showed up less in Kafka itself and more in how we were using it:
- Disk throttling. Most producers sent data uncompressed. At peaks of ~500 MBps, broker disks hit their throughput limits and slowed down for everyone.
- Untuned clients. Every team configured its own producers and consumers, usually by copying defaults, and nobody owned the overall picture.
- Batch readers sized the cluster. Spark ETL jobs made up ~90% of consumption and were read in large bursts, so clusters were sized for bursts rather than real-time traffic. On top of that, API based producers that wrote to Kafka synchronously felt every slow broker as a slow API.
- Scaling in one direction. Adding brokers and rebalancing partitions was manual, and disks could grow but not shrink. After any sale, storage sat only 20–30% used.
- Flying blind. There was no server-side view of which client produced or consumed what.
Each of these had a fix. The real question was where to apply it.
Fixing the clients first
The biggest wins came from the client side, not the brokers.
Compression. Switching producers to lz4 cut bytes on the wire by about half; moving to zstd halved them again. We ended with a compression factor of 0.236, so roughly a quarter of the original bytes now reach the broker, and disk throttling was off the table.
Batching and timeouts. We found linger.ms=0 and 10 KB buffers in production. For asynchronous flows we moved to linger.ms=25, a 100 KB batch.size and a 10 MB max.request.size, reset timeouts to sane values, upgraded clients to pick up the KIP-794 partitioner fix, and shared one producer per cluster instead of one per topic.
Spark: from batch to streaming. We moved high volume spark consumers from periodic batch reads to Spark Structured Streaming with 1-minute micro-batches. Same data, read steadily instead of all at once. Maximum egress peaks dropped by 75%.
And it worked. But only for the services we had tuned.
When good intentions didn't scale
Every fix above was a pull request into someone else's repository. With hundreds of services across tens of clusters, and a small platform team, that approach could never keep up. And the problems went beyond tuning. Every app was wired to its cluster: services hard-coded the cluster URL, topic name and partition count, so moving a topic or changing vendors meant redeployments across many teams, scheduled together. Monitoring was left to each team, so problems surfaced as incidents rather than alerts. And when a consumer fell behind, the instinct was to add partitions, even when the real bottleneck was the processing time inside the consumer so partition counts kept climbing, and so did infrastructure cost.
We needed one platform that made Kafka easy to use, tuned every client centrally, showed us what each client was doing, and let us move queues without asking anyone to deploy.
Enter MQ: turning a Kafka topic into an ID
MQ is a control plane plus a family of client libraries that sit between applications and Kafka. It started with a weekend kick-off 2 years ago; the self-serve UI went live that July. The design principle:
"Users are not expected to manually tune all the configurations of the Kafka Server and their clients. MQ owns the clusters and suggests best configurations for the clients."
From day one, MQ had four jobs: easy to use (a service talks to an MQ-ID, not a cluster, topic or partition count), optimised by default (every client gets tested, centrally managed configs), observable by default (every producer and consumer comes with metrics and alerts), and migrations without code (the platform can move a queue without the owning team shipping anything).
The model is three objects. An application is one deployable with its own auth token. An MQ-ID is a logical queue that maps to a (topic, cluster) pair only the platform controls. Producer and consumer configs are stored centrally, one per (application, MQ-ID) pair. Application code only knows the MQ-ID: a Java service produces with mqService.sendAndForget(mqId, payloads) and consumes with an @MqListener(mqId, clientId) annotation. Which topic, which cluster and how the client is tuned are all data in the control plane, served through admin APIs behind SSO and role-based access.
We ship clients for Java, Go, Spark, Flink and Python. They talk to Kafka directly, so MQ adds no extra hop on the data path. Clients fetch their config at startup and refresh it periodically, and that refresh is the trick behind the rest of this blog: change the mapping centrally, and running services pick it up without a deployment.
Easy, optimised and observable by default
An abstraction only helps if teams adopt it, so onboarding is as small as we could make it: add the library, a few flags and an auth token, once. From then on, the platform does the heavy lifting.
Easy to use. A self-serve UI lets teams onboard applications and MQs, create topics, send test messages, and edit consumer settings and offsets. Starting or stopping consumers, resetting offsets, changing alert thresholds and tuning Kafka configs all happen from the UI, with no deployment. MQ suggests sensible partition counts and client settings for a topic's traffic, and the MQ team manages clusters, credentials and capacity so service teams never do cluster homework.
Optimised by default. Every client ships with tuned settings: linger.ms around 20 ms for serving flows and 1–10 s for batch jobs, compression on, one producer per cluster. Default quotas and producer rate limits stop one noisy client from starving a cluster.
Monitored and alerted by default. Every new producer or consumer gets ~20 alerts automatically, from consumer lag to sending failures. Every service has its own Kafka credentials, so we can see, produce and consume traffic per service on the server side. Shared dashboards mean debugging starts from the same picture for every team. And changes go through the UI only, with breaking config changes needing a second approver.
The migration recipe: moving a live queue without stopping it
With every service speaking in MQ-IDs, moving a queue became a sequence of control-plane changes. We chose not to replicate data between clusters: consumer offsets differ from one cluster to another, so a mirrored consumer can't simply carry on where it stopped, and MirrorMaker 2 doesn't carry over replication factor or partitions added later. Instead of moving data, we move the producers and consumers, one MQ-ID at a time:
- Clone the topic on the target cluster with the required configurations.
- Duplicate and validate the consumers. Each gets a second config pointing at the target; after the next refresh it reads from both clusters, and we validate the new path end to end.
- Flip the producers through the control plane. After the refresh, all writes land on the target, and production on the old cluster drops to zero.
- Drain. Once the old topic has zero lag and no traffic for at least five minutes, the old consumers are switched off.
- Clean up after 24 hours: confirm no connections remain, then delete the old topic.
For topics that need ordering, consumption starts on the target only once source lag is zero; other topics use dual consumption, with zero downtime. Two safety nets back this up: an alert fires if the old topic receives any data within ten minutes of a migration, and revert APIs can flip one producer, or many, back in a single call. The recipe stayed the same across three migrations, but the effort kept shrinking: from one API call per step per service, to three calls per MQ-ID, to one click in the UI, and finally to full automation.
Three real world migrations, zero code changes
Once a service was on MQ, none of the three migrations below needed a code change from its owners.
1. Cloud managed Kafka → 3P managed Kafka (2023). The first real test came in July 2023 with a single topic; by August our first full cluster had moved. The 3P managed Kafka provider gave us auto-rebalancing, per-client auditing, usage-based pricing and no cross-AZ replication charges; that last one alone cut more than a third of the cos
2. Cloud 1 → Cloud 2 (November 2023 – February 2024). When Meesho changed clouds, MQ let us decouple two migrations that would otherwise have been tangled: service owners chose when to move their services, and the platform team moved Kafka on its own schedule, with cross-cloud connectivity between them. We built bulk tooling for every step of the recipe and a three-click flow in the UI. With them, a war room in January 2024 moved the bulk of the traffic between clouds in about a month, with zero downtime for the services on top. In total, ~2,000 topics across ~15 clusters and ~500 applications moved to Cloud 2. The control plane itself moved last, in February 2024.
3. 3P managed Kafka → self-managed Kafka (November 2025 – October 2026). The largest move by far. We added support for self-hosted Kafka to every MQ client without changing the control plane or client APIs, then moved cluster by cluster, a few MQ-IDs a day: ~5,000 topics across clusters, regions and accounts, completed in October 2026.
Each migration had its rough edges, and every lesson became part of the platform: more validation before each step, more checks after it, and better visibility into every producer and consumer while it moved.
Letting the machine migrate
By spring 2026, the recipe was routine. The bottleneck was people: engineers running each step and watching dashboards between them. So we automated it. A small auto-migration service takes each MQ-ID through a state machine: prep and validate, duplicate the consumers onto the target, migrate the producers, wait until the source topic has drained, then post-validate, including migrating alerts. Every transition is driven by an event on a Kafka topic, and each step waits for its exit condition before the next one starts. By default it takes the ordering-safe path, so keyed topics need no special handling.
It went live earlier this year. It moved 96 topics in its first three days and ~300 topics in a week, work that used to take about a month by hand. One team's 241 topics moved in four days.
Impact at a glance
Key takeaways
- Fix clients before brokers. Compression, batching and steady reads bought more headroom than any broker change.
- Abstract the address, not the protocol. Applications keep a native Kafka client and talk in MQ-IDs. The cluster is just data.
- Make the right thing the default. A default reaches every service; a best-practice doc reaches only the people who read it.
- Move clients, not data. Duplicate consumers, a producer flip and a drain are simpler than replication, and keep offsets honest.
- Automate a recipe once it's boring. We ran the playbook by hand across two migrations before turning it into a state machine, so the machine encodes lessons, not guesses.
What's next
MQ made the Kafka underneath interchangeable. That raised an obvious question: if switching providers is just a config change, which Kafka do we actually want to run? Managed Kafka got us to scale; at our volumes, cost, control over upgrades during sale events and per-client visibility pushed us to run our own. In Part 2, we'll cover why we built our own Kafka, what our POCs showed, and what broke while we stabilised it.
Team: Tushar Jaiswal (Senior Software Engineer), Shivam Gupta (Senior Software Engineer), Ujjwal Kumar Pandey (Software Engineer II), Harshit Karwal (Software Engineer II), Harshdeep Shrivastava (Software Engineer I) & Ujjwal Verma (Software Engineer I)
