Queueing at Meesho: Infrastructure - Part 1
A Kafka client never talks to “the cluster”. It talks to each broker, by name, at the address that broker gave it. That one fact shaped everything in this post.
Over the last few blogs, we told how Meesho moved its queueing onto open-source Kafka. In the Abstractions series, Meesho Queue turned every Kafka cluster into an ID, so hundreds of services could change clusters without changing code. In the Engines series, we made those clusters fast and then production-ready: Cruise Control took the pain out of scaling, configs and machines were tuned for each workload, tiered storage moved old data to object storage, and a small set of alerts watched the watcher. Our biggest day of the year was the real test of that work.
But every broker in those posts runs somewhere. It needs a machine, a disk, and an address that services in other parts of Meesho can reach. Today that layer carries 33 clusters and, at its peak, over two million client connections. On our biggest sale day it grew from 474 to 594 brokers, and the network did not change at all.
We'll tell the story of that layer in two parts. In this one, we give the brokers a home: the cluster they live in, the operator that runs them, and the machines and disks under them. Then we give every broker an address clients can reach, and take the load balancer out of the data path. Join us.
Chapter 1: A home for the brokers
Where should Kafka live?
Meesho's services run on managed Kubernetes clusters, one or more per business unit. The easy option was to put each team's Kafka next to its clients. We didn't. Kafka got a Kubernetes cluster of its own that hosts nothing else: the brokers and controllers, the proxies that let client traffic in, Cruise Control, and the metrics stack.
The reason was control. The Engines series was all about fitting the machine to Kafka, and that only works if the node pools, the upgrades and the capacity decisions are ours. Every Kafka cluster gets its own node pools, with a taint that nothing else tolerates, so its brokers only land on nodes bought for them.
One broker per node
Inside those pools, every broker gets a node of its own. A node can fail, and the cloud provider can restart or move a node for its own maintenance. When that happens, only one broker goes away. Its partitions have replicas on other brokers, on other nodes, so clients follow the new leaders and keep going. Clusters that span two zones are built to keep working through a zone problem.

Chapter 2: Strimzi, machines and disks
Who runs the brokers?
A Kafka cluster is a group of brokers and controllers. They must be configured alike, restarted one at a time and watched all the time. Kubernetes alone doesn't know any of that, so it needs an operator that does. We use Strimzi, and we use most of what it ships:
- The cluster operator runs all 33 clusters from one deployment, with three replicas and one elected leader. It turns each Kafka resource and its two node pools, one for the controllers and one for the brokers, into running pods. Every two minutes it compares each cluster with its resources and puts back anything that has drifted. When a change needs a restart, it rolls the brokers one at a time.
- The entity operator runs next to every cluster. Its user operator turns each KafkaUser into a password, permissions and quotas, thousands of users across the platform.
- Cruise Control is deployed by Strimzi on every busy cluster. A KafkaRebalance resource asks it to move partitions, for example after a broker upscale.
- Metrics come from the JMX exporter in every broker and a Kafka exporter for consumer lag, both switched on through Strimzi.
We describe each cluster in a values file: how many brokers, which node pool, which disks and which users. One Helm chart turns that file into these resources, and the operators do the rest.

A small chart on day one
That chart has been there from our first cluster. It started small, with six templates: the Kafka cluster, its broker and controller node pools, and the metrics setup. Everything else was added when a cluster first needed it:
- Kafka users and Cruise Control rebalances became templates.
- One user entry could stamp out many users with the same permissions and quotas.
- Broker settings were tuned for shared clusters, the work behind the Engines series.
- Every cluster got a Cruise Control UI.
- Adding and removing brokers got their own rebalance settings.
Teams moved to the new clusters gradually, and the platform grew from a single cluster to 33. Every new cluster starts from the same chart, and only its values file is different. How we use that chart to change 33 clusters safely, from validating every change to restarting brokers on our own schedule, is the story of Part 2.
New machines, same cluster
A values file names a node pool. It never names a machine type. To move a cluster to new machines, we create a new node pool and point the cluster at it. Strimzi rolls the brokers across one at a time, and each broker takes its own disks with it. One of our largest clusters moved this way to bigger machines during working hours.
The storage class decides the disk
A values file says very little about a disk: a size and a storage class. The class decides the rest:
- Disk type: a mix of standard, SSD and newer high-performance disks, matched to each workload
- IOPS and throughput, set when a newer disk is created.
- File system and mount options: XFS and noatime. Engines Part 1 showed these made our disks much faster, and with the settings in the class, nobody has to remember them for a new disk.
- Zones: a disk can be created in one zone, or in either of two for clusters that span zones.
- Volume expansion: a disk can grow in place, with no broker restart.
A storage class can't be edited once it exists, so every lesson becomes a new class. A cluster moves to a new class by getting new disks. Each broker's storage is a list of disks, so we add a disk on the new class, let Cruise Control move partitions onto it, and remove the old one. Every disk is set to outlive its broker, so removing a broker leaves its data in place.
Chapter 3: One address for every broker
A broker is only useful once clients can reach it. When it starts, it tells the cluster one thing about itself: the address where clients should find it. This chapter is about making that address work from every cluster and through every restart.
1. Bootstrap. The client is configured with one bootstrap address. It connects there, to any broker, to ask a single question: who is in this cluster?
2. Metadata. The broker replies with every broker's ID and its advertised address, and which broker leads each partition.
3. One connection per broker. The client opens its own connection to each broker it needs, using exactly the address that broker advertised. A producer writing to partition 7 must reach the leader of partition 7, and no other broker will do.
4. Leaders move. When a broker restarts or Cruise Control moves leadership, the client refreshes its metadata and connects to the new leader, again by its advertised address.
So whatever sits between clients and brokers has to give every broker its own stable address, keep it working through restarts, and make it reachable from everywhere a client runs. A load balancer that spreads connections across brokers breaks step 3. Keep that in mind. It shapes this chapter and the next.
Crossing a cluster boundary
Now put the interlude next to where Kafka lives. Clients run in the business-unit clusters, and brokers run in their own. Every connection from step 3 has to cross a cluster boundary, over the shared private network, and still reach one exact broker. For most protocols one load balancer would do. For Kafka, it breaks step 3.
What did the design have to satisfy?
Before picking a tool, we turned the interlude into six requirements for the path from clients to brokers. The rest of this post answers them one by one.

Why we needed a port-based proxy
Every Meesho Kubernetes cluster already runs an HTTP ingress, so we evaluated it first. It is designed to route TCP traffic by TLS server name, while our Kafka clients are routed by port. Kafka also needs its own port for every broker, hundreds per cluster. So we chose ingress-nginx, a proxy that can route plain TCP by port, and run it in our own tiers in front of the brokers.
Each broker advertises the proxy
Remember step 2 of the interlude: every broker tells clients its own address. By default, a broker on Kubernetes advertises an address that only works inside its own cluster, and a pod's IP changes on every restart.
Strimzi lets us override, broker by broker, the host and port a broker advertises. So each broker advertises its own DNS name and its own port, and both point at the proxy. The chart writes an override for every broker ID in the cluster's pre-wired slice, including brokers that don't exist yet. A cluster can carry overrides for twice as many brokers as it runs today.
The port is the broker's identity
A TCP proxy sees the port a connection arrived on, and then bytes. The port is the only thing that tells the proxy which broker a connection is for. So every Kafka cluster owns a block of ports on the proxy: one for bootstrap, and one for each broker.
We use the tcp-services map in ingress-nginx, which forwards a port on the proxy to a Kubernetes Service. Each line in it takes one port from the block to one broker's own Service, which selects exactly one pod. The broker's advertised name points at its business unit's nginx tier, and its advertised port picks the line in the map.
The per-broker Service is the trick. When a broker restarts, its pod gets a new IP. The Service follows it, and the broker is reachable again the moment it is ready. DNS, the map and every client stay exactly as they were.

Wired once, used for many sales
A new broker needs three things before a client can reach it: a listener override, a line in the nginx map and a DNS name, and all three must agree. If every sale needed all three for each new broker, a network change would land in the same week as the scale-up. One missed line would send clients to an address that goes nowhere.
So we wire a slice of ports on day one, big enough for several sales. When a cluster goes from 12 brokers to 18, brokers 13 to 18 already have a port, a map entry and a name. Only a cluster that outgrows its slice needs a network change, and we make that well before the sale.
That makes a sale upscale a one-line pull request that changes each cluster's broker count. The checks around it make it safe. The chart is rendered before and after, and the diff may contain nothing but the count. A count below three is refused, a jump of more than three times has to be typed again, and every new broker must already have its port, map entry, listener override and name. Our recent sales have run this way.
A proxy tier per business unit
A problem in one business unit's traffic should never take Kafka away from another. So each business unit gets its own proxy tier, on its own node pool. Each tier has its own pods, nodes and port map. The proxies themselves are small and many, so losing one reconnects only a small slice of clients.
Who gets in, and how much?
Every client signs in with its own credentials and permissions. Every credential is a KafkaUser in the cluster's values file, which Strimzi's user operator turns into a password, permissions and quotas, so access is reviewed like any other change. Almost every user also has produce and consume quotas, so the broker can slow before it affects its neighbours.
How much does the proxy carry?
The tiers hold a very large number of long-lived connections, and a sale week pushes that number much higher. During our biggest sale week, open connections went past 2 million.

Connections that live for hours
Kafka connections stay open for hours, so HTTP defaults would cut clients off on every proxy rollout. Each nginx pod gets fifteen minutes to drain before it stops, and the rollout waits seven minutes before replacing the next pod. Idle streams stay open for fifteen minutes, keepalive is on towards the brokers, and tiers with big batch producers get larger stream buffers.
Each tier sat behind an internal cloud load balancer, one address reachable from every network, and that felt like the simple part. It wasn't.
Chapter 4: The load balancer in the data path
Pricing that followed bytes
The load balancer in front of each proxy tier is priced per gigabyte, a model that suits most traffic. Kafka traffic is almost entirely bytes, so for us that cost grew with every sale rather than with the number of brokers. That broke the last requirement on our list.
Pod IPs, straight from DNS
The fix was to take the load balancer out of the data path. A headless Service selects a tier's nginx pods, and external-dns publishes all their IPs under one DNS name. A client resolves the name, picks an address, and connects straight to an nginx pod. The chart builds the headless Service's ports from the same facts as the port map, so the two paths stay in sync, and moving a cluster between them is one DNS record.

Today most of our traffic reaches the proxies without a load balancer. The rest moves one DNS record at a time, which is why both paths still run side by side.
Counting what a design costs
Load balancers carry a second cost, on every node. Our eBPF dataplane keeps every Service port in a per-node map, and each load balancer Service takes entries for every port and every ready backend. A headless Service takes none. So we track the map's fill level as a capacity number with its own alert, and every tier that moves to the headless path frees space in it.
Keeping address headroom
Service addresses are a second shared resource. Every broker draws one from a fixed, cluster-wide range, so we check that range before every sale. Our metrics Services needed an address only so the scraper could find pods, so we recreated them as headless Services, which need none. That keeps comfortable headroom for sale upscales.
Where we landed
The scoreboard. We started with six requirements. Here is what each one answered.
What we would tell someone starting today.
- Give every broker a node of its own, so one node problem costs one broker.
- Put disk tuning in the storage class, so every new disk gets it without anyone remembering.
- Wire ports, listeners and names for brokers you don't have yet, so a sale is only a broker count.
- Keep anything priced per byte out of the data path, and count what a design uses on every node and in the address range.
What's next?
Every broker now has a home and an address. In Part 2, we go further inside the cluster: how a change reaches all 33 clusters safely, how we decide when brokers restart, and what we built for the things Strimzi doesn't ship. That means certificate renewals and node upgrades on our own calendar, one Cruise Control UI for every cluster, Kafka UI and schema registries for the teams, and a new cluster that starts from a form. Stay tuned.
The Team
Thanks to Ramiz Mehran, Tushar Jaiswal, Avinash Devadhar, Ujjwal Kumar Pandey, Harshit Karwal, Harshdeep Shrivastava and Ujjwal Verma, who built, tuned and ran these clusters with us.
References
Engines - Part 1 · Abstractions - Part 1 · Strimzi · ingress-nginx TCP services · ExternalDNS

