Introduction: The Need for Bulletproof Deployments at Scale
As our platform scaled and user expectations rose, so did the need for rock-solid infrastructure. One critical challenge surfaced: ensuring zero-downtime deployments while rolling out new features at speed.
Despite using canary deployments, we repeatedly ran into bottlenecks and system stability issues during rollouts. It was clear—we needed a more robust, scalable, and observability-friendly deployment solution tailored for our specific needs.
To deliver seamless user experiences at scale, Meesho built Rollout Service—our in-house deployment manager—to overcome limitations of our previous solution and ensure faster, safer, and more resilient deployments.
Our Legacy Deployment Tool Couldn’t Keep Up
Our engineering teams move fast, releasing new features and improvements frequently. However, with this velocity came an increased risk of production incidents. Even with canary deployments, we found that:
- Failures slipped through the cracks due to limited validation during canary phases.
- Mean Time to Mitigation (MTTM) increased, as the sequential model delayed urgent rollbacks.
- It didn’t scale well—the synchronous model choked under high deployment loads.
- Observability was lacking—debugging issues during rollouts felt like flying blind.
Why We Couldn't Just Rip Out the Existing Tool
Our entire production deployment flow was tightly integrated with our existing deployment manager. It managed our primary deployables—the ones that served live traffic. Replacing it without careful planning could have caused massive downtime.
We initially explored extending the tool itself. But as we dug into its architecture, we realized:
- Its rigid deployment flow offered little room for customization.
- It wasn’t built for large-scale, concurrent deployments.
- Making changes meant ensuring compatibility across its entire ecosystem—a maintenance nightmare.
Our Solution: Building Rollout Service, Tailored for Meesho
We decided to build an in-house deployment orchestration platform—Rollout Service—from the ground up, engineered specifically for Meesho’s dynamic scale and platform complexity. The decision wasn’t just about replacing a tool; it was about rethinking the entire deployment pipeline to meet our current and future operational demands.
This wasn’t a quick patch—it was a deliberate effort to design a system that could deliver high availability, offer intelligent rollout control, and minimize blast radius during failures. We needed a platform that gave us visibility into every step of the deployment, flexibility to respond in real time, and the ability to scale to thousands of services without friction.
Our 3 Core Design Principles
- Simplicity – Easy to use and manage.
- Scalability – Handle multiple deployments concurrently.
- Efficiency – Minimize deployment latency and API overhead.
Key Features Introduced
- Multi-metric analysis with flexible thresholds.
- On-the-fly modification of canary weights for live performance testing.
- Instant rollback—even mid-rollout.
- Canary-specific traffic testing.
- Seamless integration with our existing setup during transition.
How We Did It: Temporal Workflows + Smart Hybrid Strategy
To orchestrate these complex workflows reliably, we used Temporal, a framework purpose-built for distributed systems. Its benefits included:
- Execution & Retries
- Built-in observability through a user-friendly UI
- Checkpoints for resilience and easy recovery
We created five key Temporal workflows to handle each rollout stage—from pre-checks and canary deployments to analysis, promotion, or rollback.
The Smart Hack: Using Our Legacy System as a Skeleton
Instead of removing the old system entirely, we built Rollout Service around it:
- It acted as a thin passthrough.
- RolloutService was triggered via API and handled everything from there.
- This allowed for gradual onboarding without disrupting live services.
Architecture Before vs. After Rollout Service
Before Rollout Service:
- The existing system handled the full deployment lifecycle.
- It lacked flexibility, offered minimal visibility, and delayed rollback response times.
- System degradation risks were higher due to insufficient monitoring and mitigation control.
After Rollout Service:
- The existing system simply triggers Rollout Service through API calls.
- Temporal handles all core logic, from rollout orchestration and metric evaluation to promotion and rollback.
- We now have robust rollback mechanisms, precise canary control, and real-time observability across services.
Challenges We Faced
Rolling out a rollout tool comes with its own set of high-stakes challenges. Our top priority was minimizing the blast radius during the migration process.
We couldn’t risk switching everything at once. So, we implemented a whitelisting strategy, where services were progressively onboarded to Rollout Service in a controlled manner.
How This Helped:
- Controlled Adoption – Only vetted services were onboarded initially, allowing us to iron out bugs early.
- Failure Isolation – Any issues during the rollout were contained within a small subset of services, reducing risk to production.
- Safe Deprecation – The old system could continue to serve uninvolved services while we refined the new one.
This approach ensured a smooth, non-disruptive transition and allowed us to scale the new platform confidently.
The Result: Safer, Smarter, Faster Deployments
Today, Rollout Service orchestrates multiple production deployments across Meesho, delivering consistent, reliable, and resilient rollouts at scale. By incorporating intelligent workflows, real-time observability, and rollback capabilities, we've redefined how deployments happen at Meesho.
This isn't just a tool—it’s a foundation that supports our engineering velocity while minimizing risk. Rollout Service is now deeply integrated into our deployment ecosystem and continues to evolve, supporting new service types, dynamic traffic shaping, and adaptive rollout strategies based on live feedback.
Key Takeaways:
- Custom beats generic – When scale and complexity grow, tailor-made infra often outperforms one-size-fits-all solutions.
- Visibility is power – Real-time observability and traceable workflows reduce debugging time and boost developer confidence.
- Mitigation speed matters – Fast, intelligent rollbacks are essential to protect user experience and reduce incident impact.
- Migrate methodically – Gradual rollouts and hybrid approaches are key when replacing core infrastructure without disrupting business.
- Plan for growth – Build systems with extensibility and scalability as first-class principles—not afterthoughts.
Rollout Service has given us the confidence to move fast and stay safe—exactly what high-growth engineering teams need.
Conclusion
What began as an initiative to improve rollout speed and reduce incident risk has now transformed into a fundamental pillar of our infrastructure. Rollout Service is more than a deployment tool—it represents our belief that platform resilience, developer velocity, and user experience can all go hand in hand.
By building a system tailored to our unique scale, we unlocked:
- Confidence for engineers to ship faster without fear of hidden bugs reaching production.
- Transparency for operators through rich observability and real-time insights into deployment states.
- Continuity for users by eliminating downtime during critical rollouts.
This journey taught us that sometimes, the right answer isn’t adapting an existing tool—it’s rethinking the problem entirely. We challenged conventional norms, used the strengths of our legacy system as a launchpad, and architected something that could grow with us.
But the journey doesn’t stop here.
We’re continuously expanding Rollout Service to support even more granulan live traffic behavior, anr control, adaptive rollouts based od deeper integrations across our developer experience stack.
The takeaway? When infrastructure becomes a growth enabler—not a blocker—you unlock the freedom to innovate at scale.
Stay tuned—we’ll soon share deeper dives into how Rollout Service is evolving to meet the demands of tomorrow’s engineering teams.
