Part 1 of a multi-part series. A quick note before we begin: the first chapter of Khoj.AI was built before the generative AI wave really took off, so some early choices reflect the tooling of that time. This post walks through how the system has evolved over the last 1 year, what worked, what didn't, and how those lessons are pushing us toward a much more powerful, agentic version that we'll cover in the next post.
A 2 AM problem
It's the middle of the night. An on-call engineer's phone buzzes. Something is wrong in production. By the time the laptop is open, alerts are firing across multiple services. Several dashboards are already up. None of them clearly point to what's actually broken.
Every engineer at a fast-growing company knows this moment. The alert fires, but the service that's alerted is rarely the cause. It's a symptom. Somewhere in a tangled graph of services, one of them is the real culprit and finding it is the hard part.
This is the problem we set out to solve at Meesho. We called our solution Khoj.AI. Khoj means "search" in Hindi, and that's exactly what it does: it searches for the needle in the stack so the engineer doesn't have to.
This is a story of evolution. Let's start at the beginning.
What it looks like today
When an alert fires now, an engineer opens the team channel and finds a follow-up message already waiting. Inside it:
- The predicted root cause service
- The top suspect operations ranked by probability
- A call graph showing how the request flowed
- A plain-English summary of what likely went wrong
For deeper investigation, engineers can move to a web interface and ask follow-up questions in natural language ("show me the latency trends for this API," "list the databases involved") and get answers grounded in the incident's actual data.
The investigation no longer starts from a blank dashboard. It starts from a hypothesis.
Why finding root causes is genuinely hard
Modern applications aren't single programs. They're constellations of small services that talk to each other, one for inventory, one for pricing, one for payments, one for recommendations. A single user action can touch many of them in sequence. Great for scale but painful for debugging.
A few reasons why it is painful:
- Cascading failures make every service look guilty. When one service breaks, everything downstream starts erroring. By the time alerts fire, a dozen services may be flashing red, but only one is the actual origin. The rest are just victims.
- Quiet services can be the real culprit. Sometimes a failing service has a fallback path that masks the failure to upstream services. Engineers chase the noisier service while the quiet one keeps misbehaving.
- Tracing data is incomplete by design. Distributed systems generate enormous volumes of trace data, too much to ingest in full, cost-efficiently. So sampling is applied. The trade-off: no individual trace is guaranteed to tell the full story.
- Context takes time to build. Engineers newer to the system don't yet have the breadth of knowledge to debug across many services quickly. During an active incident, that learning curve is expensive.
Dashboard fatigue is real. Scanning dashboards one by one under time pressure is slow, exhausting, and error-prone.
The result? Investigation often took longer than the actual fix. We wanted to change that.
What we tried first, and why we changed direction
The first chapter of Khoj.AI took a simpler approach. It worked entirely from aggregated service-level metrics and the dependency graph between services. The idea was reasonable: if you know which services are misbehaving, and you know which depends on which, you should be able to walk the graph and find the origin.
It worked in clean cases, a single service with a clear, isolated anomaly. But the real world wasn't clean.
When multiple services showed metric deviations at the same time, the metric-only approach struggled to separate cause from consequence. When fallbacks masked failures, service-level metrics didn't reveal the source. When degradations were subtle rather than sharp, metric thresholds missed them.
The granularity simply wasn't there.
The lesson: service-level signals are too coarse. To find the real root cause, we needed to look one level deeper, at the individual operations within services. In distributed tracing, these are called spans: a single database call, a single cache lookup, a single downstream HTTP request.
That shift from service-level to span-level analysis turned out to be the most important design decision in the system.
The three ideas that shaped Khoj.AI
1. Spans, not services
Every request leaves behind a trail of spans, fine-grained records of each operation it triggered. By analyzing spans rather than aggregated service metrics, the system can pinpoint exactly which operation in which service is misbehaving. A service might look healthy in aggregate while one specific span inside it is the actual problem. Span-level analysis surfaces that.
2. Probability over single traces
Because sampling means individual traces are often incomplete, we don't try to find one perfect trace that explains everything. Instead, the system processes many sampled traces, failed, slow, and successful, and aggregates findings statistically.
The intuition is simple: if a span shows up frequently in failed transactions but rarely in successful ones, it's likely the real cause. If it shows up just as often in both, it's probably just on the path. By weighing presence in failures against presence in successes, the system separates real culprits from coincidental travelers.
This probabilistic view also helps fill in gaps. Where any single trace might be missing entries due to sampling, aggregating across many traces lets the system reconstruct dependencies and patterns no individual trace could reveal alone.
3. The language model is a thoughtful assistant, not the engine
The temptation with anything AI-flavoured is to put the language model at the centre. We chose differently. Detection, identifying the likely root cause span, is done by deterministic statistical analysis on trace data. The LLM comes in afterward, to explain what was found, summarize the incident, and answer follow-up questions.
We designed the LLM interaction in two phases:
- Phase 1, lightweight context. The initial summary uses only high-signal findings: top impacted spans, root cause service, anomalous metrics, aggregated trace stats. No raw traces. Fast and inexpensive.
- Phase 2, full context for follow-ups. When an engineer wants to dig deeper, the system loads sanitized incident data, including sampled traces, to answer specific questions about call paths, latency patterns, and individual request behavior.
Before any data reaches the model, it goes through aggressive sanitization, capping transactions, capping spans per transaction, filtering to the fields most relevant to root cause analysis.The principle: the LLM is good at explaining and querying. It's less good at being the source of truth. So the source of truth, detection itself, stays with deterministic analysis.
How it fits together
At a high level, Khoj.AI is a pipeline. Alerts arrive from monitoring systems and land in a processing queue. After a short configurable delay (giving transient issues a chance to self-resolve), the alert is picked up. The system collects relevant traces, failed, slow, and successful, and runs span-level probability aggregation to identify the likely root cause. It checks for metric anomalies in the implicated service, sanitizes the incident data, and passes it to the LLM for summarization. The output goes back to the team's channel, with a deeper interactive view available through the web UI.
A few engineering details worth flagging
- Rolling sums for fast baselining. Instead of recomputing means from raw data every time, the system maintains rolling cumulative sums over a sliding window. Baseline comparisons stay fast, predictable in cost, and older data is auto-pruned.
- Batched metric collection. Different metrics have very different cardinality. Low-cardinality metrics go in larger batches; high-cardinality ones in smaller ones. Balances freshness against query load.Continuous warming. Background tasks fetch fresh metrics into cache, process the alert queue, and clear stale time-series data. By the time an alert lands, the system already has the context it needs.
These aren't headline ideas, but they're what determines whether a system works in production or only on paper.
The honest scorecard
Here's where we get real. The first chapter of Khoj.AI landed at roughly 38.5% accuracy in identifying and classifying incidents correctly. Useful, definitely better than starting from a blank dashboard, but nowhere near where we wanted it to be. So we sat down and asked: why?
Five things stood out.
- Too much reliance on a single signal. Khoj.AI leaned heavily on traces. The thinking was that traces had everything: upstream and downstream dependencies, function calls, stateful resource health, and point-in-time metrics. In theory, complete. In practice, the realities of high-volume distributed systems, including sampling and varying instrumentation coverage across runtimes, meant traces alone couldn't carry the full weight of analysis. We saw value in bringing metrics and logs into the picture.
- Service dependency knowledge was fragile. The dependency graph was also derived from traces. As coverage shrank in parts of our stack, end-to-end dependency analysis got unreliable. We also realised that complex dependency patterns between services could challenge graph-based reasoning. Strengthening this foundation became a clear priority for the next chapter.
- The search domain was too large. Khoj.AI was a singleton, one big system trying to do everything in one shot. LLMs perform much better when problems are decomposed and each fragment handles a well-defined slice. We learned that giving an LLM too broad a problem space at once limits the quality of its reasoning. Decomposing the problem into focused slices was a clear next step.
- No memory, no learning. There was no storage or learning layer. Every incident started from scratch, including rebuilding the dependency graph each time. That left an opportunity on the table: a system that learns from past incidents and carries context forward, so each investigation builds on the last.The code wasn't built to evolve. Khoj.AI started as a POC, and the code structure showed it. We saw an opportunity to redesign the foundation so it could more easily adopt newer signals, models, and patterns as the AI ecosystem evolves. Newer agentic orchestration frameworks now offer modularity out of the box, but the original structure couldn't take advantage of them without significant rework.
Put together, these five issues capped the system's ceiling. They're not failures so much as the natural limits of a first attempt built before the tooling and patterns we have today existed.
What's next: an agentic Khoj.AI
The next chapter of this story is already being written. The upcoming version of Khoj.AI is agentic, designed around specialized agents that each handle a focused slice of the investigation, coordinate with one another, draw on memory from past incidents, and weave together signals from traces, metrics, and logs. The singleton becomes a team. The static analyzer becomes a system that learns.
We'll dive deep into the agentic version, the architecture, the trade-offs, the wins, the surprises, in the next post in this series. If the first chapter took us from a blank dashboard to a starting hypothesis, the next one is about getting much closer to a confident answer.
What we took away
If there's one lesson from building the first chapter of Khoj.AI we'd offer to other teams: in distributed systems, the limiting factor in incident response is rarely the absence of data. The data is almost always there, in metrics, in traces, in logs. The limiting factor is signal. The work isn't collecting more data; it's separating the small set of meaningful signals from a much larger set of noisy ones, and presenting them in a form an engineer can act on quickly.
Span-level analysis gave us granularity. Probability aggregation gave us confidence despite incomplete data. A scoped role for the language model gave us interpretability without overreliance. None of these ideas is exotic on its own. The combination is what made the first chapter work as well as it did.
It's also what made it clear where the ceiling was, and why the next chapter had to be a fundamentally different kind of system.
Stay tuned.
Acknowledgements
Special thanks to Vignesh Ganesan and Aditya Madvesh Nadig for their invaluable help with everything related to infrastructure, and to Kumar Pratik, Siddharth Gupta, Subham Kamalapuri, Shubham Sharma, Anupam Satsangi, and Amlan Shekhar Das for their continuous guidance throughout the journey.
