Statistical ratios are vital in gauging product performance, and thus, company growth.

Statistical ratios like a/b are calculated on day-to-day tasks across domains. For the hospitality industry, such ratios can be booking/view, cancellation/booking, etc. In the e-commerce industry, ratios such as clicks/views, cart/click, order/cart, return/order, etc are crucial indicators of product conversion rate and performance.

So, it’s important to calculate reliable ratios but in the case of fewer impressions, these ratios can be unreliable and need fixing.

Challenge in calculating statistical ratios

While this approach of calculating ratios is fairly common across industries, this post illustrates the problem with an example of scoring performance of products on an ecommerce platform.

Specifically, we analyse the problem we faced at Meesho while calculating ratios relating to products with low impressions.

Meesho hosts millions of products on its platform and a substantial number of these products suffer from the cold-start problem. Meaning: These products don’t get enough traction when they are uploaded, and end up falling deep inside the discovery page. They don’t have a high “click” or “views” rate.

These low “views” and “clicks” products create unstable ratios. Here’s how:

Assume that products A, B, and C were listed on Meesho exactly 2 months ago.

  1. A got 15 views and 0 clicks; its click/view (c/v) ratio is 0
  2. B got 15 views and 15 clicks; its c/v ratio is 1
  3. C got 50,000 views and 25,000 clicks; its c/v ratio is 0.5

The c/v ratio in the case of B is higher than the c/v ratio in the case of C. So, according to the ratios, B is performing better than C but actually, that might not be the case. It is also noteworthy that C’s data is more reliable because of the high impression count.

Also, the ratio in low impression cases (such as in cases A & B) is very volatile. It changes drastically even with a little change in values. These cases are called low confidence cases.

To eliminate this problem, there was a need to devise a solution that:

  • stabilises the product ratios with fewer impressions taking into account the properties of the product like the category it belongs to
  • has zero to minimal effect on products with already good history or impression count
  • takes into account cold products with fewer impressions and breaks the symmetry

Using the Bayesian Framework to calculate reliable ratios

Bayesian inference is a method of statistical inference in which the Bayes’ theorem is used to update the probability for a hypothesis as more evidence or information becomes available.

Bayesian inference is used to derive Posterior Probability (the revised or updated probability of an event occurring after taking into consideration new information). This derivation is made by taking into account a prior probability (prior belief) and a likelihood function (calculated from observed data).

Posterior Probability ∝ Prior Probability * Likelihood Function

Let’s study the approaches used to derive each element of this framework.

High-level idea of the approach to calculate reliable ratios

Here’s the approach of calculating reliable ratios at a product level:

  1. Calculate the prior probability/ratio at the category level. These categories can be saree, shoes, jewelleries, etc. Each category has multiple products in it. The priors calculated at the category level learn the property of the category.
  2. Look at the observed data at the product level to calculate the likelihood function.
  3. Update posterior probability/ratio considering prior and likelihood from steps 1 and 2, which becomes our reliable adjusted ratio.

Calculating Category-level Priors

We start with learning the prior ratios for the category. These categories can be saree, shoes, jewelleries, etc. Each category has multiple products in it.

  1. We calculate the impression counts like views, clicks, carts, orders, and returns for each product within the category for a period of 90+ days.
  2. With the above impressions count we can calculate the ratio like “click/view, order/click, return/order”, etc for all the products.
  3. We pass all the product’s ratios within a category to fit a beta distribution. A beta distribution is fitted per category and per ratio type click/view, “order/click, return/order”.
Categories in the Meesho App

The beta distribution is an ideal fit for modeling proportions defined on the interval [0,1]. It is parameterised by two positive shape parameters, alpha (α) and beta (β) which control the shape of the distribution. On fitting the distribution for each category, we get to learn its properties:

It is important to note that in the above formula chart if we scale up α and β by a constant factor, the mean of the distribution will still remain the same but the variance will drastically reduce.

The distribution Beta(5,40) is shown below:

Image derived from https://homepage.divms.uiowa.edu/~mbognar/applets/beta.html

For, one of the categories, the alpha-beta values obtained are, α = 0.05 and β = 4.92 for the category’s order/click distribution. For other categories also the alpha-beta values are in a similar range.

We scale up the α and β values by multiplying them by any factor, let’s say 10. This doesn’t change the expected value (mean of the distribution) but reduces the variance significantly and provides stability to the distribution. As a result the scaled-up parameters become α = 0.5 and β = 49.2. The expected conversion ratio becomes 0.5/(0.5 + 49.2) = ~0.01 i.e. 1% chance of a clicked product in the category being converted in order on average.

This is how we calculate the “Prior Probability” at the category level where we fit a beta distribution to every category on some past data.

Posterior Probability ∝ Prior Probability * Likelihood Function

Now that we’ve figured out “Prior Probability”, let’s move on to finding “Likelihood Function”. But first, let us know why Beta Distribution is chosen to learn Category-Level Priors.

The Beta distribution is chosen as it is a suitable distribution for the random behavior of percentages and proportions defined on the interval [0, 1]. Also, in Bayesian inference, the beta distribution is the conjugate prior probability distribution for the Bernoulli, binomial, negative binomial, and geometric distributions, which basically means the posterior ends up being in the same distribution as the prior.

Calculating Likelihood Function at product-level

The Likelihood function needs to be calculated at the present data at the product level. For every date, we go back 30+ days and calculate the impression counts like views, clicks, carts, orders, and returns for each product.

Such a setup is an ideal scenario for Binomial likelihood. The Binomial distribution is the probability distribution that describes the probability of getting k successes in n trials where the probability of success at each trial is p. In our scenario, success is the orders and total trials are clicks.

Calculating the Reliable Posterior Ratio at product-level

We then update the prior probability via the likelihood to obtain a reliable posterior ratio at the product level.

So basically, we take the prior, Beta (α,β), and add the success (orders) to α and the failures (clicks-orders) to β, and we obtain Beta (α+orders, β + (clicks-orders))

Problems solved

The key problems solved by this approach are:

  1. The final stable ratio for a product is drawn from Beta(α+orders, β+(clicks-orders)).
  2. Whenever the product has a fewer impression, α and β provide stabilising and smoothing effects.
  3. When the product has significant impressions, α and β don’t have much impact, as intended.
  4. For cold products having fewer impressions, mostly α and β drive the ratio which is nothing but the average property of the product in the category.
  5. α and β represent the property of category as well as the type of ratio pretty well. We have seen the expected values of cart/click and order/cart are very different and α and β learn it well.
  6. Since we are drawing the final probability from Beta(α+orders, β+(clicks-orders)), which is a distribution, the probabilities vary and there is symmetry-breaking even for cold products.
  7. For some likelihood functions, when we choose a certain prior, the posterior ends up being in the same distribution as the prior getting the name conjugate prior. In our scenario, the beta is a conjugate prior used with binomial likelihood.
  8. Instead of drawing just one ratio/probability from the posterior distribution, we draw k ratios/probabilities for each product. At run-time, any one of the k-probabilities/ratios can be chosen by the downstream task. This provides an even better exploit mechanism.

Additional advantages of this approach:

  1. Once we have the reliable ratio, they can be used in many downstream tasks as input. For example, a return/order for all the products of the supplier can be used to generate a supplier rating.
  2. We can get the list of products sorted by popularity with these ratios.
  3. We can get the list of products sorted by conversion rate (converting to order) with these ratios.
  4. A sorted list of top products by ratios can form a good candidate (filtering stage) to pass to a ranker. (AML model to find the user-level personalised rank ordering).
  5. Tracking the changes in the ratios of a product over days can be an indicator of whether the quality of the product has improved or degraded with time.
  6. Also, tracking the changes in the ratio of a product or category over months can help to discover product/category seasonality.

Results

We tested the goodness of Bayesian framework-adjusted ratios versus plain-vanilla ratios by backtesting in the past time period.

We calculated the top-k ranked products sorted by vanilla “click/view * order/view” and others sorted by adjusted ratios in the same manner till a given time point.

Then we looked at the immediate future and what the recommendation metrics like Precision and MAP (Mean Average Precision) would have looked for the following recommendations. We found a significant gain in Map@k% across different k’s as expected. Here, k is the slot:

We also analysed a list of viewed products across all the users for a period of 7 days. We re-ranked these products per user once randomly and next with adjusted ratios. We got an overall MAP improvement of 66% on average (ranking should ideally push ordered products out of viewed ones to top slots).

Road Ahead

As a next step, we will experiment with tweaking the likelihood part of the equation by trying out a weighted average of interactions (views, clicks, carts, orders, and returns) over a period of time giving more weightage to recent days. This should capture the recent trends well and bring more diversity to the product collection on the website.

While the numbers are still maturing, we are expecting to see increased effectiveness across tasks. Stay tuned to this page to know about the impact of this exercise.

Think you have the calibre to take up similar challenges? Hop onto our careers page and find a role that best fits your aptitude.

Special thanks to Rama Badrinath, Divay Jindal, and Neha Mangal for working closely on the project and Ravindra Kumar Yadav and Debdoot Mukherjee for their guidance.

Credits: