The decision

A catalogue of 600 items, 1,200 users, and a slot on the home page showing ten recommendations. The questions the team actually has to answer:

1. Is a personalised recommender worth building at all, against showing the most popular items?
2. Which users should it be applied to?
3. How do we know the offline number is not a fantasy?

The third question is the hard one, and it is why this capstone exists. The recommender page measured that evaluating against 100 sampled negatives instead of the full catalogue inflates every hit rate by 1.90× to 2.14×. Every number below is a full-catalogue number.

Step 1 — the protocol, decided before the model

Writing this table first is the whole trick. Every reported improvement in recommender systems is negotiable if the protocol is chosen after the results.

Step 2 — the results

figurePopularity reaches hit@10 0.1976 by recommending 20 items to everybodymatplotlib

Personalisation is worth building here. hit@10 goes from 0.1976 to 0.3481 — a 1.76× improvement, measured against the full catalogue, using six lines of numpy.

Catalogue coverage is the finding nobody asks for. Popularity’s 0.1976 comes from recommending the same 20 items to all 1,017 users. As an accuracy metric that is fine; as a product it means 96.7% of the catalogue is never seen, new items can never be discovered, and every user’s home page looks identical. Report coverage next to hit rate, always.

Step 3 — the blend, and where it comes from

The cold-start measurement showed that personalisation loses on users with almost no history. So the policy is two lines:

def scores_for(user):
    """Popularity for cold users, collaborative filtering for everyone else."""
    if interactions_in_training[user] < MIN_HISTORY:      # 5
        return popularity
    return train[user] @ item_similarity

def scores_for(user):
    """Popularity for cold users, collaborative filtering for everyone else."""
    if interactions_in_training[user] < MIN_HISTORY:      # 5
        return popularity
    return train[user] @ item_similarity

The blend gains +0.0059 overall, and all of it comes from the 81 users where item-item scored 0.1728 and popularity scored 0.2469. That is a small aggregate number produced by a large segment-level one — +0.0741 for the users it affects — which is the shape most good production changes have.

See it move

MIN_HISTORYMIN_HISTORY is the only parameter in that policy, and it is a routing parameter rather than a model one — so its effect can be computed exactly from the segment table. Drag it and watch each segment switch strategy and the aggregate follow.

The weighted average reproduces the published numbers exactly, which is a useful check that the blend is arithmetic rather than magic: cutoff 0 gives 0.3481 (item-item everywhere), cutoff 5 gives 0.3540 (the blend), and cutoff 62 gives 0.1976 (popularity everywhere). Push the cutoff past 5 and the aggregate falls — 0.3363 at 8, 0.3127 at 11, 0.2724 at 16 — because you are now handing popularity to users who have enough history for personalisation to win. The optimum sits exactly where the two per-segment bars cross, and nowhere else.

Step 4 — what the online system needs

CONFIG = {"slate_size": 10, "min_history": 5, "model_version": "reco-3.0.1"}
 
 
def recommend(user_id, artefact, config):
    """A slate, plus the fields that make an offline/online comparison possible."""
    history = artefact["history"](user_id)
    cold = len(history) < config["min_history"]
    scores = (artefact["popularity"] if cold
              else artefact["similarity_scores"](history))
 
    ranked = [item for item in np.argsort(-scores) if item not in history]
    slate = ranked[:config["slate_size"]]
    return {
        "user_id": user_id,
        "slate": slate,
        "strategy": "popularity" if cold else "item_item",
        "history_size": len(history),
        "score_span": round(float(scores[slate[0]] - scores[slate[-1]]), 6),
        "model_version": config["model_version"],
        "catalogue_share_today": artefact["coverage_counter"].add(slate),
    }

CONFIG = {"slate_size": 10, "min_history": 5, "model_version": "reco-3.0.1"}
 
 
def recommend(user_id, artefact, config):
    """A slate, plus the fields that make an offline/online comparison possible."""
    history = artefact["history"](user_id)
    cold = len(history) < config["min_history"]
    scores = (artefact["popularity"] if cold
              else artefact["similarity_scores"](history))
 
    ranked = [item for item in np.argsort(-scores) if item not in history]
    slate = ranked[:config["slate_size"]]
    return {
        "user_id": user_id,
        "slate": slate,
        "strategy": "popularity" if cold else "item_item",
        "history_size": len(history),
        "score_span": round(float(scores[slate[0]] - scores[slate[-1]]), 6),
        "model_version": config["model_version"],
        "catalogue_share_today": artefact["coverage_counter"].add(slate),
    }

Three of those fields exist because of what this phase measured. strategystrategy makes the blend auditable — when the cold-user share moves, the aggregate hit rate moves with it and you need to know which. ** score_spanscore_span** is near zero when the model has nothing to say, which is a better abstention signal than a threshold on the raw score. catalogue_share_todaycatalogue_share_today is coverage as a live metric rather than an offline curiosity.

Step 5 — the honest limits of the offline number

The last two are the ones with teeth. A leave-one-out benchmark excludes exactly the users the blend was built for, and it can leak the future by holding out an early interaction. A temporal split fixes the second; nothing fixes the first except deciding, as a product, what a brand-new user sees.

Offline evaluation ranks candidate policies; it does not predict online performance. The purpose of the protocol above is to shortlist honestly, cheaply and reproducibly. The number that matters comes from an A/B test, and the offline number’s job is to make sure the A/B test is worth running.

The loop that number lives inside, with the leak that makes it optimistic drawn explicitly:

  flowchart TD
  L[("interaction log")] --> SPLIT{{"leave-one-out per user,
full 600-item ranking"}}
  SPLIT -->|"183 users have
fewer than 5 interactions"| EXCL["excluded from evaluation.
15% of the base, and exactly
the users the blend targets."]
  SPLIT --> TRAIN["fit: popularity vector
and item-item cosine"]
  TRAIN --> POL{{"blend: popularity below
5 interactions, item-item above"}}
  POL --> OFF["offline: hit@10 0.3540,
coverage 323 of 600"]
  OFF --> GATE{"Better than the
shipped policy, on both
hit rate and coverage?"}
  GATE -->|"no"| STOP["do not run the A/B test"]
  GATE -->|"yes"| AB["online A/B test --
the only number that decides"]
  AB --> SERVE["serve slates"]
  SERVE --> CLICK["users interact with
what they were shown"]
  CLICK -.->|"the feedback loop:
today's log was shaped
by yesterday's slates"| L
  SERVE -->|"small random-slate slice"| UNBIASED["unbiased evaluation data --
the only defence against
the dotted edge"]
  UNBIASED --> L
  SERVE --> POS["log slate position
per interaction"]
  POS -->|"item 1 outclicks item 10
whatever the relevance"| DEBIAS["model position explicitly,
or randomise within the slate"]

The dotted edge is why an offline hit rate cannot be a forecast: the data it is measured on was produced by the policy it is being compared against. The random-slate slice is the cheap structural fix — a few percent of traffic showing unranked items buys you an evaluation set that no policy has contaminated, and without it every future offline comparison inherits the same bias.

Recap

• 1,017 evaluable users of 1,200; every number computed against the full 600-item catalogue.
• Popularity: hit@10 0.1976 while showing only 20 distinct items (3.3% of the catalogue).
• Item-item cosine: hit@10 0.3481 across 339 items — a 1.76× improvement, in six lines.
• The blend (popularity below five interactions) reaches 0.3540, gaining +0.0741 for the 81 cold users and +0.0059 overall.
• Catalogue coverage belongs next to hit rate in every report.
• Offline numbers shortlist policies. Feedback loops, position bias and the 183 excluded users mean the decision belongs to an online test.

🧪 Try It Yourself

Exercise 1 – Set up the protocol

Exercise 2 – Score three policies at three cut-offs

Exercise 3 – Count the catalogue

Exercise 4 – Segment by activity, and find the blend’s real effect

Exercise 5 – Show what the sampled protocol would have claimed

Next

Phase 12 - Capstone Projects — the phase overview, and the six questions every one of these projects had to answer before a model was worth building.