What you’ll learn

• why the empty cells of a user-item matrix are the target, not missing data
• the popularity baseline, which is not a straw man: hit rate 0.1976 at 10
• item-item collaborative filtering in six lines of numpy: 0.3481
• BPR matrix factorisation by SGD, and why it lost here: 0.3196
• the evaluation trap that doubles every published number: 100 sampled negatives
• cold start, measured: with 4 interactions, popularity beats personalisation 0.25 to 0.17

Implicit feedback

The data is 1,200 users, 600 items, and a 1 wherever a user interacted with an item. Generated from latent factors plus an item popularity term, so the reason any cell is filled is known — which lets us separate “the model learned taste” from “the model learned what is popular”.

figure2.02% of the cells are filled, and the empty ones are the prediction targetmatplotlib

The long tail, and why popularity is hard to beat

figureRecommending the most popular items is not a straw man: it is a real 0.1976 hit ratematplotlib

Evaluation, before any model

Leave-one-out: for every user with at least 5 interactions, hide one at random. That gives 1,017 evaluable users. Then, for each of them, rank all 600 items — excluding the ones they already interacted with in training — and ask where the hidden item landed.

With one held-out item per user, NDCG reduces to a rank-discounted hit rate: it rewards position 1 over position 9, which hit rate does not.

Four scorers

figureItem-item cosine, in six lines of numpy, beats matrix factorisation at this data sizematplotlib

Item-item collaborative filtering

The whole method, once the matrix is in memory:

norms = np.linalg.norm(train, axis=0)          # column norms = item popularity
norms[norms == 0] = 1.0                        # unseen items: avoid dividing by 0
unit = train / norms                           # L2-normalise each item column
similarity = unit.T @ unit                     # cosine between every item pair
np.fill_diagonal(similarity, 0.0)              # an item is not its own neighbour
 
scores = train[user] @ similarity               # sum of similarities to what you took

norms = np.linalg.norm(train, axis=0)          # column norms = item popularity
norms[norms == 0] = 1.0                        # unseen items: avoid dividing by 0
unit = train / norms                           # L2-normalise each item column
similarity = unit.T @ unit                     # cosine between every item pair
np.fill_diagonal(similarity, 0.0)              # an item is not its own neighbour
 
scores = train[user] @ similarity               # sum of similarities to what you took

The last line is the recommendation: an item scores highly if it is similar to the items this user already interacted with. No optimisation, no hyperparameters, no training loop. On this data it is the best of the four.

Two properties are worth knowing. The cosine normalisation is what stops popular items dominating — without it, train[user] @ train.T @ traintrain[user] @ train.T @ train is essentially a popularity ranking. And the similarity matrix is $600 \times 600$ here but $|I|^2$ in general, so at a million items you need approximate neighbours (NearestNeighborsNearestNeighbors with a tree, FAISS, ScaNN) instead of a dense product.

BPR matrix factorisation

Matrix factorisation learns a vector per user and per item such that the dot product ranks correctly. The BPR objective takes a seen item $i$ and an unseen item $j$ and pushes them apart:

def bpr(train, k=16, epochs=30, lr=0.05, reg=0.01, seed=1):
    """Rank a seen item above a randomly drawn unseen one, by SGD."""
    rng = np.random.default_rng(seed)
    n_users, n_items = train.shape
    P = rng.normal(0, 0.1, (n_users, k))
    Q = rng.normal(0, 0.1, (n_items, k))
    pairs = np.argwhere(train == 1)
 
    for _ in range(epochs):
        rng.shuffle(pairs)
        for u, i in pairs:
            j = int(rng.integers(0, n_items))          # sample a negative
            while train[u, j] == 1:
                j = int(rng.integers(0, n_items))
            gradient = 1.0 / (1.0 + np.exp(P[u] @ (Q[i] - Q[j])))
            pu, qi, qj = P[u].copy(), Q[i].copy(), Q[j].copy()
            P[u] += lr * (gradient * (qi - qj) - reg * pu)
            Q[i] += lr * (gradient * pu - reg * qi)
            Q[j] += lr * (-gradient * pu - reg * qj)
    return P, Q

def bpr(train, k=16, epochs=30, lr=0.05, reg=0.01, seed=1):
    """Rank a seen item above a randomly drawn unseen one, by SGD."""
    rng = np.random.default_rng(seed)
    n_users, n_items = train.shape
    P = rng.normal(0, 0.1, (n_users, k))
    Q = rng.normal(0, 0.1, (n_items, k))
    pairs = np.argwhere(train == 1)
 
    for _ in range(epochs):
        rng.shuffle(pairs)
        for u, i in pairs:
            j = int(rng.integers(0, n_items))          # sample a negative
            while train[u, j] == 1:
                j = int(rng.integers(0, n_items))
            gradient = 1.0 / (1.0 + np.exp(P[u] @ (Q[i] - Q[j])))
            pu, qi, qj = P[u].copy(), Q[i].copy(), Q[j].copy()
            P[u] += lr * (gradient * (qi - qj) - reg * pu)
            Q[i] += lr * (gradient * pu - reg * qi)
            Q[j] += lr * (-gradient * pu - reg * qj)
    return P, Q

It reached 0.3196 after 30 epochs over 13,497 positive pairs — and 0.2281 after 10 epochs, which is worth knowing before you conclude that factorisation does not work. It is an iterative method and it needs its iterations.

Why it still loses to a six-line neighbourhood method here: 13,497 interactions is not much data for 1,800 factor vectors. At 16 factors, BPR is fitting 1,200 × 16 + 600 × 16 = 28,800 parameters from 13,497 observations. Item-item cosine fits nothing. The ordering reverses on real catalogues with millions of interactions, where factorisation generalises across the sparsity in a way neighbourhoods cannot — but “matrix factorisation is better” is a statement about data volume, not about algorithms.

The evaluation trap

Ranking the held-out item against all items is expensive, so a very common shortcut is to rank it against a sample of 100 unseen items. Here is what that does:

figureThe same models, two evaluation protocolsmatplotlib

The arithmetic is straightforward — being in the top 10 of 101 candidates is far easier than the top 10 of 600 — but two consequences are not:

• Cross-paper comparison breaks. A published 0.66 and your 0.35 may be the same model.
• The ranking can change. The inflation is not a constant factor: it depends on how a model distributes scores over the tail. Krichene and Rendle showed in 2020 that sampled metrics can reorder methods, which is why full ranking (or a documented, fixed candidate set) is now expected.

If you must sample for speed, sample the same negatives for every model and say so in the report.

Cold start, measured

figurePersonalisation needs history; popularity does notmatplotlib

Both halves of this table are load-bearing.

Personalisation loses on cold users. With four interactions there is not enough signal, and the best available guess is what everyone else likes. This is why production systems are almost always a blend: popularity (or editorial curation) for new users, collaborative filtering once history accumulates, usually with a smooth interpolation rather than a switch.

Popularity gets worse as users get more active — 0.25 down to 0.13 — because the recommendation excludes items the user already has, and heavy users have consumed most of the head. An aggregate hit rate hides both effects. Always segment by activity.

Note also the 183 users with fewer than five interactions who were excluded from evaluation entirely. They are 15% of the user base, they are the hardest cases, and every leave-one-out benchmark quietly drops them. Their existence is a product problem, not a modelling one.

What to serve whom

The measured table above is a routing policy waiting to be written down:

  flowchart TD
  R["A request for recommendations"] --> A{"How many interactions
does this user have?"}
  A -->|"0 -- brand new"| Z["Editorial or popularity.
No model can help;
collect signal instead."]
  A -->|"1 to 4"| P["Popularity.
Measured 0.25 against
0.17 for item-item."]
  A -->|"5 to 15"| M["Personalised: item-item
cosine, 0.31 to 0.41"]
  A -->|"16 or more"| M2["Personalised, and drop
popularity entirely --
it decays to 0.13"]
  P --> B{"Blend or switch?"}
  M --> B
  M2 --> B
  B -->|"switch at a hard
threshold"| B1["Visible jump in the feed
the day a user crosses it"]
  B -->|"interpolate on
interaction count"| B2["Preferred: score =
w(n) * personal + (1-w(n)) * popular"]
  B1 --> E["Evaluate per bucket,
never in aggregate"]
  B2 --> E
  E --> F{"Reporting a number
to anyone?"}
  F -->|"yes"| G["State the protocol:
full ranking over all items,
or a fixed sampled candidate set.
Sampling roughly doubled every
hit rate here."]

Two of those boxes exist only because of measurements on this page: the 1-to-4 bucket routes to popularity because personalisation loses there, and the reporting box exists because the same three models scored between 1.90× and 2.14× higher under sampled evaluation. Neither is a design preference; both are what the numbers said.

See it move

Item-item cosine is one formula, and the effect of normalisation is easiest to feel by toggling it. Below, five users have interacted with six items; the similarity between two items is the cosine of their columns.

Fill an entire column — make one item universally popular — and watch what happens. Under raw co-occurrence that item becomes the most similar item to everything, so it is recommended to everybody. Under cosine it does not, because the normalisation divides by its own norm. That single division is the difference between a recommender and a bestseller list.

Pitfalls

Recap

• The matrix is 1,200 × 600 and 2.02% filled; the zeros are the prediction target, not missing data.
• The top 10% of items hold 40.3% of interactions, which is why popularity reaches 0.1976 at 10 with no personalisation.
• Item-item cosine, six lines of numpy, reached 0.3481 and median rank 23 of 600.
• BPR matrix factorisation reached 0.3196 after 30 epochs and 0.2281 after 10 — it lost here because 13,497 interactions is thin for 28,800 parameters.
• Ranking against 100 sampled negatives inflated hit rate by 1.90× to 2.14×, non-uniformly.
• With four interactions, popularity beat personalisation 0.25 to 0.17; with 16 or more it lost 0.13 to 0.40.

🧪 Try It Yourself

Exercise 1 – Build the interaction matrix

Exercise 2 – Leave one out, then try popularity

Exercise 3 – Item-item cosine in six lines

Exercise 4 – BPR by hand

Exercise 5 – Reproduce the sampled-negatives inflation

Next

Anomaly and Outlier Detection — the last applied problem, and the only one where you usually have no labels at all, which changes what “evaluation” can even mean.