Faciem

Turning a face model into a face search engine

A modern face recognition model is a solved-looking thing. You hand it a crop, it hands back a 512-dimensional vector, and cosine similarity tells you whether two crops are the same person. On the standard verification benchmarks it will score above 99%. You could be forgiven for thinking the hard part is over.

It isn't. A model gives you a comparison. A search engine needs an answer: given one photo, who is this, and where else does this person appear across a million images. Getting from the first to the second is a lot of work, and almost none of it is the model.

Faciem is a reverse face search system built and tested over a corpus of roughly one million scraped images. This is a note on the parts that aren't off the shelf, and — more usefully — on how we found out which of our own ideas were wrong.

Where it is

Everything below is measured on that corpus, on one consumer GPU (RTX 5060ti)

1. Two models, one scale

Faciem runs an ensemble: an ArcFace backbone and an AdaFace backbone, both ONNX, no PyTorch at runtime. Ensembling recognition models is an old idea, and the naive version of it is broken in a way that is easy to miss.

The problem is that cosine similarities from two different models are not comparable numbers. A 0.4 from one model and a 0.4 from the other do not mean the same thing, different training regimes put the impostor distribution in different places. Averaging them means silently weighting whichever model happens to run hotter.

So each model is calibrated onto a common scale before fusion, using the corpus's own pairwise distribution:

fused(i,j) = Σ_m  (w_m / Σw) · ( a_m · cos_m(i,j) + b_m )

a_m = MAD_ref / MAD_m
b_m = median_ref − median_m · a_m

median and MAD come from a sample of the gallery's own pairs, which are overwhelmingly impostor pairs, since most pairs of faces in any large corpus are different people. It's a robust affine map of each model onto a reference model's units. Cheap, recomputed on every save, and it means a threshold tuned once means the same thing after a model is swapped.

One consequence that surpising: calibrated similarities can exceed 1.0. That's an affine transform (not cosine)

2. Identities! (not nearest neighbours)

Most face search is nearest-neighbour retrieval: you get back a ranked list of images. Faciem clusters the entire corpus into persistent identities first, and searches those.

Clustering 740k faces is where the real engineering lives. The approach is Chinese Whispers over a k-nearest-neighbour graph (k=20, edge threshold 0.53) rather than agglomerative linkage. The reason is chaining: with linkage, A–B strong and B–C strong merges A with C on no evidence at all. Early in the project a single identity reached 51,149 faces.. an estimated ten thousand different people welded into one blob by exactly that named mechanism. Chinese Whispers is label-propagation with a vote, so a weak bridge between two dense regions gets outvoted instead of honoured.

3. The cheap invarient

One line holds the whole storage layer together:

faces.id  ==  vector row  ==  memmap row

Vectors live in per-model float32 memmaps, metadata in SQLite, and the row index is the primary key. Reading face 431,905's embedding is a seek. It also means a face row can never be deleted (excluding a face orphans it (identity_id = NULL) instead)

The same idea pays off again in identity comparison. For identities A and B, the mean pairwise similarity across every cross pair is exactly:

cross(A,B) = ⟨S_A, S_B⟩ / (|A|·|B|)        S = Σ unit vectors

One O(n) pass to sum, then every pair afterwards is a dot product. Scoring all 447,169 candidate identity pairs in the corpus takes seconds instead of hours. The same algebra gives a label-free health score for a single identity.. how much it resembles itself:

cohesion(A) = (‖S_A‖² − n) / (n(n−1))

4. One rule. Pixels Only

The corpus carries rich metadata: who posted each image, when, where. None of it is allowed anywhere near recognition. Not as a feature, not as a prior, not as a tiebreak.

This is a deliberate cost. Using metadata would measurably improve clustering on this corpus and would produce a system that only works on this corpus. The goal is face recognition that generalises past its training ground.

The rule is enforced mechanically rather than by discipline.

The harness that fooled

For months the clustering was scored against 152,878 face pairs drawn from the assumption "two faces posted in the same submission are the same person."

That assumption is false, and worse, it is false asymmetrically. Group shots and multi-person posts mean the harness rewarded merging different people together. Every tuning decision validated against it had been optimising toward its bias.

The lesson stuck: a benchmark that agrees with you is the most dangerous kind.

When the public benchmark disagrees with your domain

We built a proper labelled bench on LFW: 13,185 faces, 5,736 real identities, scored on both pairwise and BCubed metrics, about four minutes per parameter sweep.

On that bench, raising the graph neighbourhood from k=20 to k=30 was a clear win: pairwise recall 0.927 → 0.976. So we tried it in production, where one verified identity went from a clean cluster of 4 faces to a 647-face blob.

Both measurements were correct. LFW is curated, frontal, and roughly balanced. A scraped corpus is none of those, and its density lets a larger neighbourhood weld distinct people together.

Predictions before tests

The k=30 result came with a hypothesis attached: it blows up in production because the corpus contains low-quality junk faces acting as bridges; purge them and k=30 will convert.

That's a falsifiable claim, so we tested it, and it was refuted. Low-cohesion identities turned out to be rare identities. Manually verified faces sat between the 0.4th and 7.4th percentile of the "junk" score. Purging would have deleted real people.

The failed prediction produced the actual finding: k=30's blowups are chain welds, and welds are size-asymmetric; a small correct cluster gets outvoted by a large neighbour, so the damage always runs the same direction.

A related idea got the same treatment. A "corroboration" vote, where a neighbour only counts if its own top neighbours agree, sounded obviously better than a plain similarity sum. On real labels it collapsed: pairwise recall 0.235 against the plain sum's 0.929. And a margin-based vote that genuinely beat the baseline on false-positive rate still lost, on a two-sided benefit/cost audit, to the much dumber intervention of raising the threshold from 0.50 to 0.53 — 15.2 against 4.5. Neither shipped.

We also recorded the methodological error that nearly hid that last result: comparing false-positive rates at matched identity counts doesn't discriminate between methods unless you run the same audit against the unmodified baseline. Several sweeps had to be redone.

The blind spot nothing could see

Here is the problem that stayed open longest. Every instrument we had: false-positive rate, weld detection, cohesion, manual review, detects over-merging. Two people in one cluster is visible: you scroll the cluster and see two faces.

None of them can detect over-splitting. One person wrongly split into two identities means noticing a cluster that isn't there. You cannot find it by browsing, and no pixel-derived metric can flag it without circularity, face similarity is precisely the signal that already failed to merge them.

The break came from outside the pixels. If a single posting account uploads face A and face B in different images, and A and B also look alike, they are probably one person the clusterer split. Neither half is sufficient: accounts post more than one person, and similarity alone already failed. Together they're strong. Measured against a 40,000-sample random-pair null:

Of 447,169 identity pairs sharing an account, 2,875 clear the clustering threshold. 711 of those are explained away by the two identities appearing in the same image — a group photo or a double detection — leaving 2,164 genuine over-split candidates, 1,500 of them with ten or more faces on both sides. The first pass merged 766 identities.

Crucially, this signal is used to measure and to propose, never as a clustering input. Feeding it back would put metadata in the pipeline, which is the one thing the architecture forbids. The grep test would fail on purpose.

An independent check nobody designed for

A pleasant accident. The clustering has never seen a username. But once faces are grouped, you can ask whether accounts linked by face alone have similar names — the user123 / user123alt pattern.

Across 18,734 face-linked account pairs, 4.26% have name similarity ≥0.5, against 0.39% of random pairs — a 10.8× lift. At ≥0.7 similarity the lift is 213×.

That is evidence from a channel the system cannot access, confirming a conclusion it reached from pixels alone. It's not proof — anyone choosing deliberately unrelated names is invisible to it — but it's the closest thing to an out-of-band validation we have.

What's still open

• ~2,000 over-split candidates above threshold awaiting review, ~1,500 of them

substantial on both sides.

• Chained merge groups held back deliberately: a component where A–B and B–C are strong

but A–C is weak is exactly the failure that produced the 51,149-face blob, so any group whose weakest internal link falls below the threshold is quarantined for manual review rather than fused.

• Threshold tuning using the split labels. They're valid as an evaluation set. Using

them to tune is a different question — the line between measuring with an outside signal and training on it is thinner than it looks, and worth being paranoid about.

The short version

The model was never the hard part. The hard parts were putting two models on the same numeric scale, choosing a clustering algorithm whose failure mode we could live with, finding a storage invariant that makes identity comparison free — and, above all, noticing that our primary benchmark had been quietly rewarding the wrong answer for months.

Most of what we tried didn't work. The record of what didn't, and why we believe that, is the most valuable artifact the project has produced.