Turning 47,591 face detections into 22 characters

The problem I solve with Face Recognition Track (FRT) is, in simple words, putting a consistent label to a face. When I first started, that label was just a colored bar. After a few iterations, the less helpful colored bar became a text, such as "Char. 1", and finally I arrived at the character name.

Getting there meant solving the "consistent" part first, then the "name" part. This post walks through both, using a real run: The Departed (2006), 151 minutes.

Collecting faces

What I do is basically scan every 6 frames for all faces present. In movies and shows running at ~23–24 frames per second, that's around one scan every 0.25 seconds. Each face found is stored as a vector (an embedding) along with its timestamp and bounding box.

For The Departed, that produced 47,591 raw detections.

This step is where almost all the time goes. Clustering and headshot matching are cheap by comparison; running the detector over a two-and-a-half-hour film is not. The whole pipeline takes about 16 minutes for a 151-minute movie, and the detection pass dominates that.

I'm using InsightFace's buffalo_l pack, which bundles SCRFD for detection and an ArcFace model for embeddings. I didn't think much into this choice; it's the default pack, it's fast enough on GPU, and it produced clean results on my first tests. The embeddings are 512-dimensional and L2-normalized, which is why cosine distance is the natural metric for the DBSCAN step below.

Why every 6 frames

I tried sampling more densely. Going from every 6 frames to every 3 roughly doubled the runtime; every 2 roughly tripled it. The overlay looked essentially the same.

That makes sense once you think about what the overlay actually needs to do. It isn't tracking a face pixel-by-pixel, it's just answering "who is on screen right now," and that answer doesn't change four times a second. A quarter-second sampling rate is already finer than the thing being measured. So I kept 6.

Clustering

With a list of embeddings in hand, I run DBSCAN with metric="cosine" to group them. Each resulting cluster is, in theory, one person.

DBSCAN is a good fit here for one specific reason: I don't know in advance how many people are in the film. Algorithms like k-means want that number up front, and "how many characters are in this movie" is precisely the question I'm trying to answer. DBSCAN figures out the group count from the data, and it has a built-in notion of noise (points that don't belong to any cluster) which turns out to matter a lot, as we'll see. For the value of eps, I simply tried three different ones and eyeballed the results till I chose 0.6.

This clustering step, plus consistent numbering of the resulting clusters, was already the concept in the earliest versions with the colored bars.

Why 274 clusters for 22 people

Here's the part I didn't expect. The Departed has around 22 characters worth labeling. DBSCAN gave me 274 clusters.

The gap comes from the fact that "the same face" is not one thing to an embedding model. The same actor produces meaningfully different vectors depending on:

• Lighting. A face in a dim bar and the same face in daylight can land far apart in embedding space.
• Angle. Profile shots and three-quarter shots often don't cluster with frontal ones.
• Occlusion and expression. Partial coverage, heavy shadow, a shouting face versus a still one.

So one person fragments into several clusters. On top of that fragmentation, there's genuine noise: background extras with a few seconds of screen time, and outright false positives, such as faces detected on posters, photographs, TV screens, and reflections inside the shot.

These are two different problems that both show up as "too many clusters," and they need different handling.

Filtering down to 22

Before labeling anything, I drop clusters that barely appeared. Without this, the screen ends up crowded with labels for people the viewer will never need to identify, which defeats the entire point of the overlay.

Screen time is a surprisingly effective filter because it catches both problems at once. Fragmented pieces of a main character are usually still substantial enough to survive; genuine extras and false detections are not. For The Departed, filtering took 274 clusters down to 22. While it may mean that a real character who appears very subtly is dropped, I found the minimums 0.5% for screen time and 10 embeddings for cluster size to be a good, reasonable balance.

From cluster to character name

Consistent numbering was enough for "Char. 1". Getting to "Billy Costigan" needed a reference to match against, so I integrated headshots into the pipeline. Each surviving cluster gets compared against those headshots, and on a match, the character's name becomes the label.

Of the 22 kept clusters in The Departed, 19 matched to an actor headshot.

Unmatched clusters with sufficient screen time are kept, they just stay on the generic, consistent "Char. #" label. That's deliberate. A character who is clearly present and clearly the same person across the film is still useful to the viewer even without a name attached, and dropping them would leave visible gaps in the overlay.

What it doesn't handle

The approach is face recognition, which means anything that hides a face defeats it. Masked and helmeted characters aren't trackable at all, which unfortunately rules out a whole category of films. Heavy prosthetics cause the same problem to a lesser degree. And in long-running series, actors visibly age across seasons, which pushes their embeddings apart in exactly the way lighting and angle do. There's a small irony here: the characters FRT can't track are often the ones you least need it for. A helmet or a mask is a distinctive marker in its own right, which is exactly what makes it easy to follow a character you can't identify by face.

The funnel

Put end to end, one film looks like this:

47,591 down to 19 names, in about 16 minutes.

If you've read this far, thank you for your time :)