Finding the spinning heads of poi in videos the model has never seen, and measuring honestly whether each change helps.
Problem
FIG. 1The same model, two kinds of video. F1 of the first fine-tuned model (v1). The rule runs from 0 to 1. Source: Poi head detection sheet, Problem; project.json timeline 2026-10-16
Poi are weights on cords, spun around the body in patterns. A sister project, Prop Theory, wants to turn any video of a move into an animation, and that starts with knowing where each poi head is in every frame. The heads are small, fast and blurred; on fire poi they are a flame, on LED poi a streak of light, in contact moves they sit in the hand.
I started from BlurBall, a detector built for blurred table-tennis balls, and fine-tuned it on frames I labelled. The first model did well on my own practice video and much worse on videos it had never seen. That second number is the one that matters.
Approach
FIG. 2Where every labelled frame goes. Training and the frozen validation set do all the work; the three test sets are a final look, never a selection tool. Source: Poi head detection sheet, Approach (Data, Rules of the bench)
I label frames by hand. Every training run starts from the original BlurBall weights, a frozen validation set picks the model, and the test sets are a final look, never a selection tool.
FIG. 3Night search, successive halving. 38 full trials, 9.4 hours of GPU time on one GTX 1070. Source: poi-det/hpbench.py (plan full), bench/results.jsonl, 2026-10-17
Before tuning anything, I sorted the first model's errors on unseen videos by cause. Poi type explained far more of them than head size did.
Results
FIG. 4Every finalist lands inside the noise. With 48 wild frames, the interval around v1 is wider than every difference the search produced. Each finalist is drawn as its two seeds. Source: poi-det/bench/full_final.json, 2026-10-17; interval from the sheet's Results
Run
Val F1 (selection)
iso1 F1 (final look)
wild F1 (final look)
full_r2_base_e20_s0
0.59
0.85
0.53
full_r2_base_e20_s1
0.63
0.83
0.49
full_r2_c06_e20_s0
0.66
0.64
0.53
full_r2_c06_e20_s1
0.67
0.80
0.54
full_r2_c19_e20_s0
0.68
0.75
0.52
full_r2_c19_e20_s1
0.66
0.80
0.53
TABLE 1Finalists after the search. The two winners on validation (c19, c06) do not carry over to unseen videos: on wild they sit inside the baseline's own seed-to-seed spread. Source: poi-det/bench/full_final.json, 2026-10-17 08:37
Tuning did not measurably beat the baseline. Two seeds of the same settings already differ by up to 0.04 F1.
FIG. 5More of the same data helps slowly. Winning settings, one seed per point. Overall F1 moves from 0.61 to 0.66; per-type lines jump around because each type has only a few validation heads. Source: poi-det/bench/curve.json, 2026-10-17 10:41FIG. 6A general model as labeller. F1 on wild2 (61 frames, 102 heads), labelled without any model suggestion on screen. The hosted model is slow (about 8 s a frame) and not a fixed version, so it is used to label training data, not to run in the product. Source: poi-det/bench/scorecard.json, 2026-10-17 01:10
Training on the hosted model's labels landed inside the noise as well. The validation labels also still follow my old labelling rule, which penalises the hosted model's labels on fire and LED frames.
What I learned
FIG. 7What v1 got wrong on the wild set. The 80 errors sorted by cause before any tuning. Small heads explained only 5 of 41 misses: the poi type mattered more than the size. Source: Poi head detection sheet, Approach (Error dump first)
Freeze the yardstick first. The validation set used to change whenever a new video was added, so early models were never comparable.
Count the noise before reading a gain. Every comparison now shows the seed spread next to the difference.
Label rules must be physical. I had marked the middle of a light streak; the head is at the leading edge, and on fire poi at the wick. The frames are being relabelled.
Settings were not the bottleneck. Targeted frames per poi type and better labels are.
Open questions
For a data scientist: how do I get a test set big enough to see per-type effects without labelling thousands of frames? Is the seed-pair spread a fair noise floor?
For a cloud architect: training moves to GCP next. What is the simplest setup for queued GPU jobs that write the same log files back here?