Self supervised compatibility measure improves puzzle piece matching
Fix the Match, Not the Solver: STITCH -- a Self-Supervised Compatibility Measure for Eroded Visual Puzzles
Computer Vision and Pattern Recognition
Summary
Real-world jigsaw puzzles often have missing or damaged edges that make it hard for computers to figure out which pieces fit together. The authors found that the key problem is how the pieces are matched, not the puzzle-solving method used. They created STITCH, a tool that looks inside puzzle pieces to compare them without needing perfect edges, and it learns on its own without labeled examples. This tool works very well, even when pieces are heavily damaged or missing, and improves puzzle-solving accuracy significantly.
What this means in practice
- •For industrial quality control teams: Improve automated reassembly of fragmented objects during inspection using a better match scoring method for damaged parts.
- •For cultural heritage conservators: Assist digital reconstruction of eroded or incomplete artifacts by enhancing compatibility evaluation of fragment images.
Authors
Nadav Alali, Ohad Ben-Shahar
Abstract
Unlike the clean and pixel-aligned complete pieces of standard jigsaw benchmarks, real-world puzzle fragments lose their boundary regions to erosion, destroying the signal that classical compatibility measures (CMs) rely on. We first re-establish, via oracle experiments, that the CM, not the solver, is the performance bottleneck: with a perfect CM, even a simple greedy solver achieves perfect reconstruction. This motivates STITCH, a self-supervised CM that sidesteps missing boundaries by comparing frozen ViT patch tokens from piece interiors via a lightweight cross-attention head (~249K trainable parameters), trained without manual annotation. Plugged into unchanged compatibility-based solvers, STITCH reaches .992/.954 Abs/Perf on JPwLEG-5 (+.629 Perf over the best prior method), nearly saturating the benchmark. On a new ImageNet 7x7 benchmark, it outperforms all alternatives, including inpaint-then-match pipelines built on modern generative models. Importantly, performance remains robust across erosion levels and generalizes to puzzles with over 40% missing pieces without retraining. Trained on the recent GAP datasets of irregular, heavily eroded fragments, the same CM recipe enables compatibility-based solvers to surpass recent learning-based solvers on GAP-3 and GAP-5. Since this leap in performance is achieved by a pluggable CM module, the takeaway is clear: fix the match, not the solver.