Image based prototypes improve document classification accuracy moderately
ProtoSemImage: Image-Valued Prototypes with Deformable Row Alignment for Interpretable Document Classification
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Classifiers usually use vectors that are hard to understand as examples. This paper shows how using little images made from document text can represent categories more naturally and understandably. These image prototypes let the model compare documents spatially and explain where a document differs from its typical category. Their approach improved accuracy on a ten-category task compared to vector methods but also revealed areas where the matching method could be better. The authors provide both the successes and failures to highlight insights about explainable document classification.
What this means in practice
- •For document processing engineers: Build document classifiers that identify category-specific features spatially to aid explainability and error analysis.
- •For digital humanities developers: Use image prototypes to visually explore linguistic and structural document features for textual analysis projects.
Authors
Mohammad Zare, Pirooz Shamsinejadbabaki
Abstract
Prototypes in classification models are almost always vectors, and a vector has no readable form. This paper asks what happens when a prototype is an image. Documents give the question a natural form, because a document can be rendered as a multi-channel image in which every token becomes a pixel, so a class representative can take the same shape and the same channel semantics as the inputs it stands for. ProtoSemImage represents each class by one or more visual archetypes: prototype images in a four-channel HSV space whose channels carry named linguistic factors. A Skip-Gram objective learns that color space end to end through a four-dimensional bottleneck, discourse boundary rows become differentiable typed difference rows, and classification reduces to 2D visual template matching: a deformable row alignment between a document image and the archetype bank, in the spirit of dynamic time warping. Because the match is a spatial pattern comparison rather than a linear readout, the model reports where an input departs from its archetype and along which channel, and a generative head decodes each archetype back into text. The image representation works: it beats an otherwise identical model with vector prototypes in all three paired seeds, by between 4.3 and 11.8 points on a ten-class task. The distance-based matching does not. A diagnostic that keeps the representation fixed and swaps only the classifier recovers the sequence baselines, which locates a 20.6-point shortfall in the matching rather than in the color compression, and a benchmark built so that a pair of documents shares a bag of words and differs only in arrangement confirms the layout-preservation it was designed for. We report both directions, because for a representation whose whole purpose is inspect ability, the failure modes are as informative as the gains.