ByteAction: Byte-space Action Recognition Foundation Model
2026-08-24 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors designed a model called ByteAction to recognize human actions directly from compressed image data, without needing to decode the full images. Their approach works even when the data is corrupted, making it useful for privacy and reliability. They introduced new techniques, like Bitstream Pattern Augmentation to teach the model to handle missing parts, and Corruption Consistency Training to keep predictions stable under different corruption levels. Their tests showed ByteAction is robust against corrupted data while performing well on clean data too.
Byte-space Action RecognitionCompressed image bitstreamBitstream Pattern AugmentationCorruption Consistency TrainingHuman action recognitionPixel decodingByteFormer backboneBidirectional KL divergenceStanford40 datasetPASCAL VOC 2012 Action
Authors
Fangcheng Li, Zhen Yu, Kejun Wu, Qiong Liu, You Yang
Abstract
Byte-space Action Recognition (BAR) aims to recognize human actions directly from compressed image bitstreams without any pixel decoding. By operating entirely in byte space, BAR is inherently independent of file integrity and pixel-level reconstruction, making it naturally applicable to privacy-sensitive scenarios and robust against bitstream corruption. In this paper, we propose ByteAction, a BAR foundation model that achieves accurate action recognition on corrupted image bitstreams. ByteAction follows a dual-view byte-level recognition framework. It constructs weakly and strongly corrupted bitstream views, which are augmented by Bitstream Pattern Augmentation (BPA) and encoded with a shared ByteFormer backbone. The model is optimized with both classification and corruption consistency objectives. Specifically, we propose Bitstream Pattern Augmentation (BPA), which reshapes one-dimensional byte sequences into two-dimensional byte matrix and applies region-level erasure to encourage the model to learn robust cross-region byte dependencies. We further propose a Corruption Consistency Training strategy that constrains the model to maintain stable predictions across different corruption severities through bidirectional KL divergence. Experiments on the image bitstream from Stanford40, PPMI, and PASCAL VOC 2012 Action demonstrate that ByteAction achieves state-of-the-art corruption robustness across all scenarios while maintaining competitive intact bitstream performance.