GenStream cuts video bandwidth by recreating humans from metadata
GenStream: Semantic Streaming Framework for Generative Reconstruction of Human-centric Media
Computer Vision and Pattern RecognitionArtificial IntelligenceMultimedia
Summary
Streaming videos usually send all the pixels in each frame equally, which wastes data when only a few people move and the background stays the same. The authors introduce GenStream, which sends only the important scene details like human poses, camera angles, and a fixed background model. Then, the viewer’s computer uses these details to generate realistic images of people and place them inside the background. This method can shrink the data needed for video by over 99.9%, but it requires more computing power on the viewer’s side.
What this means in practice
- •For live sports broadcasters: Deliver figure skating and other human-focused sports video streams with drastically reduced bandwidth requirements by transmitting pose and background data instead of full video frames.$Commercial implications: Enables broadcasters to offer high-quality sports streaming at a fraction of the current data cost by selling efficient delivery solutions.
- •For virtual reality developers: Use structured metadata to recreate realistic human avatars in fixed 3D scenes for immersive VR applications with minimal streaming data.
Tested on one dataset.
Authors
Emanuele Artioli, Daniele Lorenzi, Shivi Vats, Farzad Tashtarian, Christian Timmerer
Abstract
Video streaming dominates global internet traffic, yet conventional pipelines remain inefficient for structured, human-centric content such as sports, performance, or interactive media. Standard codecs re-encode entire frames, foreground and background alike, treating all pixels uniformly and ignoring the semantic structure of the scene. This leads to significant bandwidth waste, particularly in scenarios where backgrounds are static and motion is constrained to a few salient actors. We introduce GenStream, a semantic streaming framework that replaces dense video frames with compact, structured metadata. Instead of transmitting pixels, GenStream encodes each scene as a combination of skeletal keypoints, camera viewpoint parameters, and a static 3D background model. These elements are transmitted to the client, where a generative model reconstructs photorealistic human figures and composites them into the 3D scene from the original viewpoint. This paradigm enables extreme compression, achieving over 99.9% bandwidth reduction compared to HEVC for the continuous data stream. We partially validate GenStream on Olympic figure skating footage and demonstrate potential for high perceptual fidelity under minimal data. While acknowledging the significant computational costs shifted to the client and challenges in generalization, GenStream opens new directions in volumetric avatar synthesis, canonical 3D actor fusion across views, and personalized viewing experiences, laying the groundwork for scalable, intelligent streaming in the post-codec era.