Interactive audiovisual journey reshapes video and sound with ai
Passing: An Endless Journey through Reconstructed Spacetime with AI-Generated Sound
SoundArtificial Intelligence
Summary
This work creates an interactive art piece that transforms a single video of a train ride into an endlessly flowing and changing visual experience. Instead of playing the video normally, the system rewrites space and time in the footage to create unusual motion and perspective effects. It detects when someone is watching and uses that presence to change how the video flows. An AI then listens to these altered visuals and creates matching sounds, not to reproduce reality but to imagine what such a changed world could sound like. The artists, AI, and audience all share control over how the piece unfolds.
What this means in practice
- •For media artists: Create immersive installations where viewer presence dynamically alters video and sound in real time based on reconstructed spatial-temporal content.
- •For game designers: Integrate AI-driven audiovisual transformations that respond to player presence for evolving, non-linear environmental storytelling.
Authors
Akira Takahashi, Chihiro Nagashima, Zhi Zhong, Shusuke Takahashi, Yuki Mitsufuji
Abstract
This paper introduces Passing, an interactive audiovisual installation that generates an endless journey from a single continuous monorail-window recording by reconstructing it as a spatiotemporal volume. Rather than replaying the footage linearly, the work resamples its spatial and temporal structure along nonlinear trajectories, producing a continuously passing landscape whose depth, speed, and temporal order become unstable. A camera-based viewer-presence detection system estimates whether a viewer is present in the viewing zone and uses this presence state to influence transitions among rendered video sequences. The resulting video stream is fed into SpecMaskFoley, a real-time video-to-audio synthesis model that generates a synchronized soundscape for the reconfigured image. The model is not used to reconstruct an objectively correct soundtrack, but functions as a speculative listener, proposing a possible auditory interpretation of a world whose conventional spatial and temporal premises have been disrupted. Passing distributes creative agency across the artist, who defines the rules of spacetime reconstruction; the AI model, which interprets the emergent visual flow as sound; and the audience, whose embodied presence influences the audiovisual trajectory. Through this structure, the work investigates how authorship and listening may be negotiated among human intention, machine inference, and audience interpretation. Artwork page: https://ryufurusawa.com/passing