Benchmark measures story continuity across text image and speech
What Comes Next? Omni-StoryBench for Evaluating Story-Grounded Omnimodal Generation
Computer Vision and Pattern RecognitionMultimedia
Summary
It is hard for computers to create stories that make sense using words, pictures, and sounds all together. The authors made a test called Omni-StoryBench that checks if a computer can continue a story in all three ways at once. They used real children’s books and tested many computer setups, finding that some methods worked better for keeping the story consistent. The biggest difficulty was making pictures that match the story well. This benchmark helps measure how well computers keep stories coherent across different types of media.
What this means in practice
- •For multimedia content developers: Evaluate tools that generate consistent stories across images, narration, and speech to ensure cohesive multimedia storytelling.
- •For interactive media producers: Test and improve story-based applications combining text, visuals, and dialogue using a standardized benchmark from children's book transitions.
Authors
Sieun Hyeon, Yejoon Lee, Mintaek Lim, Woojin Kim, Jaeik Kim, Jaeyoung Do
Abstract
Omnimodal evaluation should go beyond independent text, image, and speech production: individually plausible outputs may not express a coherent shared event. We introduce Omni-StoryBench, a story-grounded omnimodal benchmark evaluating whether models can coherently continue stories across image, narration, and speech. Each instance provides a current storybook page and structured next-page conditions, requiring models to generate the next illustration, narration, and spoken character utterance. Omni-StoryBench contains 900 rigorously validated story transitions from openly licensed children's books, with ground-truth next-page references and speech metadata. We evaluate systems with modality-specific metrics and consistency-centered LLM-as-a-judge rubrics for context preservation, condition following, reference consistency, and cross-modal coherence. Across 32 baseline configurations spanning orchestration, semi-orchestration, and native any-to-any paradigms, we find orchestration with strong VLM planning most reliable, while current native omnimodal models often struggle with output completeness and controllability. Our analysis shows text-side performance is associated with image and speech quality, but image generation and visual continuity form the clearest observed bottleneck among the evaluated configurations. These results position Omni-StoryBench as a system-level benchmark measuring coherent omnimodal generation beyond isolated modality quality.