Sora100K dataset captures video creation as step-by-step editing process
From Video Clips to Creation Trajectory: Sora100K for AI-Native Video Creation
Computer Vision and Pattern Recognition
Summary
Creating videos with AI usually means making or editing single clips without connecting these steps. The authors introduce Sora100K, a large dataset that shows how videos are made through multiple steps, from text instructions to several rounds of editing. This dataset helps AI models learn the full video creation workflow, not just isolated tasks. Tests show improvements in video quality and consistency but also reveal that following multiple editing steps remains hard for AI.
What this means in practice
- •For video production teams: Use Sora100K to train AI tools that support multi-step video creation blending generation and editing.
- •For media content platforms: Improve automated video editing features by integrating models trained on structured video editing sequences from Sora100K.
Authors
Sicong Yang, Ruihuan Yang, Jian Lu, Jianfei Yuan, Xiaodong Cun, Xiuli Bi
Abstract
AI-Native video creation is shifting from isolated video clips toward iterative video creation workflows. However, existing datasets remain largely video clips, representing video generation and editing as separate tasks rather than connected stages of a video creation workflow. In this paper, we introduce Sora100K, a dataset that represents the AI-Native video creation workflow as a structured video creation trajectory. Specifically, we first identify video creation trajectories and decompose them into three subsets according to their structural roles: text-to-video generation records as roots, single-turn video editing records as editing edges, and multi-turn video editing records as complete trajectories. Then, we use a VLM to assign semantic annotations for generation roots and editing-operation annotations for editing edges. A strict construction pipeline further reconstructs source-to-edit lineage, editing order, and intermediate video states while ensuring data quality. Finally, we perform lightweight adaptation on LTX-2 models to assess the supervision value of Sora100K. The results show improvements in visual quality, multi-shot generation, and cross-shot consistency, while successive-turn evaluation reveals that following multi-turn editing instructions remains challenging. Sora100K establishes a new data foundation for AI-Native video creation beyond isolated video clips and toward structured video creation trajectory. The dataset and supplementary materials are publicly available at https://huggingface.co/datasets/ysicong/Sora100K.