Survey Maps Convergence of Video Understanding, Generation, and World Models
A comprehensive survey analyzes 600+ papers across video understanding, generation, unified architectures, and world models, establishing a framework for physical state tracking.
A new 102-page survey paper synthesizes the rapid convergence across video understanding, video generation, unified models, and video world models. Published under the title "Reading, Writing, Sharing, and Interacting with the World through Video," the study analyzes over 600 references and maps 180 milestone systems developed between 2017 and 2026. The authors propose a unified perspective: a video sequence is a partial, lossy, and time-ordered trace of an underlying world that continues to evolve whether or not it is captured on camera.
The Four Pillars of Video Intelligence
The framework organizes the field into four fundamental operations connecting observable video clips with underlying physical world states:
- Reading (Video to World): Encompasses video comprehension architectures like Video-LLaVA and Qwen3-VL, which extends context lengths up to 256K tokens. The survey notes that the primary bottleneck in video understanding is keeping track of hidden physical states and object permanence, rather than expanding token context limits.
- Writing (World to Video): Focuses on video generation systems such as Sora, Wan, HunyuanVideo, Movie Gen, and Seedance 2.0. The authors highlight that visual fidelity and stateful control are distinct axes, recommending that controllability be evaluated by comparing matched rollouts rather than relying on single visual quality scores.
- Sharing (Video-World Coupling): Examines unified architectures including Emu3, Show-o, and UniVideo that link reading and writing through shared parameters, discrete or continuous latents, or interfaces. While parameter sharing is increasingly common, proven bidirectional synergy remains rare in video domains.
- Interacting (World Transitions): Maps predictive models using a 2x3 matrix across two independent axes: prediction space (rendering raw observations versus predicting latent states) and transition interface (passive, action-conditioned, or interactive). Systems like Genie, GameNGen, Oasis, and Genie 3 offer interactive frame generation, whereas systems like V-JEPA 2-AC and DINO-WM project actions into feature spaces.
Core Technical Bottlenecks and Challenges
The survey identifies seven open challenges across video world modeling:
- Long-horizon world consistency: Models struggle to maintain accurate representations of entities that move off-screen for extended durations.
- Physical and causal grounding: Systems frequently fail to isolate physical laws, such as gravity, without altering un-targeted elements of the scene.
- Measurable synergy in unified models: Controlled experiments are needed to confirm whether shared reading and writing representations produce genuine performance gains over specialized models.
- Standardized evaluation: Probing latent transition models alongside frame generators requires unified decision-based testing frameworks.
- Scaling control from unlabeled video: Learning action controls without explicit annotations often leads models to confound camera movement with environmental changes.
- Generalization across scenes and embodiments: Transferring learned dynamics across new physical environments or robotic hardware remains an open problem.
- Stochastic worlds and rare events: High visual fidelity often masks a model's inability to accurately assess the statistical probabilities of rare occurrences like physical collisions.
What it means for developers
For engineers building AI systems, this survey provides a practical framework for selecting architectures and designing evaluation pipelines. Rather than focusing strictly on context window length or raw perceptual output, developers working on video tasks must evaluate how effectively models track hidden state, preserve spatial geometry, and maintain physical constraints across time.
As video understanding, text-to-video generation, and interactive world modeling continue to merge, engineers will frequently need to test and integrate different model paradigms. Developers can try top AI models cheaply through one API at https://apixoai.online to evaluate diverse architectures without setting up separate infrastructure for each provider.
Understanding the structural trade-offs identified in the survey—such as whether a model operates on interactive observation rollouts or latent feature predictions—helps developers select the appropriate tools for autonomous agents, robotics control, simulation environments, and video processing software.
Source: Reading, Writing, Sharing, and Interacting with the World through Video — Towards AI. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
