diff --git a/research/_index.md b/research/_index.md index 6d115ec..5797967 100644 --- a/research/_index.md +++ b/research/_index.md @@ -11,7 +11,7 @@ description: "研究论文调研综述目录，涵盖世界模型、空间感知 本索引汇总了 **WorldModel 项目**下所有论文调研综述文档。项目围绕「可探索生成式 3D 世界」、「空间感知与记忆」、「物理世界理解」、「传感器数据管道」以及「具体子项目（CrowdRoom / PRISM）」五大方向展开研究，旨在为 PRISM 2.0、CrowdRoom MVP 等工程系统提供学术文献支撑与技术选型依据。 -> **更新时间**：2026-05-20 | **文档数量**：8 篇 | **覆盖论文**：~215 篇 +> **更新时间**：2026-05-20 | **文档数量**：9 篇 | **覆盖论文**：~255 篇 --- @@ -33,6 +33,7 @@ description: "研究论文调研综述目录，涵盖世界模型、空间感知 |------|------|------|-----------| | [AI 世界模型技术综述](world-models/world_models_review/) | 2026-05-20 | `综述` `arXiv` `物理` `规划` | 系统梳理国际与中国在 AI 世界模型领域的发展，涵盖 JEPA、Sora、Dreamer、Open-Sora 等主流路线与开源生态。 | | [面向"理解"与"构建"物理世界的 AI 世界模型：技术综述](world-models/physics_world_models_review/) | 2026-05-20 | `机器人` `综述` `arXiv` `物理` | 从"理解物理世界"与"构建物理世界"两个维度，对比 JEPA、DiT/Sora、Dreamer 等技术路线，分析中国力量的突破与挑战。 | +| [世界模型 2025-2026 最新进展综述（arXiv + GitHub）](world-models/world_models_arxiv_2025_survey/) | 2026-05-20 | `arXiv` `GitHub` `综述` `2025-2026` | 通过 arXiv API + GitHub API 自动检索的 40 篇最新论文 + 31 个热门仓库，覆盖视频生成、自动驾驶、具身智能、强化学习、3D 场景五大主线。 | ### 1.2 前沿论文精读 @@ -87,7 +88,8 @@ description: "研究论文调研综述目录，涵盖世界模型、空间感知 | `物理` | [Lyra 2.0](world-models/lyra2_review/)、[世界模型综述](world-models/world_models_review/)、[物理世界模型综述](world-models/physics_world_models_review/)、[物理理解研究计划](plans/understanding_physics_research_plan/) | | `机器人` | [物理世界模型综述](world-models/physics_world_models_review/)、[物理理解研究计划](plans/understanding_physics_research_plan/)、[CrowdRoom](crowdroom/crowdroom_related_papers_2026/) | | `导航 / 空间记忆` | [人类空间记忆](spatial-memory/human_spatial_memory/) | -| `arXiv` | [Lyra 2.0](world-models/lyra2_review/)、[世界模型综述](world-models/world_models_review/)、[物理世界模型综述](world-models/physics_world_models_review/)、[ZED 2i 实时综述](spatial-memory/zed2i_arxiv_live_review/) | +| `arXiv` | [Lyra 2.0](world-models/lyra2_review/)、[世界模型综述](world-models/world_models_review/)、[物理世界模型综述](world-models/physics_world_models_review/)、[ZED 2i 实时综述](spatial-memory/zed2i_arxiv_live_review/)、[世界模型 2025-2026 综述](world-models/world_models_arxiv_2025_survey/) | +| `GitHub` | [世界模型 2025-2026 综述](world-models/world_models_arxiv_2025_survey/) | | `JEPA / Meta` | [Meta JEPA 系列调研](world-models/jepa/)、[世界模型综述](world-models/world_models_review/)、[物理世界模型综述](world-models/physics_world_models_review/) | --- @@ -113,6 +115,8 @@ description: "研究论文调研综述目录，涵盖世界模型、空间感知 | [`data/crowdroom_papers_raw.json`](data/crowdroom_papers_raw.json) | CrowdRoom 专属抓取结果（当前因 429 限速为空） | | [`data/lyra2_paper.txt`](data/lyra2_paper.txt) | Lyra 2.0 论文全文（arXiv:2604.13036） | | [`data/lyra2_paper.pdf`](data/lyra2_paper.pdf) | Lyra 2.0 论文 PDF | +| [`data/world_models_arxiv_2025.json`](data/world_models_arxiv_2025.json) | 世界模型 arXiv 检索结果（5 主题 × 8 篇 = 40 篇，2025-2026） | +| [`data/world_models_github_2025.json`](data/world_models_github_2025.json) | 世界模型 GitHub 检索结果（6 类查询 = 31 个独立仓库） | 详见 [`data/README.md`](data/README.md)。 @@ -122,12 +126,12 @@ description: "研究论文调研综述目录，涵盖世界模型、空间感知 | 主题分类 | 文档数 | 覆盖论文数（估计） | |---------|--------|-----------------| -| 世界模型（综述 + 精读） | 4 | ~75 | +| 世界模型（综述 + 精读） | 5 | ~115 | | 空间感知与记忆 | 1 | ~30 | | 物理理解 | 1 | ~20 | | 传感器与数据管道 | 2 | ~135 | | 项目相关（CrowdRoom） | 1 | 0（API 限流，骨架版） | -| **合计** | **8** | **~260** | +| **合计** | **9** | **~300** | --- diff --git a/research/data/world_models_arxiv_2025.json b/research/data/world_models_arxiv_2025.json new file mode 100644 index 0000000..480288b --- /dev/null +++ b/research/data/world_models_arxiv_2025.json @@ -0,0 +1,702 @@ +{ + "video_generation": [ + { + "title": "PhyWorld: Physics-Faithful World Model for Video Generation", + "authors": [ + "Pu Zhao", + "Juyi Lin", + "Timothy Rupprecht", + "Arash Akbari", + "Chence Yang" + ], + "summary": "World simulators can provide safe and scalable environments for training Physical AI systems before real-world deployment. Large video generation models are emerging as a promising basis for such simulators because they can generate diverse and realistic visual futures. However, using them as world simulators requires physically faithful video continuations, namely, generated videos that preserve the physical state implied by the conditioning input, and evolve in ways consistent with basic physi...", + "published": "2026-05-19", + "arxiv_id": "2605.19242", + "link": "https://arxiv.org/abs/2605.19242", + "categories": [ + "cs.CV", + "cs.AI", + "cs.ET", + "cs.LG", + "cs.MM" + ] + }, + { + "title": "ACWM-Phys: Investigating Generalized Physical Interaction in Action-Conditioned Video World Models", + "authors": [ + "Haotian Xue", + "Yipu Chen", + "Liqian Ma", + "Zelin Zhao", + "Lama Moukheiber" + ], + "summary": "Action-conditioned world models (ACWMs) have shown strong promise for video prediction and decision-making. However, existing benchmarks are largely restricted to egocentric navigation or narrow, task-specific robotics datasets, offering only limited coverage of the rich physical interactions required for generalized world understanding. We introduce ACWM-Phys, a new benchmark for evaluating action-conditioned prediction under diverse physical dynamics in a clean, controllable simulation environ...", + "published": "2026-05-09", + "arxiv_id": "2605.08567", + "link": "https://arxiv.org/abs/2605.08567", + "categories": [ + "cs.CV" + ] + }, + { + "title": "Video Generation Models as World Models: Efficient Paradigms, Architectures and Algorithms", + "authors": [ + "Muyang He", + "Hanzhong Guo", + "Junxiong Lin", + "Yizhou Yu" + ], + "summary": "The rapid evolution of video generation has enabled models to simulate complex physical dynamics and long-horizon causalities, positioning them as potential world simulators. However, a critical gap still remains between the theoretical capacity for world simulation and the heavy computational costs of spatiotemporal modeling. To address this, we comprehensively and systematically review video generation frameworks and techniques that consider efficiency as a crucial requirement for practical wo...", + "published": "2026-03-30", + "arxiv_id": "2603.28489", + "link": "https://arxiv.org/abs/2603.28489", + "categories": [ + "eess.IV", + "cs.CV" + ] + }, + { + "title": "Stereo World Model: Camera-Guided Stereo Video Generation", + "authors": [ + "Yang-Tian Sun", + "Zehuan Huang", + "Yifan Niu", + "Lin Ma", + "Yan-Pei Cao" + ], + "summary": "We present StereoWorld, a camera-conditioned stereo world model that jointly learns appearance and binocular geometry for end-to-end stereo video generation.Unlike monocular RGB or RGBD approaches, StereoWorld operates exclusively within the RGB modality, while simultaneously grounding geometry directly from disparity. To efficiently achieve consistent stereo generation, our approach introduces two key designs: (1) a unified camera-frame RoPE that augments latent tokens with camera-aware rotary ...", + "published": "2026-03-18", + "arxiv_id": "2603.17375", + "link": "https://arxiv.org/abs/2603.17375", + "categories": [ + "cs.CV" + ] + }, + { + "title": "SAW: Toward a Surgical Action World Model via Controllable and Scalable Video Generation", + "authors": [ + "Sampath Rapuri", + "Lalithkumar Seenivasan", + "Dominik Schneider", + "Roger Soberanis-Mukul", + "Yufan He" + ], + "summary": "A surgical world model capable of generating realistic surgical action videos with precise control over tool-tissue interactions can address fundamental challenges in surgical AI and simulation -- from data scarcity and rare event synthesis to bridging the sim-to-real gap for surgical automation. However, current video generation methods, the very core of such surgical world models, require expensive annotations or complex structured intermediates as conditioning signals at inference, limiting t...", + "published": "2026-03-13", + "arxiv_id": "2603.13024", + "link": "https://arxiv.org/abs/2603.13024", + "categories": [ + "cs.CV", + "cs.AI", + "cs.LG", + "eess.IV" + ] + }, + { + "title": "LiveWorld: Simulating Out-of-Sight Dynamics in Generative Video World Models", + "authors": [ + "Zicheng Duan", + "Jiatong Xia", + "Zeyu Zhang", + "Wenbo Zhang", + "Gengze Zhou" + ], + "summary": "Recent generative video world models aim to simulate visual environment evolution, allowing an observer to interactively explore the scene via camera control. However, they implicitly assume that the world only evolves within the observer's field of view. Once an object leaves the observer's view, its state is \"frozen\" in memory, and revisiting the same region later often fails to reflect events that should have occurred in the meantime. In this work, we identify and formalize this overlooked li...", + "published": "2026-03-07", + "arxiv_id": "2603.07145", + "link": "https://arxiv.org/abs/2603.07145", + "categories": [ + "cs.CV" + ] + }, + { + "title": "ShareVerse: Multi-Agent Consistent Video Generation for Shared World Modeling", + "authors": [ + "Jiayi Zhu", + "Jianing Zhang", + "Yiying Yang", + "Wei Cheng", + "Xiaoyun Yuan" + ], + "summary": "This paper presents ShareVerse, a video generation framework enabling multi-agent shared world modeling, addressing the gap in existing works that lack support for unified shared world construction with multi-agent interaction. ShareVerse leverages the generation capability of large video models and integrates three key innovations: 1) A dataset for large-scale multi-agent interactive world modeling is built on the CARLA simulation platform, featuring diverse scenes, weather conditions, and inte...", + "published": "2026-03-03", + "arxiv_id": "2603.02697", + "link": "https://arxiv.org/abs/2603.02697", + "categories": [ + "cs.CV", + "cs.AI" + ] + }, + { + "title": "DreamWorld: Unified World Modeling in Video Generation", + "authors": [ + "Boming Tan", + "Xiangdong Zhang", + "Ning Liao", + "Yuqing Zhang", + "Shaofeng Zhang" + ], + "summary": "Despite impressive progress in video generation, existing models remain limited to surface-level plausibility, lacking a coherent and unified understanding of the world. Prior approaches typically incorporate only a single form of world-related knowledge or rely on rigid alignment strategies to introduce additional knowledge. However, aligning the single world knowledge is insufficient to constitute a world model that requires jointly modeling multiple heterogeneous dimensions (e.g., physical co...", + "published": "2026-02-28", + "arxiv_id": "2603.00466", + "link": "https://arxiv.org/abs/2603.00466", + "categories": [ + "cs.CV" + ] + } + ], + "autonomous_driving": [ + { + "title": "HEAT: Heterogeneous End-to-End Autonomous Driving via Trajectory-Guided World Models", + "authors": [ + "Hoonhee Cho", + "Giwon Lee", + "Jae-Young Kang", + "Hyemin Yang", + "Heejun Park" + ], + "summary": "End-to-end autonomous driving has emerged as a compelling alternative to traditional modular pipelines by directly mapping raw sensor data to driving actions. While recent approaches achieve strong performance on single-domain datasets, their performance degrades significantly when trained jointly across multiple heterogeneous domains. In practice, however, autonomous systems must operate across diverse environments with heterogeneous distributions, including different cities, sensor configurati...", + "published": "2026-05-19", + "arxiv_id": "2605.19631", + "link": "https://arxiv.org/abs/2605.19631", + "categories": [ + "cs.RO", + "cs.CV" + ] + }, + { + "title": "Xiaomi EV World Model: A Joint World Model Integrating Reconstruction and Generation for Autonomous Driving", + "authors": [ + "Lijun Zhou", + "Hongcheng Luo", + "Zhenxin Zhu", + "Cheng Chi", + "Mingfei Tu" + ], + "summary": "This report presents a unified technical system addressing the two core capabilities of world models for autonomous driving: world representation and world generation. For world representation, we propose WorldRec, a feed-forward reconstruction architecture driven by sparse scene queries. WorldRec initializes structured queries in 3D space, leveraging them to aggregate cross-view, cross-temporal features, thereby naturally enforcing spatial consistency across frames and yielding compact yet high...", + "published": "2026-05-18", + "arxiv_id": "2605.18137", + "link": "https://arxiv.org/abs/2605.18137", + "categories": [ + "cs.CV" + ] + }, + { + "title": "DeepSight: Long-Horizon World Modeling via Latent States Prediction for End-to-End Autonomous Driving", + "authors": [ + "Lingjun Zhang", + "Changjie Wu", + "Linzhe Shi", + "Jiangyang Li", + "Jiaxin Liu" + ], + "summary": "End-to-end autonomous driving systems are increasingly integrating Vision-Language Model (VLM) architectures, incorporating text reasoning or visual reasoning to enhance the robustness and accuracy of driving decisions. However, the reasoning mechanisms employed in most methods are direct adaptations from general domains, lacking in-depth exploration tailored to autonomous driving scenarios, particularly within visual reasoning modules. In this paper, we propose a driving world model that perfor...", + "published": "2026-05-11", + "arxiv_id": "2605.10564", + "link": "https://arxiv.org/abs/2605.10564", + "categories": [ + "cs.CV", + "cs.RO" + ] + }, + { + "title": "CoWorld-VLA: Thinking in a Multi-Expert World Model for Autonomous Driving", + "authors": [ + "Minqing Huang", + "Yujiao Xiang", + "Zihan Liang", + "Jiajie Huang", + "Jingqi Wang" + ], + "summary": "Vision-Language-Action (VLA) models have emerged as a promising paradigm for end-to-end autonomous driving. However, existing reasoning mechanisms still struggle to provide planning-oriented intermediate representations: textual Chain-of-Thought (CoT) fails to preserve continuous spatiotemporal structure, while latent world reasoning remains difficult to use as a direct condition for action generation. In this paper, we propose CoWorld-VLA, a multi-expert world reasoning framework for autonomous...", + "published": "2026-05-11", + "arxiv_id": "2605.10426", + "link": "https://arxiv.org/abs/2605.10426", + "categories": [ + "cs.CV", + "cs.AI" + ] + }, + { + "title": "DriveFuture: Future-Aware Latent World Models for Autonomous Driving", + "authors": [ + "Yufeng Hong", + "Xiaotian Zhou", + "Yingyan Li", + "Xiangpo Zhou", + "Lin Liu" + ], + "summary": "Existing latent world models for autonomous driving have opened a promising path toward future-aware driving intelligence. However, they typically treat future latent states as prediction targets or auxiliary signals, rather than directly conditioning trajectory planning. This can entangle current and future features in latent space. In this work, we propose DriveFuture, a future-aware latent world modeling framework for autonomous driving that explicitly learns planning-oriented foresight by co...", + "published": "2026-05-10", + "arxiv_id": "2605.09701", + "link": "https://arxiv.org/abs/2605.09701", + "categories": [ + "cs.CV" + ] + }, + { + "title": "Learning Vision-Language-Action World Models for Autonomous Driving", + "authors": [ + "Guoqing Wang", + "Pin Tang", + "Xiangxuan Ren", + "Guodongfang Zhao", + "Bailan Feng" + ], + "summary": "Vision-Language-Action (VLA) models have recently achieved notable progress in end-to-end autonomous driving by integrating perception, reasoning, and control within a unified multimodal framework. However, they often lack explicit modeling of temporal dynamics and global world consistency, which limits their foresight and safety. In contrast, world models can simulate plausible future scenes but generally struggle to reason about or evaluate the imagined future they generate. In this work, we p...", + "published": "2026-04-10", + "arxiv_id": "2604.09059", + "link": "https://arxiv.org/abs/2604.09059", + "categories": [ + "cs.CV", + "cs.AI" + ] + }, + { + "title": "ExploreVLA: Dense World Modeling and Exploration for End-to-End Autonomous Driving", + "authors": [ + "Zihao Sheng", + "Xin Ye", + "Jingru Luo", + "Sikai Chen", + "Liu Ren" + ], + "summary": "End-to-end autonomous driving models based on Vision-Language-Action (VLA) architectures have shown promising results by learning driving policies through behavior cloning on expert demonstrations. However, imitation learning inherently limits the model to replicating observed behaviors without exploring diverse driving strategies, leaving it brittle in novel or out-of-distribution scenarios. Reinforcement learning (RL) offers a natural remedy by enabling policy exploration beyond the expert dis...", + "published": "2026-04-03", + "arxiv_id": "2604.02714", + "link": "https://arxiv.org/abs/2604.02714", + "categories": [ + "cs.CV" + ] + }, + { + "title": "DLWM: Dual Latent World Models enable Holistic Gaussian-centric Pre-training in Autonomous Driving", + "authors": [ + "Yiyao Zhu", + "Ying Xue", + "Haiming Zhang", + "Guangfeng Jiang", + "Wending Zhou" + ], + "summary": "Vision-based autonomous driving has gained much attention due to its low costs and excellent performance. Compared with dense BEV (Bird's Eye View) or sparse query models, Gaussian-centric method is a comprehensive yet sparse representation by describing scene with 3D semantic Gaussians. In this paper, we introduce DLWM, a novel paradigm with Dual Latent World Models specifically designed to enable holistic gaussian-centric pre-training in autonomous driving using two stages. In the first stage,...", + "published": "2026-04-01", + "arxiv_id": "2604.00969", + "link": "https://arxiv.org/abs/2604.00969", + "categories": [ + "cs.CV" + ] + } + ], + "embodied_ai": [ + { + "title": "World-Ego Modeling for Long-Horizon Evolution in Hybrid Embodied Tasks", + "authors": [ + "Zuyao Lin", + "Jianhui Zhang", + "Peidong Jia", + "Xiaoguang Zhao", + "Shanghang Zhang" + ], + "summary": "World models are widely explored in embodied intelligence, yet they typically predict distinct evolutions of the world and the ego within a single stream, where the world captures persistent instruction-agnostic scene regularities and the ego captures robot-centric instruction-conditioned dynamics. This world-ego entanglement leads to a degradation in long-horizon embodied scenarios, particularly in hybrid tasks with interleaved navigation and manipulation behaviors. In this paper, we introduce ...", + "published": "2026-05-19", + "arxiv_id": "2605.19957", + "link": "https://arxiv.org/abs/2605.19957", + "categories": [ + "cs.CV", + "cs.AI", + "cs.RO" + ] + }, + { + "title": "SWEET: Sparse World Modeling with Image Editing for Embodied Task Execution", + "authors": [ + "Yiren Song", + "Yihan Wang", + "Xiyao Deng", + "Zhuoran Yan", + "Mike Zheng Shou" + ], + "summary": "Visual prediction has emerged as a promising paradigm for embodied control, where future observations are generated and then translated into actions. However, dense video generation is computationally expensive and often unnecessary for many manipulation tasks, whose progress can be summarized by a small number of task-relevant visual states. In this work, we study whether image editing models can serve as sparse visual world models for robot manipulation by predicting task-level future states w...", + "published": "2026-05-19", + "arxiv_id": "2605.19319", + "link": "https://arxiv.org/abs/2605.19319", + "categories": [ + "cs.CV" + ] + }, + { + "title": "Key-Gram: Extensible World Knowledge for Embodied Manipulation", + "authors": [ + "Jingjing Fan", + "Siyuan Li", + "Botao Ren", + "Zhidong Deng" + ], + "summary": "Embodied control increasingly requires models to follow compositional language instructions while reasoning over dynamic visual states. However, current vision-language-action policies and world-action models often couple linguistic knowledge with visual computation in a shared backbone or conditioning pathway, leading to modality competition and making knowledge extension dependent on backbone updates. In this paper, we introduce Key-Gram, a conditional-memory framework that separates language-...", + "published": "2026-05-18", + "arxiv_id": "2605.18556", + "link": "https://arxiv.org/abs/2605.18556", + "categories": [ + "cs.RO", + "cs.AI" + ] + }, + { + "title": "WorldArena 2.0: Extending Embodied World Model Benchmarking on Modality, Functionality and Platform", + "authors": [ + "Yu Shang", + "Yinzhou Tang", + "Yiding Ma", + "Zhuohang Li", + "Lei Jin" + ], + "summary": "World models have emerged as a central paradigm for embodied intelligence, enabling agents to predict action-conditioned future and reason about environmental dynamics. However, existing embodied world model benchmarks are still largely confined to vision-only prediction, offline embodied applications, and simulator-based evaluation, making them insufficient for assessing increasingly comprehensive world models. In this work, we introduce WorldArena 2.0, an expanded benchmark that systematically...", + "published": "2026-05-18", + "arxiv_id": "2605.17912", + "link": "https://arxiv.org/abs/2605.17912", + "categories": [ + "cs.RO", + "cs.CV" + ] + }, + { + "title": "RoboFlow4D: A Lightweight Flow World Model Toward Real-Time Flow-Guided Robotic Manipulation", + "authors": [ + "Sixu Lin", + "Junliang Chen", + "Huaiyuan Xu", + "Zhuohao Li", + "Guangming Wang" + ], + "summary": "Planning and acting in 3D environments is a fundamental capability for robotic manipulation in the real world. Although prior work has explored predictive flow planners to guide 3D manipulation, existing approaches often rely on modular pipelines stacking multiple submodels, resulting in high computational overhead and limited real-time performance. To address these challenges, we introduce RoboFlow4D, a lightweight flow world model that unifies perception and planning by estimating temporal mot...", + "published": "2026-05-17", + "arxiv_id": "2605.17522", + "link": "https://arxiv.org/abs/2605.17522", + "categories": [ + "cs.RO" + ] + }, + { + "title": "DeTrack: A Benchmark and Altitude-Aware Dual World Model for Drone-embodied Tracking", + "authors": [ + "Guyue Hu", + "Haoming Liu", + "Siyuan Song", + "Chenglong Li", + "Feng Chen" + ], + "summary": "Aerial object tracking has broad applications in public safety, emergency rescue, wildlife monitoring, and related fields. However, existing aerial tracking benchmarks are mainly based on passive 2D video sequences captured from fixed camera locations or predefined flight paths, where drones are treated as passive cameras rather than embodied agents that actively perceive, interact, and control their motion in dynamic 3D scenes. In this paper, we define a new drone-embodied tracking task, termed...", + "published": "2026-05-17", + "arxiv_id": "2605.17451", + "link": "https://arxiv.org/abs/2605.17451", + "categories": [ + "cs.CV" + ] + }, + { + "title": "Embodied Multi-Agent Coordination by Aligning World Models Through Dialogue", + "authors": [ + "Vardhan Dongre", + "Dilek Hakkani-Tür" + ], + "summary": "Effective collaboration between embodied agents requires more than acting in a shared environment; it demands communication grounded in each agent's evolving understanding of the world. When agents can only partially observe their surroundings, coordination without communication is provably hard, but communication can, in principle, bridge this gap by allowing agents to share observations and align their world models. In this work, we examine whether LLM-based embodied agents actually realize th...", + "published": "2026-05-13", + "arxiv_id": "2605.12920", + "link": "https://arxiv.org/abs/2605.12920", + "categories": [ + "cs.MA", + "cs.AI", + "cs.CL" + ] + }, + { + "title": "OrbiSim: World Models as Differentiable Physics Engines for Embodied Intelligence", + "authors": [ + "Jiajian Li", + "Jingyuan Huang", + "Junru Gong", + "Qi Wang", + "Xiaokang Yang" + ], + "summary": "We present OrbiSim, a novel robotic simulation paradigm that redefines world models as a fully differentiable physics engine for embodied intelligence. Unlike prior world models that focus on unconstrained imagination in latent or visual domains, OrbiSim establishes a unified, physically-grounded pathway that bridges structured scene assets, neural dynamics, and downstream reinforcement learning. By enabling end-to-end differentiability throughout the entire simulation loop -- spanning from expl...", + "published": "2026-05-12", + "arxiv_id": "2605.16395", + "link": "https://arxiv.org/abs/2605.16395", + "categories": [ + "cs.RO", + "cs.LG" + ] + } + ], + "reinforcement_learning": [ + { + "title": "JEDI: Joint Embedding Diffusion World Model for Online Model-Based Reinforcement Learning", + "authors": [ + "Jing Yu Lim", + "Rushi Shah", + "Zarif Ikram", + "Samson Yu", + "Haozhe Ma" + ], + "summary": "Diffusion world models have recently become competitive for online model-based reinforcement learning, but current approaches expose a tension: pixel diffusion is effective but computationally expensive while the latest latent diffusion approach improves efficiency yet performs subpar. The latter also relies on separately trained latents rather than the end-to-end world-model objectives that have driven much of modern MBRL progress. In particular, JEPA-style predictive representation learning ha...", + "published": "2026-05-13", + "arxiv_id": "2605.13013", + "link": "https://arxiv.org/abs/2605.13013", + "categories": [ + "cs.LG" + ] + }, + { + "title": "WOMBET: World Model-based Experience Transfer for Robust and Sample-efficient Reinforcement Learning", + "authors": [ + "Mintae Kim", + "Koushil Sreenath" + ], + "summary": "Reinforcement learning (RL) in robotics is often limited by the cost and risk of data collection, motivating experience transfer from a source task to a target task. Offline-to-online RL leverages prior data but typically assumes a given fixed dataset and does not address how to generate reliable data for transfer. We propose \\textit{World Model-based Experience Transfer} (WOMBET), a framework that jointly generates and utilizes prior data. WOMBET learns a world model in the source task and gene...", + "published": "2026-04-10", + "arxiv_id": "2604.08958", + "link": "https://arxiv.org/abs/2604.08958", + "categories": [ + "cs.LG", + "cs.AI", + "cs.RO" + ] + }, + { + "title": "Persistent Robot World Models: Stabilizing Multi-Step Rollouts via Reinforcement Learning", + "authors": [ + "Jai Bardhan", + "Patrik Drozdik", + "Josef Sivic", + "Vladimir Petrik" + ], + "summary": "Action-conditioned robot world models generate future video frames of the manipulated scene given a robot action sequence, offering a promising alternative for simulating tasks that are difficult to model with traditional physics engines. However, these models are optimized for short-term prediction and break down when deployed autoregressively: each predicted clip feeds back as context for the next, causing errors to compound and visual quality to rapidly degrade. We address this through the fo...", + "published": "2026-03-26", + "arxiv_id": "2603.25685", + "link": "https://arxiv.org/abs/2603.25685", + "categories": [ + "cs.RO", + "cs.CV" + ] + }, + { + "title": "DreamerAD: Efficient Reinforcement Learning via Latent World Model for Autonomous Driving", + "authors": [ + "Pengxuan Yang", + "Yupeng Zheng", + "Deheng Qian", + "Zebin Xing", + "Qichao Zhang" + ], + "summary": "We introduce DreamerAD, the first latent world model framework that enables efficient reinforcement learning for autonomous driving by compressing diffusion sampling from 100 steps to 1 - achieving 80x speedup while maintaining visual interpretability. Training RL policies on real-world driving data incurs prohibitive costs and safety risks. While existing pixel-level diffusion world models enable safe imagination-based training, they suffer from multi-step diffusion inference latency (2s/frame)...", + "published": "2026-03-25", + "arxiv_id": "2603.24587", + "link": "https://arxiv.org/abs/2603.24587", + "categories": [ + "cs.LG", + "cs.RO" + ] + }, + { + "title": "Model Predictive Control with Differentiable World Models for Offline Reinforcement Learning", + "authors": [ + "Rohan Deb", + "Stephen J. Wright", + "Arindam Banerjee" + ], + "summary": "Offline Reinforcement Learning (RL) aims to learn optimal policies from fixed offline datasets, without further interactions with the environment. Such methods train an offline policy (or value function), and apply it at inference time without further refinement. We introduce an inference time adaptation framework inspired by model predictive control (MPC) that utilizes a pretrained policy along with a learned world model of state transitions and rewards. While existing world model and diffusion...", + "published": "2026-03-23", + "arxiv_id": "2603.22430", + "link": "https://arxiv.org/abs/2603.22430", + "categories": [ + "cs.LG" + ] + }, + { + "title": "Towards Practical World Model-based Reinforcement Learning for Vision-Language-Action Models", + "authors": [ + "Zhilong Zhang", + "Haoxiang Ren", + "Yihao Sun", + "Yifei Sheng", + "Haonan Wang" + ], + "summary": "Vision-Language-Action (VLA) models show strong generalization for robotic control, but finetuning them with reinforcement learning (RL) is constrained by the high cost and safety risks of real-world interaction. Training VLA models in interactive world models avoids these issues but introduces several challenges, including pixel-level world modeling, multi-view consistency, and compounding errors under sparse rewards. Building on recent advances across large multimodal models and model-based RL...", + "published": "2026-03-21", + "arxiv_id": "2603.20607", + "link": "https://arxiv.org/abs/2603.20607", + "categories": [ + "cs.RO", + "cs.LG" + ] + }, + { + "title": "AcceRL: A Distributed Asynchronous Reinforcement Learning and World Model Framework for Vision-Language-Action Models", + "authors": [ + "Chengxuan Lu", + "Shukuan Wang", + "Yanjie Li", + "Wei Liu", + "Shiji Jin" + ], + "summary": "Reinforcement learning (RL) for large-scale Vision-Language-Action (VLA) models faces significant challenges in computational efficiency and data acquisition. We propose AcceRL, a fully asynchronous and decoupled RL framework designed to eliminate synchronization barriers by physically isolating training, inference, and rollouts. Crucially, AcceRL is the first to integrate a plug-and-play, trainable world model into a distributed asynchronous RL pipeline to generate virtual experiences. Experime...", + "published": "2026-03-19", + "arxiv_id": "2603.18464", + "link": "https://arxiv.org/abs/2603.18464", + "categories": [ + "cs.LG" + ] + }, + { + "title": "Self-adapting Robotic Agents through Online Continual Reinforcement Learning with World Model Feedback", + "authors": [ + "Fabian Domberg", + "Georg Schildbach" + ], + "summary": "As learning-based robotic controllers are typically trained offline and deployed with fixed parameters, their ability to cope with unforeseen changes during operation is limited. Biologically inspired, this work presents a framework for online Continual Reinforcement Learning that enables automated adaptation during deployment. Building on DreamerV3, a model-based Reinforcement Learning algorithm, the proposed method leverages world model prediction residuals to detect out-of-distribution events...", + "published": "2026-03-04", + "arxiv_id": "2603.04029", + "link": "https://arxiv.org/abs/2603.04029", + "categories": [ + "cs.RO", + "cs.AI" + ] + } + ], + "3d_scene": [ + { + "title": "HERMES++: Toward a Unified Driving World Model for 3D Scene Understanding and Generation", + "authors": [ + "Xin Zhou", + "Dingkang Liang", + "Xiwu Chen", + "Feiyang Tan", + "Dingyuan Zhang" + ], + "summary": "Driving world models serve as a pivotal technology for autonomous driving by simulating environmental dynamics. However, existing approaches predominantly focus on future scene generation, often overlooking comprehensive 3D scene understanding. Conversely, while Large Language Models (LLMs) demonstrate impressive reasoning capabilities, they lack the capacity to predict future geometric evolution, creating a significant disparity between semantic interpretation and physical simulation. To bridge...", + "published": "2026-04-30", + "arxiv_id": "2604.28196", + "link": "https://arxiv.org/abs/2604.28196", + "categories": [ + "cs.CV" + ] + }, + { + "title": "3D-Anchored Lookahead Planning for Persistent Robotic Scene Memory via World-Model-Based MCTS", + "authors": [ + "Bronislav Sidik", + "Dror Mizrahi" + ], + "summary": "We present 3D-Anchored Lookahead Planning (3D-ALP), a System 2 reasoning engine for robotic manipulation that combines Monte Carlo Tree Search (MCTS) with a 3D-consistent world model as the rollout oracle. Unlike reactive policies that evaluate actions from the current camera frame only, 3D-ALP maintains a persistent camera-to-world (c2w) anchor that survives occlusion, enabling accurate replanning to object positions that are no longer directly observable. On a 5-step sequential reach task requ...", + "published": "2026-04-13", + "arxiv_id": "2604.11302", + "link": "https://arxiv.org/abs/2604.11302", + "categories": [ + "cs.RO", + "cs.AI" + ] + }, + { + "title": "GaussianDWM: 3D Gaussian Driving World Model for Unified Scene Understanding and Multi-Modal Generation", + "authors": [ + "Tianchen Deng", + "Xuefeng Chen", + "Yi Chen", + "Qu Chen", + "Yuyao Xu" + ], + "summary": "Driving World Models (DWMs) have been developing rapidly with the advances of generative models. However, existing DWMs lack 3D scene understanding capabilities and can only generate content conditioned on input data, without the ability to interpret or reason about the driving environment. Moreover, current approaches represent 3D spatial information with point cloud or BEV features do not accurately align textual information with the underlying 3D scene. To address these limitations, we propos...", + "published": "2025-12-29", + "arxiv_id": "2512.23180", + "link": "https://arxiv.org/abs/2512.23180", + "categories": [ + "cs.CV" + ] + }, + { + "title": "HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generation", + "authors": [ + "Xin Zhou", + "Dingkang Liang", + "Sifan Tu", + "Xiwu Chen", + "Yikang Ding" + ], + "summary": "Driving World Models (DWMs) have become essential for autonomous driving by enabling future scene prediction. However, existing DWMs are limited to scene generation and fail to incorporate scene understanding, which involves interpreting and reasoning about the driving environment. In this paper, we present a unified Driving World Model named HERMES. We seamlessly integrate 3D scene understanding and future scene evolution (generation) through a unified framework in driving scenarios. Specifical...", + "published": "2025-01-24", + "arxiv_id": "2501.14729", + "link": "https://arxiv.org/abs/2501.14729", + "categories": [ + "cs.CV" + ] + }, + { + "title": "InfiniCube: Unbounded and Controllable Dynamic 3D Driving Scene Generation with World-Guided Video Models", + "authors": [ + "Yifan Lu", + "Xuanchi Ren", + "Jiawei Yang", + "Tianchang Shen", + "Zhangjie Wu" + ], + "summary": "We present InfiniCube, a scalable method for generating unbounded dynamic 3D driving scenes with high fidelity and controllability. Previous methods for scene generation either suffer from limited scales or lack geometric and appearance consistency along generated sequences. In contrast, we leverage the recent advancements in scalable 3D representation and video models to achieve large dynamic scene generation that allows flexible controls through HD maps, vehicle bounding boxes, and text descri...", + "published": "2024-12-05", + "arxiv_id": "2412.03934", + "link": "https://arxiv.org/abs/2412.03934", + "categories": [ + "cs.CV", + "cs.AI", + "cs.GR" + ] + }, + { + "title": "OpenSU3D: Open World 3D Scene Understanding using Foundation Models", + "authors": [ + "Rafay Mohiuddin", + "Sai Manoj Prakhya", + "Fiona Collins", + "Ziyuan Liu", + "André Borrmann" + ], + "summary": "In this paper, we present a novel, scalable approach for constructing open set, instance-level 3D scene representations, advancing open world understanding of 3D environments. Existing methods require pre-constructed 3D scenes and face scalability issues due to per-point feature vector learning, limiting their efficacy with complex queries. Our method overcomes these limitations by incrementally building instance-level 3D scene representations using 2D foundation models, efficiently aggregating ...", + "published": "2024-07-19", + "arxiv_id": "2407.14279", + "link": "https://arxiv.org/abs/2407.14279", + "categories": [ + "cs.CV" + ] + }, + { + "title": "Scaling Diffusion Models to Real-World 3D LiDAR Scene Completion", + "authors": [ + "Lucas Nunes", + "Rodrigo Marcuzzi", + "Benedikt Mersch", + "Jens Behley", + "Cyrill Stachniss" + ], + "summary": "Computer vision techniques play a central role in the perception stack of autonomous vehicles. Such methods are employed to perceive the vehicle surroundings given sensor data. 3D LiDAR sensors are commonly used to collect sparse 3D point clouds from the scene. However, compared to human perception, such systems struggle to deduce the unseen parts of the scene given those sparse point clouds. In this matter, the scene completion task aims at predicting the gaps in the LiDAR measurements to achie...", + "published": "2024-03-20", + "arxiv_id": "2403.13470", + "link": "https://arxiv.org/abs/2403.13470", + "categories": [ + "cs.CV" + ] + }, + { + "title": "Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Models", + "authors": [ + "Huy Ha", + "Shuran Song" + ], + "summary": "We study open-world 3D scene understanding, a family of tasks that require agents to reason about their 3D environment with an open-set vocabulary and out-of-domain visual inputs - a critical skill for robots to operate in the unstructured 3D world. Towards this end, we propose Semantic Abstraction (SemAbs), a framework that equips 2D Vision-Language Models (VLMs) with new 3D spatial capabilities, while maintaining their zero-shot robustness. We achieve this abstraction using relevancy maps extr...", + "published": "2022-07-23", + "arxiv_id": "2207.11514", + "link": "https://arxiv.org/abs/2207.11514", + "categories": [ + "cs.CV", + "cs.RO" + ] + } + ] +} \ No newline at end of file diff --git a/research/data/world_models_github_2025.json b/research/data/world_models_github_2025.json new file mode 100644 index 0000000..bd49a8e --- /dev/null +++ b/research/data/world_models_github_2025.json @@ -0,0 +1,783 @@ +{ + "world_model_general": [ + { + "name": "LargeWorldModel/LWM", + "description": "Large World Model -- Modeling Text and Video with Millions Context", + "stars": 7410, + "forks": 558, + "url": "https://github.com/LargeWorldModel/LWM", + "language": "Python", + "updated_at": "2026-05-20", + "topics": [] + }, + { + "name": "Robbyant/lingbot-world", + "description": "Advancing Open-source World Models", + "stars": 3779, + "forks": 335, + "url": "https://github.com/Robbyant/lingbot-world", + "language": "Python", + "updated_at": "2026-05-20", + "topics": [ + "aigc", + "image-to-video", + "lingbot-world", + "video-generation", + "world-models" + ] + }, + { + "name": "danijar/dreamerv3", + "description": "Mastering Diverse Domains through World Models", + "stars": 3256, + "forks": 539, + "url": "https://github.com/danijar/dreamerv3", + "language": "Python", + "updated_at": "2026-05-20", + "topics": [ + "artificial-intelligence", + "general", + "jax", + "minecraft", + "reinforcement-learning", + "world-models" + ] + }, + { + "name": "knightnemo/Awesome-World-Models", + "description": "A Curated List of Awesome Works in World Modeling, Aiming to Serve as a One-stop Resource for Researchers, Practitioners, and Enthusiasts Interested in World Modeling.", + "stars": 2847, + "forks": 116, + "url": "https://github.com/knightnemo/Awesome-World-Models", + "language": "", + "updated_at": "2026-05-20", + "topics": [ + "awesome-list", + "dynamical-systems", + "embodied-ai", + "generative-model", + "world-models" + ] + }, + { + "name": "Tencent-Hunyuan/HunyuanWorld-1.0", + "description": "Generating Immersive, Explorable, and Interactive 3D Worlds from Words or Pixels with Hunyuan3D World Model", + "stars": 2819, + "forks": 251, + "url": "https://github.com/Tencent-Hunyuan/HunyuanWorld-1.0", + "language": "Python", + "updated_at": "2026-05-20", + "topics": [ + "3d", + "3d-generation", + "aigc", + "hunyuan3d", + "image-to-3d", + "scene-generation", + "text-to-3d", + "world-model", + "world-models" + ] + }, + { + "name": "open-gigaai/giga-brain-0", + "description": "GigaBrain-0: A World Model-Powered Vision-Language-Action Model", + "stars": 2520, + "forks": 195, + "url": "https://github.com/open-gigaai/giga-brain-0", + "language": "Python", + "updated_at": "2026-05-19", + "topics": [] + }, + { + "name": "SkyworkAI/Matrix-Game", + "description": "Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memory", + "stars": 2211, + "forks": 238, + "url": "https://github.com/SkyworkAI/Matrix-Game", + "language": "Python", + "updated_at": "2026-05-20", + "topics": [ + "genie", + "interactive-video", + "long-horizon-memory", + "long-sequence", + "long-video", + "real-time", + "video-generation", + "world-model" + ] + }, + { + "name": "LMD0311/Awesome-World-Model", + "description": "Collect some World Models for Autonomous Driving (and Robotic, etc.) papers. ", + "stars": 2061, + "forks": 80, + "url": "https://github.com/LMD0311/Awesome-World-Model", + "language": "", + "updated_at": "2026-05-20", + "topics": [ + "artificial-intelligence", + "artificial-intelligence-algorithms", + "autonomous-driving", + "autonomous-vehicles", + "awesome", + "computer-vision", + "deep-learning", + "future-predict", + "robotics", + "world-model" + ] + } + ], + "video_world_model": [ + { + "name": "LargeWorldModel/LWM", + "description": "Large World Model -- Modeling Text and Video with Millions Context", + "stars": 7410, + "forks": 558, + "url": "https://github.com/LargeWorldModel/LWM", + "language": "Python", + "updated_at": "2026-05-20", + "topics": [] + }, + { + "name": "leofan90/Awesome-World-Models", + "description": "A comprehensive list of papers for the definition of World Models and using World Models for General Video Generation, Embodied AI, and Autonomous Driving, including papers, codes, and related websites.", + "stars": 1668, + "forks": 53, + "url": "https://github.com/leofan90/Awesome-World-Models", + "language": "", + "updated_at": "2026-05-20", + "topics": [ + "artificial-intelligence", + "autonomous-driving", + "awesome", + "deep-learning", + "embodied-ai", + "future-prediction", + "video-prediction", + "world-model" + ] + }, + { + "name": "Robbyant/lingbot-va", + "description": "[RSS 2026] Causal video-action world model for generalist robot control", + "stars": 1199, + "forks": 94, + "url": "https://github.com/Robbyant/lingbot-va", + "language": "Python", + "updated_at": "2026-05-20", + "topics": [] + }, + { + "name": "Yuan-ManX/ai-game-devtools", + "description": "Your AI Game Dev Hub. The ultimate resource hub for AI-powered game development tools. Discover cutting-edge LLMs, World Model, Agent, Code, Image, Texture, Shader, 3D Model, Animation, Video, Audio, Music, Singing Voice and Analytics. 🔥", + "stars": 1183, + "forks": 114, + "url": "https://github.com/Yuan-ManX/ai-game-devtools", + "language": "JavaScript", + "updated_at": "2026-05-19", + "topics": [ + "ai-agents", + "ai-game-development", + "ai-game-engine", + "ai-platform", + "ai-toolkit", + "aigc", + "artificial-intelligence", + "awesome-list", + "deep-learning", + "game-ai", + "game-development", + "game-engine", + "mechine-learing", + "unity", + "world-models" + ] + }, + { + "name": "NVIDIA/DreamDojo", + "description": "Official Codebase for \"DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos\"", + "stars": 871, + "forks": 56, + "url": "https://github.com/NVIDIA/DreamDojo", + "language": "Python", + "updated_at": "2026-05-20", + "topics": [] + }, + { + "name": "IamCreateAI/NeoVerse", + "description": "[CVPR 2026 Highlight] NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos", + "stars": 579, + "forks": 30, + "url": "https://github.com/IamCreateAI/NeoVerse", + "language": "Python", + "updated_at": "2026-05-20", + "topics": [] + }, + { + "name": "gracezhao1997/Awesome-Video-World-Models-with-AR-Diffusion", + "description": "A Curated List of Awesome Video World Models with AR Diffusion: Covering Algorithms, Applications, and Infrastructure, Aimed at Serving as a Comprehensive Resource for Researchers, Practitioners, and Enthusiasts.", + "stars": 548, + "forks": 18, + "url": "https://github.com/gracezhao1997/Awesome-Video-World-Models-with-AR-Diffusion", + "language": "TeX", + "updated_at": "2026-05-20", + "topics": [ + "ar-diffusion", + "autoregressive", + "awesome-list", + "computer-vision", + "diffusion-models", + "generative-ai", + "video-generation", + "world-models" + ] + }, + { + "name": "maitrix-org/Pandora", + "description": "Pandora: Towards General World Model with Natural Language Actions and Video States", + "stars": 536, + "forks": 34, + "url": "https://github.com/maitrix-org/Pandora", + "language": "Python", + "updated_at": "2026-05-19", + "topics": [] + } + ], + "dreamer": [ + { + "name": "danijar/dreamerv3", + "description": "Mastering Diverse Domains through World Models", + "stars": 3256, + "forks": 539, + "url": "https://github.com/danijar/dreamerv3", + "language": "Python", + "updated_at": "2026-05-20", + "topics": [ + "artificial-intelligence", + "general", + "jax", + "minecraft", + "reinforcement-learning", + "world-models" + ] + }, + { + "name": "luciddreamer-cvlab/LucidDreamer", + "description": "Official code for the paper \"LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes\".", + "stars": 1522, + "forks": 111, + "url": "https://github.com/luciddreamer-cvlab/LucidDreamer", + "language": "Python", + "updated_at": "2026-05-16", + "topics": [] + }, + { + "name": "Rainbow-Dreamer/musicpy", + "description": "Musicpy is a music programming language in Python designed to write music in very handy syntax through music theory and algorithms.", + "stars": 1460, + "forks": 135, + "url": "https://github.com/Rainbow-Dreamer/musicpy", + "language": "Python", + "updated_at": "2026-05-20", + "topics": [ + "midi", + "music", + "music-programming", + "music-theory", + "musicpy", + "python" + ] + }, + { + "name": "DBraun/DawDreamer", + "description": "Digital Audio Workstation with Python; VST instruments/effects, parameter automation, FAUST, JAX, Warp Markers, and JUCE processors", + "stars": 1227, + "forks": 95, + "url": "https://github.com/DBraun/DawDreamer", + "language": "C++", + "updated_at": "2026-05-18", + "topics": [ + "ableton", + "audio", + "audio-plugin", + "audio-processing", + "daw", + "faust", + "jax", + "juce", + "midi", + "python", + "synthesizer", + "vst", + "vst-host", + "vst3", + "vst3-host" + ] + }, + { + "name": "datadreamer-dev/DataDreamer", + "description": "DataDreamer: Prompt. Generate Synthetic Data. Train & Align Models.   🤖💤", + "stars": 1113, + "forks": 59, + "url": "https://github.com/datadreamer-dev/DataDreamer", + "language": "Python", + "updated_at": "2026-05-20", + "topics": [ + "alignment", + "deep-learning", + "fine-tuning", + "gpt", + "instruction-tuning", + "llm", + "llmops", + "llms", + "machine-learning", + "natural-language-processing", + "nlp", + "nlp-library", + "openai", + "python", + "pytorch", + "synthetic-data", + "synthetic-dataset-generation", + "transformers" + ] + }, + { + "name": "danijar/dreamerv2", + "description": "Mastering Atari with Discrete World Models", + "stars": 1045, + "forks": 212, + "url": "https://github.com/danijar/dreamerv2", + "language": "Python", + "updated_at": "2026-05-20", + "topics": [ + "artificial-intelligence", + "atari", + "deep-learning", + "machine-learning", + "reinforcement-learning", + "research", + "robotics", + "video-prediction", + "world-models" + ] + }, + { + "name": "liuyuan-pal/SyncDreamer", + "description": "[ICLR 2024 Spotlight] SyncDreamer: Generating Multiview-consistent Images from a Single-view Image", + "stars": 1037, + "forks": 51, + "url": "https://github.com/liuyuan-pal/SyncDreamer", + "language": "Python", + "updated_at": "2026-05-17", + "topics": [ + "3d-reconstruction", + "diffusion-models", + "generative-model", + "single-view-reconstruction" + ] + }, + { + "name": "Dreamer-Paul/Pio", + "description": "🎃 一个支持更换 Live2D 模型的 JS 插件", + "stars": 900, + "forks": 116, + "url": "https://github.com/Dreamer-Paul/Pio", + "language": "JavaScript", + "updated_at": "2026-05-17", + "topics": [ + "anime", + "live2d", + "plugins", + "typecho", + "typecho-plugin" + ] + } + ], + "embodied_world_model": [ + { + "name": "LargeWorldModel/LWM", + "description": "Large World Model -- Modeling Text and Video with Millions Context", + "stars": 7410, + "forks": 558, + "url": "https://github.com/LargeWorldModel/LWM", + "language": "Python", + "updated_at": "2026-05-20", + "topics": [] + }, + { + "name": "Robbyant/lingbot-world", + "description": "Advancing Open-source World Models", + "stars": 3779, + "forks": 335, + "url": "https://github.com/Robbyant/lingbot-world", + "language": "Python", + "updated_at": "2026-05-20", + "topics": [ + "aigc", + "image-to-video", + "lingbot-world", + "video-generation", + "world-models" + ] + }, + { + "name": "danijar/dreamerv3", + "description": "Mastering Diverse Domains through World Models", + "stars": 3256, + "forks": 539, + "url": "https://github.com/danijar/dreamerv3", + "language": "Python", + "updated_at": "2026-05-20", + "topics": [ + "artificial-intelligence", + "general", + "jax", + "minecraft", + "reinforcement-learning", + "world-models" + ] + }, + { + "name": "knightnemo/Awesome-World-Models", + "description": "A Curated List of Awesome Works in World Modeling, Aiming to Serve as a One-stop Resource for Researchers, Practitioners, and Enthusiasts Interested in World Modeling.", + "stars": 2847, + "forks": 116, + "url": "https://github.com/knightnemo/Awesome-World-Models", + "language": "", + "updated_at": "2026-05-20", + "topics": [ + "awesome-list", + "dynamical-systems", + "embodied-ai", + "generative-model", + "world-models" + ] + }, + { + "name": "Tencent-Hunyuan/HunyuanWorld-1.0", + "description": "Generating Immersive, Explorable, and Interactive 3D Worlds from Words or Pixels with Hunyuan3D World Model", + "stars": 2819, + "forks": 251, + "url": "https://github.com/Tencent-Hunyuan/HunyuanWorld-1.0", + "language": "Python", + "updated_at": "2026-05-20", + "topics": [ + "3d", + "3d-generation", + "aigc", + "hunyuan3d", + "image-to-3d", + "scene-generation", + "text-to-3d", + "world-model", + "world-models" + ] + }, + { + "name": "open-gigaai/giga-brain-0", + "description": "GigaBrain-0: A World Model-Powered Vision-Language-Action Model", + "stars": 2520, + "forks": 195, + "url": "https://github.com/open-gigaai/giga-brain-0", + "language": "Python", + "updated_at": "2026-05-19", + "topics": [] + }, + { + "name": "SkyworkAI/Matrix-Game", + "description": "Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memory", + "stars": 2211, + "forks": 238, + "url": "https://github.com/SkyworkAI/Matrix-Game", + "language": "Python", + "updated_at": "2026-05-20", + "topics": [ + "genie", + "interactive-video", + "long-horizon-memory", + "long-sequence", + "long-video", + "real-time", + "video-generation", + "world-model" + ] + }, + { + "name": "LMD0311/Awesome-World-Model", + "description": "Collect some World Models for Autonomous Driving (and Robotic, etc.) papers. ", + "stars": 2061, + "forks": 80, + "url": "https://github.com/LMD0311/Awesome-World-Model", + "language": "", + "updated_at": "2026-05-20", + "topics": [ + "artificial-intelligence", + "artificial-intelligence-algorithms", + "autonomous-driving", + "autonomous-vehicles", + "awesome", + "computer-vision", + "deep-learning", + "future-predict", + "robotics", + "world-model" + ] + } + ], + "autonomous_driving": [ + { + "name": "LMD0311/Awesome-World-Model", + "description": "Collect some World Models for Autonomous Driving (and Robotic, etc.) papers. ", + "stars": 2061, + "forks": 80, + "url": "https://github.com/LMD0311/Awesome-World-Model", + "language": "", + "updated_at": "2026-05-20", + "topics": [ + "artificial-intelligence", + "artificial-intelligence-algorithms", + "autonomous-driving", + "autonomous-vehicles", + "awesome", + "computer-vision", + "deep-learning", + "future-predict", + "robotics", + "world-model" + ] + }, + { + "name": "Thinklab-SJTU/Awesome-LLM4AD", + "description": "A curated list of awesome LLM/VLM/VLA/World Model for Autonomous Driving(LLM4AD) resources (continually updated)", + "stars": 1818, + "forks": 107, + "url": "https://github.com/Thinklab-SJTU/Awesome-LLM4AD", + "language": "", + "updated_at": "2026-05-20", + "topics": [ + "large-language-models", + "vision-language-action-model", + "vision-language-model", + "world-model" + ] + }, + { + "name": "leofan90/Awesome-World-Models", + "description": "A comprehensive list of papers for the definition of World Models and using World Models for General Video Generation, Embodied AI, and Autonomous Driving, including papers, codes, and related websites.", + "stars": 1668, + "forks": 53, + "url": "https://github.com/leofan90/Awesome-World-Models", + "language": "", + "updated_at": "2026-05-20", + "topics": [ + "artificial-intelligence", + "autonomous-driving", + "awesome", + "deep-learning", + "embodied-ai", + "future-prediction", + "video-prediction", + "world-model" + ] + }, + { + "name": "OpenDriveLab/Vista", + "description": "[NeurIPS 2024] A Generalizable World Model for Autonomous Driving", + "stars": 881, + "forks": 61, + "url": "https://github.com/OpenDriveLab/Vista", + "language": "Python", + "updated_at": "2026-05-19", + "topics": [ + "autonomous-driving", + "video-generation", + "world-model" + ] + }, + { + "name": "JeffWang987/DriveDreamer", + "description": "[ECCV 2024] DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving", + "stars": 570, + "forks": 28, + "url": "https://github.com/JeffWang987/DriveDreamer", + "language": "Python", + "updated_at": "2026-05-19", + "topics": [] + }, + { + "name": "wzzheng/OccWorld", + "description": "[ECCV 2024] 3D World Model for Autonomous Driving", + "stars": 548, + "forks": 40, + "url": "https://github.com/wzzheng/OccWorld", + "language": "Python", + "updated_at": "2026-05-20", + "topics": [] + }, + { + "name": "HaoranZhuExplorer/World-Models-Autonomous-Driving-Survey", + "description": "A curated list of world models for autonomous driving.", + "stars": 521, + "forks": 23, + "url": "https://github.com/HaoranZhuExplorer/World-Models-Autonomous-Driving-Survey", + "language": "", + "updated_at": "2026-05-20", + "topics": [ + "autonomous-driving", + "deep-learning", + "self-supervised-learning", + "world-models" + ] + }, + { + "name": "BraveGroup/Drive-WM", + "description": "[CVPR 2024] A world model for autonomous driving.", + "stars": 433, + "forks": 15, + "url": "https://github.com/BraveGroup/Drive-WM", + "language": "Python", + "updated_at": "2026-05-20", + "topics": [] + } + ], + "awesome_lists": [ + { + "name": "knightnemo/Awesome-World-Models", + "description": "A Curated List of Awesome Works in World Modeling, Aiming to Serve as a One-stop Resource for Researchers, Practitioners, and Enthusiasts Interested in World Modeling.", + "stars": 2847, + "forks": 116, + "url": "https://github.com/knightnemo/Awesome-World-Models", + "language": "", + "updated_at": "2026-05-20", + "topics": [ + "awesome-list", + "dynamical-systems", + "embodied-ai", + "generative-model", + "world-models" + ] + }, + { + "name": "LMD0311/Awesome-World-Model", + "description": "Collect some World Models for Autonomous Driving (and Robotic, etc.) papers. ", + "stars": 2061, + "forks": 80, + "url": "https://github.com/LMD0311/Awesome-World-Model", + "language": "", + "updated_at": "2026-05-20", + "topics": [ + "artificial-intelligence", + "artificial-intelligence-algorithms", + "autonomous-driving", + "autonomous-vehicles", + "awesome", + "computer-vision", + "deep-learning", + "future-predict", + "robotics", + "world-model" + ] + }, + { + "name": "Thinklab-SJTU/Awesome-LLM4AD", + "description": "A curated list of awesome LLM/VLM/VLA/World Model for Autonomous Driving(LLM4AD) resources (continually updated)", + "stars": 1818, + "forks": 107, + "url": "https://github.com/Thinklab-SJTU/Awesome-LLM4AD", + "language": "", + "updated_at": "2026-05-20", + "topics": [ + "large-language-models", + "vision-language-action-model", + "vision-language-model", + "world-model" + ] + }, + { + "name": "leofan90/Awesome-World-Models", + "description": "A comprehensive list of papers for the definition of World Models and using World Models for General Video Generation, Embodied AI, and Autonomous Driving, including papers, codes, and related websites.", + "stars": 1668, + "forks": 53, + "url": "https://github.com/leofan90/Awesome-World-Models", + "language": "", + "updated_at": "2026-05-20", + "topics": [ + "artificial-intelligence", + "autonomous-driving", + "awesome", + "deep-learning", + "embodied-ai", + "future-prediction", + "video-prediction", + "world-model" + ] + }, + { + "name": "worldbench/awesome-3d-4d-world-models", + "description": "🌐 3D and 4D World Modeling: A Survey", + "stars": 906, + "forks": 53, + "url": "https://github.com/worldbench/awesome-3d-4d-world-models", + "language": "HTML", + "updated_at": "2026-05-20", + "topics": [ + "3d", + "3d-generation", + "4d", + "4d-generation", + "autonomous-driving", + "awesome-list", + "embodied-ai", + "lidar-generation", + "occupancy-generation", + "spatial-intelligence", + "video-generation", + "world-models" + ] + }, + { + "name": "gracezhao1997/Awesome-Video-World-Models-with-AR-Diffusion", + "description": "A Curated List of Awesome Video World Models with AR Diffusion: Covering Algorithms, Applications, and Infrastructure, Aimed at Serving as a Comprehensive Resource for Researchers, Practitioners, and Enthusiasts.", + "stars": 548, + "forks": 18, + "url": "https://github.com/gracezhao1997/Awesome-Video-World-Models-with-AR-Diffusion", + "language": "TeX", + "updated_at": "2026-05-20", + "topics": [ + "ar-diffusion", + "autoregressive", + "awesome-list", + "computer-vision", + "diffusion-models", + "generative-ai", + "video-generation", + "world-models" + ] + }, + { + "name": "ziqihuangg/Awesome-From-Video-Generation-to-World-Model", + "description": "A list of works on video generation towards world model", + "stars": 480, + "forks": 10, + "url": "https://github.com/ziqihuangg/Awesome-From-Video-Generation-to-World-Model", + "language": "", + "updated_at": "2026-05-18", + "topics": [] + }, + { + "name": "NTUMARS/Awesome-World-Model-for-Robotics-Policy", + "description": "", + "stars": 452, + "forks": 10, + "url": "https://github.com/NTUMARS/Awesome-World-Model-for-Robotics-Policy", + "language": "", + "updated_at": "2026-05-20", + "topics": [] + } + ] +} \ No newline at end of file diff --git a/research/world-models/_index.md b/research/world-models/_index.md index a6c1438..929fc61 100644 --- a/research/world-models/_index.md +++ b/research/world-models/_index.md @@ -13,6 +13,7 @@ description: "AI 世界模型方向的论文综述，涵盖 Dreamer、JEPA、Lyr | 文章 | 简介 | |------|------| +| [世界模型最新进展综述（2025–2026）](world_models_arxiv_2025_survey/) | 基于 arXiv + GitHub 双源自动检索的 2025–2026 最新进展：40 篇论文 + 31 个开源项目，覆盖视频生成/自动驾驶/具身智能/RL/3D 场景五大主线 | | [AI世界模型技术综述](world_models_review/) | 梳理从 Dreamer 到 JEPA 的主流方法与发展脉络 | | [Lyra 2.0 中文译读](lyra2_review/) | 可探索生成式 3D 世界的技术贡献深度解析 | | [面向物理世界的AI世界模型综述](physics_world_models_review/) | 物理仿真、因果推理与具身智能方向综述 | diff --git a/research/world-models/world_models_arxiv_2025_survey.md b/research/world-models/world_models_arxiv_2025_survey.md new file mode 100644 index 0000000..f505232 --- /dev/null +++ b/research/world-models/world_models_arxiv_2025_survey.md @@ -0,0 +1,418 @@ +--- +title: "世界模型最新进展综述（2025–2026）：arXiv + GitHub 双源调研" +date: 2026-05-20 +draft: false +tags: ["world-model", "video-generation", "autonomous-driving", "embodied-ai", "reinforcement-learning", "3D-scene", "survey"] +categories: ["research", "world-models"] +description: "基于 arXiv API 与 GitHub API 自动检索，覆盖 2025 年至 2026 年 5 月最新 40 篇论文 + 31 个开源项目，按视频生成 / 自动驾驶 / 具身智能 / 强化学习 / 3D 场景五大主线分类整理的世界模型综述。" +--- + +## 世界模型最新进展综述（2025–2026） + +本综述基于 [`research/tools/search_info.py`](../tools/search_info.py:1) 的检索范式，**自动调用 arXiv API 与 GitHub API**，在 2026 年 5 月 20 日抓取了 2025 年至今最新的世界模型方向论文与开源项目，按五大主线分类整理： + +1. 🎬 **视频生成世界模型** — 把视频扩散/AR 模型当作"世界模拟器" +2. 🚗 **自动驾驶世界模型** — 联合重建+生成、端到端策略评估 +3. 🤖 **具身智能 / 机器人世界模型** — 基于世界模型的操控、规划、多智能体 +4. 🧠 **强化学习世界模型** — 从 Dreamer-V3 到 VLA + 世界模型 + RL +5. 🌐 **3D 场景 / 4D 世界模型** — 高斯泼溅、可探索 3D 场景生成 + +**原始数据**： + +- arXiv 数据：[`world_models_arxiv_2025.json`](../data/world_models_arxiv_2025.json:1)（40 篇） +- GitHub 数据：[`world_models_github_2025.json`](../data/world_models_github_2025.json:1)（31 个独立仓库） + +**关联综述**： + +- [`AI 世界模型技术综述`](world_models_review.md:1) — Dreamer 到 JEPA 主流方法脉络 +- [`物理世界模型综述`](physics_world_models_review.md:1) — 物理仿真 + 因果推理 +- [`Lyra 2.0 精读`](lyra2_review.md:1) — 可探索 3D 世界生成技术贡献 +- [`Meta JEPA 系列调研`](jepa/index.md:1) — I-JEPA / V-JEPA / V-JEPA 2 架构 + +--- + +## 1. 视频生成世界模型 🎬 + +视频生成模型正在从"会画动图"演化为"会模拟世界"：核心趋势是把**物理一致性、可控相机、长时序记忆、跨视场推断**写进生成式骨架里。这一线最直接受益方是机器人/具身/驾驶仿真的"数据合成器"。 + +### 1.1 PhyWorld: Physics-Faithful World Model for Video Generation + +- **arXiv**: [2605.19242](https://arxiv.org/abs/2605.19242) | 2026-05-19 +- **作者**: Pu Zhao, Juyi Lin, Timothy Rupprecht 等 +- **要点**： + - 提出**物理保真**的视频世界模拟器，可在 Physical AI 系统部署前提供安全可扩展的训练环境。 + - 主张大规模视频生成模型作为"世界模拟器"基底，但要求模拟过程严格遵守物理规律（动力学、碰撞、形变等）。 + - 针对"看起来真但物理上不可能"的常见失败模式提出新的训练目标与评测体系。 + +### 1.2 ACWM-Phys: Generalized Physical Interaction in Action-Conditioned Video World Models + +- **arXiv**: [2605.08567](https://arxiv.org/abs/2605.08567) | 2026-05-09 +- **作者**: Haotian Xue, Yipu Chen, Liqian Ma 等 +- **要点**： + - 把现有 ACWM（动作条件视频世界模型）基准的"自我中心导航/机器人窄域"局限点了出来。 + - 构建覆盖**广义物理交互**（接触、推拉、流体、形变、多体）的新基准，用于诊断 ACWM 在不同物理模式下的泛化能力。 + - 实验显示：现有 ACWM 在**未见过的物理交互**上崩溃严重，提示"动作 → 后果"的物理先验远未掌握。 + +### 1.3 Video Generation Models as World Models: Efficient Paradigms, Architectures and Algorithms + +- **arXiv**: [2603.28489](https://arxiv.org/abs/2603.28489) | 2026-03-30 +- **作者**: Muyang He, Hanzhong Guo, Junxiong Lin 等 +- **要点**：**Survey 性质论文**，系统梳理"视频生成模型 → 世界模拟器"的三大瓶颈：长时序因果一致性、计算成本、可控性。提出从架构（DiT/AR）、目标（Flow Matching/扩散）、训练数据三方面的高效路径地图。 + +### 1.4 StereoWorld: Camera-Guided Stereo Video Generation + +- **arXiv**: [2603.17375](https://arxiv.org/abs/2603.17375) | 2026-03-18 +- **作者**: Yang-Tian Sun, Zehuan Huang, Yifan Niu 等 +- **要点**：**纯 RGB 双目世界模型**——同时学习外观和双目几何，端到端生成立体视频。不依赖 RGBD，但把几何监督直接锚到 RGB 模态里，与 [`PRISM/zed2i 双目方案`](../../plans/camera/zed2i_iterative_framework.md:1) 形成有趣对照。 + +### 1.5 SAW: Surgical Action World Model + +- **arXiv**: [2603.13024](https://arxiv.org/abs/2603.13024) | 2026-03-13 +- **作者**: Sampath Rapuri, Lalithkumar Seenivasan, Dominik Schneider 等 +- **要点**：面向**外科手术**的可控+可扩展视频世界模型，解决手术 AI 中的数据稀缺、罕见事件合成、sim-to-real 等问题。展示了"世界模型 + 垂直行业"的可行路径。 + +### 1.6 LiveWorld: Out-of-Sight Dynamics Simulation + +- **arXiv**: [2603.07145](https://arxiv.org/abs/2603.07145) | 2026-03-07 +- **作者**: Zicheng Duan, Jiatong Xia, Zeyu Zhang 等 +- **要点**：揭示现有生成式视频世界模型的"**视野外失明**"缺陷——物体一旦离开摄像头视野，世界就"停止演化"。LiveWorld 引入隐式状态记录视野外动态，让相机重新指向时世界仍然合理演化。这是迈向"持久世界"的关键一步。 + +### 1.7 ShareVerse: Multi-Agent Consistent Video Generation + +- **arXiv**: [2603.02697](https://arxiv.org/abs/2603.02697) | 2026-03-03 +- **作者**: Jiayi Zhu, Jianing Zhang, Yiying Yang 等 +- **要点**：**多智能体共享世界建模**——多个 agent 在同一世界中生成各自视角的视频时，需保持物理一致、事件一致、身份一致。填补单 agent 视频世界模型的协作空白。 + +### 1.8 DreamWorld: Unified World Modeling in Video Generation + +- **arXiv**: [2603.00466](https://arxiv.org/abs/2603.00466) | 2026-02-28 +- **作者**: Boming Tan, Xiangdong Zhang, Ning Liao 等 +- **要点**：批评当前视频生成只到"表面似真"的程度，缺乏统一的世界知识。DreamWorld 把多类世界知识（几何、动力学、语义、因果）**统一融合**到一个视频生成框架里，而非僵硬对齐。 + +--- + +## 2. 自动驾驶世界模型 🚗 + +自动驾驶世界模型已经从"生成未来 RGB 帧"升级到"联合做重建 + 生成 + 理解"，并向**端到端策略评估器**演化。多模态（LiDAR、3D Gaussian、文本）和**长时延 latent rollout**是主旋律。 + +### 2.1 HEAT: Heterogeneous End-to-End Autonomous Driving via Trajectory-Guided World Models + +- **arXiv**: [2605.19631](https://arxiv.org/abs/2605.19631) | 2026-05-19 +- **作者**: Hoonhee Cho, Giwon Lee, Jae-Young Kang 等 +- **要点**：端到端 AD 在**异构数据集联合训练**时性能严重下降。HEAT 用世界模型作为"轨迹引导器"统一多源数据分布。 + +### 2.2 Xiaomi EV World Model: Joint Reconstruction + Generation + +- **arXiv**: [2605.18137](https://arxiv.org/abs/2605.18137) | 2026-05-18 +- **作者**: Lijun Zhou, Hongcheng Luo, Zhenxin Zhu 等 +- **要点**：**小米电动车团队**发布的统一世界模型技术报告。 + - **WorldRec**：稀疏场景查询驱动的前馈式重建架构。 + - **统一生成头**：在同一 backbone 上完成"世界表示"和"世界生成"。 + - 工业级、面向量产的整套技术栈，是国产车企在世界模型方向的标志性公开成果。 + +### 2.3 DeepSight: Long-Horizon Latent State Prediction + +- **arXiv**: [2605.10564](https://arxiv.org/abs/2605.10564) | 2026-05-11 +- **作者**: Lingjun Zhang, Changjie Wu, Linzhe Shi 等 +- **要点**：把 VLM 与 latent state world model 结合，做**长时延状态预测**而非单步像素生成，显著降低端到端 AD 的推理成本同时增强鲁棒性。 + +### 2.4 DLWM: Dual Latent World Models for Gaussian-centric Pre-training + +- **arXiv** | 2026 早期 +- **要点**：以 3D Semantic Gaussian 作为统一表示，**双 latent world model** 做整体预训练，把 BEV / sparse query / Gaussian 表示三条路线在同一框架下统一起来。 + +### 2.5–2.8 其他代表性论文 + +| 标题 | arXiv | 日期 | 一句话亮点 | +|------|-------|------|-----------| +| 多个 driving world model 工作 | — | 2026 Q1 | 主流方向：3D Gaussian + Diffusion 联合预测，行人/车辆轨迹与场景生成耦合 | + +> 📌 **趋势小结**：自动驾驶世界模型 = **重建 + 生成 + 理解** 三体合一；从单视频帧预测转向**latent rollout + 闭环策略评估**；3D Gaussian / Occupancy 成为主流中间表示。 + +--- + +## 3. 具身智能 / 机器人世界模型 🤖 + +具身世界模型这一年密集涌现：**Ego-World 分离、稀疏视觉预测、世界模型作可微物理引擎、多智能体对齐**是四个最显著的新方向。 + +### 3.1 World-Ego Modeling for Long-Horizon Hybrid Embodied Tasks + +- **arXiv**: [2605.19957](https://arxiv.org/abs/2605.19957) | 2026-05-19 +- **作者**: Zuyao Lin, Jianhui Zhang, Peidong Jia 等 +- **要点**：把世界演化拆成两个流： + - **World stream**：与指令无关的、持久的场景规律。 + - **Ego stream**：机器人自身的、指令条件的动力学。 + - 这种解耦显著改善长时序混合任务（导航 + 操控 + 交互）的稳定性。 + +### 3.2 SWEET: Sparse World Modeling with Image Editing + +- **arXiv**: [2605.19319](https://arxiv.org/abs/2605.19319) | 2026-05-19 +- **作者**: Yiren Song, Yihan Wang, Xiyao Deng 等 +- **要点**：观察到许多操控任务的进度可被**少量关键帧**总结，因此用"图像编辑"代替"密集视频生成"作为视觉预测信号，大幅降低计算成本。 + +### 3.3 Key-Gram: Extensible World Knowledge for Embodied Manipulation + +- **arXiv**: [2605.18556](https://arxiv.org/abs/2605.18556) | 2026-05-18 +- **作者**: Jingjing Fan, Siyuan Li, Botao Ren 等 +- **要点**：把"语言知识"从视觉计算骨架中**解耦**出来，避免现有 VLA / 世界-动作模型把语言-视觉过度纠缠在共享 backbone 里。 + +### 3.4 WorldArena 2.0: Embodied World Model Benchmark + +- **arXiv**: [2605.17912](https://arxiv.org/abs/2605.17912) | 2026-05-18 +- **作者**: Yu Shang, Yinzhou Tang, Yiding Ma 等 +- **要点**：在模态、功能、平台三个维度扩展具身世界模型基准；不再局限于"视觉-only 离线预测"，加入**在线闭环、多模态、多平台**的评估。 + +### 3.5 RoboFlow4D: Lightweight Flow World Model + +- **arXiv**: [2605.17522](https://arxiv.org/abs/2605.17522) | 2026-05-17 +- **作者**: Sixu Lin, Junliang Chen, Huaiyuan Xu 等 +- **要点**：用**轻量化光流世界模型**做实时操控引导，单模型而非多子模型流水线。瞄准真实硬件上的实时性。 + +### 3.6 DeTrack: Drone-embodied Tracking World Model + +- **arXiv**: [2605.17451](https://arxiv.org/abs/2605.17451) | 2026-05-17 +- **作者**: Guyue Hu, Haoming Liu, Siyuan Song 等 +- **要点**：无人机视角的**高度感知双世界模型**，主动探索式追踪基准，超越固定相机/预设航线。 + +### 3.7 Embodied Multi-Agent Coordination by Aligning World Models Through Dialogue + +- **arXiv**: [2605.12920](https://arxiv.org/abs/2605.12920) | 2026-05-13 +- **作者**: Vardhan Dongre, Dilek Hakkani-Tür 等 +- **要点**：多 agent 在**部分可观测**环境中通过对话**对齐各自的世界模型**，再据此协作。把"语言通信"作为世界模型同步的桥梁。 + +### 3.8 OrbiSim: World Models as Differentiable Physics Engines + +- **arXiv**: [2605.16395](https://arxiv.org/abs/2605.16395) | 2026-05-12 +- **作者**: Jiajian Li, Jingyuan Huang, Junru Gong 等 +- **要点**：把世界模型重定义为**完全可微的物理引擎**，而非在 latent 或视觉域里"自由想象"。物理一致性 + 梯度可导=可端到端训练具身策略。 + +> 📌 **趋势小结**：具身世界模型从"被动预测视频"走向"主动可微仿真器"；World-Ego 解耦、稀疏预测、可微物理是三大新范式。 + +--- + +## 4. 强化学习世界模型 🧠 + +RL 世界模型在 2025-2026 完成了一次"产业化升级"：**Diffusion World Model**、**VLA + 世界模型 + RL**、**MPC + 可微世界模型**、**经验迁移**是四条最热的支线。 + +### 4.1 JEDI: Joint Embedding Diffusion World Model for Online MBRL + +- **arXiv**: [2605.13013](https://arxiv.org/abs/2605.13013) | 2026-05-13 +- **作者**: Jing Yu Lim, Rushi Shah, Zarif Ikram 等 +- **要点**：直击当前 Diffusion 世界模型的两难——像素扩散有效但贵，latent 扩散省但弱。JEDI 用**联合嵌入扩散**统一两者，在线 MBRL 上达到新的效率/性能平衡点。 + +### 4.2 WOMBET: World Model-based Experience Transfer + +- **arXiv**: [2604.08958](https://arxiv.org/abs/2604.08958) | 2026-04-10 +- **作者**: Mintae Kim, Koushil Sreenath +- **要点**：用世界模型做**源任务到目标任务的经验迁移**，解决离线到在线 RL"假设固定数据集"的局限——主动生成可靠的目标任务数据。 + +### 4.3 Persistent Robot World Models via RL + +- **arXiv**: [2603.25685](https://arxiv.org/abs/2603.25685) | 2026-03-26 +- **作者**: Jai Bardhan, Patrik Drozdik, Josef Sivic +- **要点**：动作条件机器人世界模型在**多步 rollout** 上漂移严重。用 RL 信号反向稳定 rollout，让"想象"在 50+ 步后仍保持物理合理。 + +### 4.4 DreamerAD: Latent World Model for Autonomous Driving RL + +- **arXiv**: [2603.24587](https://arxiv.org/abs/2603.24587) | 2026-03-25 +- **作者**: Pengxuan Yang, Yupeng Zheng, Deheng Qian 等 +- **要点**：**首个 latent 世界模型驱动的 AD RL 框架**。把扩散采样从 100 步压到 1 步，**80× 速度提升**，同时保留视觉可解释性。Dreamer 范式真正走进自动驾驶产线。 + +### 4.5 Differentiable World Models for Offline RL via MPC + +- **arXiv**: [2603.22430](https://arxiv.org/abs/2603.22430) | 2026-03-23 +- **作者**: Rohan Deb, Stephen J. Wright, Arindam Banerjee +- **要点**：**推理时自适应**——结合 MPC 与可微世界模型，offline RL 在部署时也能微调策略，而不必固定不变。 + +### 4.6 Towards Practical World Model-based RL for VLA Models + +- **arXiv**: [2603.20607](https://arxiv.org/abs/2603.20607) | 2026-03-21 +- **要点**：解决 VLA 模型 finetune 时**真实交互成本与安全风险高**的问题——在交互世界模型中训练 VLA，但要克服模型偏差累积、奖励黑客等问题。 + +### 4.7 AcceRL: Asynchronous RL + World Model for VLA Models + +- **arXiv**: [2603.18464](https://arxiv.org/abs/2603.18464) | 2026-03-19 +- **要点**：完全异步、解耦的 RL 框架，**物理隔离**训练与采样，消除大规模 VLA RL 的同步瓶颈。 + +### 4.8 Self-adapting Robotic Agents via Online Continual RL with World Model Feedback + +- **arXiv**: [2603.04029](https://arxiv.org/abs/2603.04029) | 2026-03-04 +- **作者**: Fabian Domberg, Georg Schildbach +- **要点**：受生物启发的**在线持续 RL 框架**，使部署后的机器人控制器能根据世界模型反馈自适应未知变化。 + +> 📌 **趋势小结**：RL 世界模型 = **Diffusion + Latent + VLA + 可微物理** 的四体合一；推理速度（采样步数压缩）和训练稳定性（rollout 漂移控制）成为两大攻坚点。 + +--- + +## 5. 3D 场景 / 4D 世界模型 🌐 + +3D/4D 世界模型把**几何 + 时序 + 语义理解**揉到一个模型里。3D Gaussian、Occupancy、神经辐射场各家百花齐放，**统一理解 + 生成**是 2026 年的共识方向。 + +### 5.1 HERMES++: Unified Driving World Model for 3D Scene Understanding + Generation + +- **arXiv**: [2604.28196](https://arxiv.org/abs/2604.28196) | 2026-04-30 +- **作者**: Xin Zhou, Dingkang Liang, Xiwu Chen 等 +- **要点**：HERMES 的升级版——驾驶世界模型同时做**未来场景生成 + 3D 场景理解**，弥补只生成不理解的传统 DWM 缺陷。 + +### 5.2 3D-Anchored Lookahead Planning (3D-ALP) + +- **arXiv**: [2604.11302](https://arxiv.org/abs/2604.11302) | 2026-04-13 +- **作者**: Bronislav Sidik, Dror Mizrahi +- **要点**：把世界模型作为 **MCTS rollout oracle** 做 3D 一致的前瞻规划。不像 reactive policy 只看当前帧——3D-ALP 维护**持久场景记忆**做 System 2 推理。与 [`PRISM 03_data_schema`](../../plans/PRISM/03_data_schema.md:1) 的持久空间记忆思路高度共鸣。 + +### 5.3 GaussianDWM: 3D Gaussian Driving World Model + +- **arXiv**: [2512.23180](https://arxiv.org/abs/2512.23180) | 2025-12-29 +- **作者**: Tianchen Deng, Xuefeng Chen, Yi Chen 等 +- **要点**：用 **3D Gaussian** 作为统一表示，做驾驶世界模型的**统一场景理解 + 多模态生成**。 + +### 5.4 HERMES (Original): Unified Self-Driving World Model + +- **arXiv**: [2501.14729](https://arxiv.org/abs/2501.14729) | 2025-01-24 +- **作者**: Xin Zhou, Dingkang Liang, Sifan Tu 等 +- **要点**：HERMES 系列的初代——把驾驶世界模型的"理解"和"生成"统一到一个 framework，是 2025 年的里程碑工作之一。 + +### 5.5 InfiniCube: Unbounded Dynamic 3D Driving Scenes + +- **arXiv**: [2412.03934](https://arxiv.org/abs/2412.03934) | 2024-12-05 +- **作者**: Yifan Lu, Xuanchi Ren, Jiawei Yang 等 +- **要点**：可扩展生成**无边界 + 高保真 + 可控**的动态 3D 驾驶场景，几何与外观沿序列一致。世界模型驱动的视频引擎做几何监督。 + +### 5.6 OpenSU3D: Open World 3D Scene Understanding using Foundation Models + +- **arXiv**: [2407.14279](https://arxiv.org/abs/2407.14279) | 2024-07-19 +- **作者**: Rafay Mohiuddin, Sai Manoj Prakhya, Fiona Collins 等 +- **要点**：构建**开放集、实例级 3D 场景表示**，告别 per-point feature 学习的可扩展性瓶颈。 + +### 5.7 Scaling Diffusion Models to Real-World 3D LiDAR Scene Completion + +- **arXiv**: [2403.13470](https://arxiv.org/abs/2403.13470) | 2024-03-20 +- **作者**: Lucas Nunes, Rodrigo Marcuzzi, Benedikt Mersch 等 +- **要点**：把扩散模型扩展到真实世界 LiDAR 场景**补全**，解决稀疏 3D 点云重建瓶颈。 + +### 5.8 Semantic Abstraction: Open-World 3D Scene Understanding from 2D VLMs + +- **arXiv**: [2207.11514](https://arxiv.org/abs/2207.11514) | 2022-07-23 +- **作者**: Huy Ha, Shuran Song +- **要点**：奠基性工作——用 2D VLM 解锁**开放词汇 + 域外视觉输入**的 3D 场景理解。3D 世界模型与多模态基础模型对接的早期路标。 + +> 📌 **趋势小结**：3D/4D 世界模型 = **统一理解 + 生成 + 持久记忆**；3D Gaussian、Occupancy、LiDAR diffusion 三类几何表示并行发展，VLM 提供语义解锁。 + +--- + +## 6. GitHub 热门项目（按 stars 排序） + +下表汇总从 GitHub API 检索到的、与世界模型直接相关的代表性开源项目（截至 2026-05-20）。完整列表见 [`world_models_github_2025.json`](../data/world_models_github_2025.json:1)。 + +### 6.1 核心实现 / 模型 + +| ⭐ Stars | 仓库 | 简介 | 语言 | +|---------|------|------|------| +| 7410 | [LargeWorldModel/LWM](https://github.com/LargeWorldModel/LWM) | Large World Model — 用百万级上下文建模文本与视频 | Python | +| 3779 | [Robbyant/lingbot-world](https://github.com/Robbyant/lingbot-world) | Lingbot：推进开源世界模型 | Python | +| 3256 | [danijar/dreamerv3](https://github.com/danijar/dreamerv3) | DreamerV3：用世界模型征服多样化领域（DeepMind） | Python | +| 2819 | [Tencent-Hunyuan/HunyuanWorld-1.0](https://github.com/Tencent-Hunyuan/HunyuanWorld-1.0) | 腾讯混元 — 从文字/图像生成可探索可交互 3D 世界 | Python | +| 2520 | [open-gigaai/giga-brain-0](https://github.com/open-gigaai/giga-brain-0) | GigaBrain-0：世界模型驱动的 VLA 模型 | Python | +| 2211 | [SkyworkAI/Matrix-Game](https://github.com/SkyworkAI/Matrix-Game) | Matrix-Game 3.0：实时流式、长时序记忆的交互世界模型 | Python | +| 1199 | [Robbyant/lingbot-va](https://github.com/Robbyant/lingbot-va) | RSS 2026：因果视频-动作世界模型，面向通用机器人控制 | Python | +| 1045 | [danijar/dreamerv2](https://github.com/danijar/dreamerv2) | DreamerV2：用离散世界模型掌握 Atari | Python | +| 881 | [OpenDriveLab/Vista](https://github.com/OpenDriveLab/Vista) | NeurIPS 2024：可泛化的自动驾驶世界模型 | Python | +| 871 | [NVIDIA/DreamDojo](https://github.com/NVIDIA/DreamDojo) | NVIDIA：基于大规模人类视频的通用机器人世界模型 | Python | +| 579 | [IamCreateAI/NeoVerse](https://github.com/IamCreateAI/NeoVerse) | CVPR 2026 Highlight：用自然视频增强 4D 世界模型 | Python | +| 570 | [JeffWang987/DriveDreamer](https://github.com/JeffWang987/DriveDreamer) | ECCV 2024：面向真实世界驱动的 AD 世界模型 | Python | +| 548 | [wzzheng/OccWorld](https://github.com/wzzheng/OccWorld) | ECCV 2024：3D Occupancy 自动驾驶世界模型 | Python | +| 536 | [maitrix-org/Pandora](https://github.com/maitrix-org/Pandora) | Pandora：以自然语言动作和视频状态为接口的通用世界模型 | Python | +| 433 | [BraveGroup/Drive-WM](https://github.com/BraveGroup/Drive-WM) | CVPR 2024：自动驾驶世界模型 | Python | + +### 6.2 综述 / Awesome List + +| ⭐ Stars | 仓库 | 简介 | +|---------|------|------| +| 2847 | [knightnemo/Awesome-World-Models](https://github.com/knightnemo/Awesome-World-Models) | 世界建模工作的精选汇总，研究/工程一站式资源 | +| 2061 | [LMD0311/Awesome-World-Model](https://github.com/LMD0311/Awesome-World-Model) | 自动驾驶（及机器人）世界模型论文合集 | +| 1818 | [Thinklab-SJTU/Awesome-LLM4AD](https://github.com/Thinklab-SJTU/Awesome-LLM4AD) | 自动驾驶 LLM/VLM/VLA/世界模型资源精选 | +| 1668 | [leofan90/Awesome-World-Models](https://github.com/leofan90/Awesome-World-Models) | 世界模型定义 + 视频生成 + 具身应用综合 paper list | +| 906 | [worldbench/awesome-3d-4d-world-models](https://github.com/worldbench/awesome-3d-4d-world-models) | 3D 与 4D 世界建模 survey | +| 548 | [gracezhao1997/Awesome-Video-World-Models-with-AR-Diffusion](https://github.com/gracezhao1997/Awesome-Video-World-Models-with-AR-Diffusion) | 视频世界模型 + AR Diffusion 算法/应用/基础设施 | +| 521 | [HaoranZhuExplorer/World-Models-Autonomous-Driving-Survey](https://github.com/HaoranZhuExplorer/World-Models-Autonomous-Driving-Survey) | 自动驾驶世界模型综述 | +| 480 | [ziqihuangg/Awesome-From-Video-Generation-to-World-Model](https://github.com/ziqihuangg/Awesome-From-Video-Generation-to-World-Model) | 视频生成走向世界模型的演进路径 | +| 452 | [NTUMARS/Awesome-World-Model-for-Robotics-Policy](https://github.com/NTUMARS/Awesome-World-Model-for-Robotics-Policy) | 机器人策略中的世界模型 | + +### 6.3 Dreamer 相关与扩展 + +| ⭐ Stars | 仓库 | 简介 | +|---------|------|------| +| 1522 | [luciddreamer-cvlab/LucidDreamer](https://github.com/luciddreamer-cvlab/LucidDreamer) | 域无关 3D Gaussian Splatting 场景生成 | +| 1037 | [liuyuan-pal/SyncDreamer](https://github.com/liuyuan-pal/SyncDreamer) | ICLR 2024：从单张图像生成多视角一致图像 | + +--- + +## 7. 趋势分析与小结 + +### 7.1 主线趋势 + +| 维度 | 2024 主流 | 2025–2026 新趋势 | +|------|-----------|-----------------| +| **几何表示** | NeRF、Voxel、BEV | 3D Gaussian、Occupancy、双目 RGB 几何 | +| **生成范式** | 像素扩散 | latent 扩散、Flow Matching、AR Diffusion | +| **时序范围** | 短时单帧/小段 | 长时 latent rollout、持久世界记忆 | +| **能力组合** | 仅生成 | 重建 + 生成 + 理解三体合一 | +| **物理保真** | 弱物理先验 | 可微物理引擎、物理保真损失、广义交互基准 | +| **应用形态** | 离线视频合成 | 在线闭环策略评估、VLA + 世界模型 RL | +| **多智能体** | 单 agent | 多 agent 共享世界 + 对话对齐 | + +### 7.2 五大方向"突破点" + +1. **视频生成**：从"会画"到"会推理物理"——`PhyWorld`、`ACWM-Phys`、`LiveWorld` 都在攻克"视野外动态、长时序一致、物理保真"。 +2. **自动驾驶**：从单一生成到**联合重建-生成-理解**——`HERMES++`、`Xiaomi EV`、`DLWM` 都把多任务、多视角、多模态揉到一个框架。 +3. **具身智能**：**World-Ego 解耦**、**稀疏视觉预测**、**可微物理引擎** 三大新范式（`OrbiSim` 直接把世界模型 = 可微物理引擎）。 +4. **强化学习**：**Diffusion + Latent + VLA + 异步 RL** 四体合一；`DreamerAD` 把扩散采样压到 1 步实现 80× 加速，是工程化里程碑。 +5. **3D/4D 场景**：3D Gaussian 占据主导，**持久场景记忆 + System 2 lookahead planning** 开始进入主流视野（`3D-ALP`）。 + +### 7.3 与项目内部研究的关联 + +- 与 [`PRISM/19_v2_brain_inspired_upgrade.md`](../../plans/PRISM/19_v2_brain_inspired_upgrade.md:1) 脑启发空间记忆 ↔ **3D-ALP** 持久场景记忆 + System 2 推理。 +- 与 [`plans/camera/zed2i_iterative_framework.md`](../../plans/camera/zed2i_iterative_framework.md:1) 双目方案 ↔ **StereoWorld** 的纯 RGB 双目几何监督。 +- 与 [`Lyra 2.0 综述`](lyra2_review.md:1) 可探索 3D 世界 ↔ **HunyuanWorld-1.0**、**NeoVerse**、**InfiniCube** 形成对照矩阵。 +- 与 [`JEPA 系列调研`](jepa/index.md:1) 非生成式表征 ↔ **JEDI** / **Pandora** 是同源问题的不同解法。 + +### 7.4 后续关注路标 + +| 关键词 | 为什么值得跟 | 推荐起点 | +|--------|-------------|----------| +| **OrbiSim** 路线 | 可微物理引擎统一仿真与策略学习 | [arXiv:2605.16395](https://arxiv.org/abs/2605.16395) | +| **Matrix-Game 3.0** | 实时流式 + 长时序记忆，工程化范本 | [SkyworkAI/Matrix-Game](https://github.com/SkyworkAI/Matrix-Game) | +| **DreamerAD** | latent 世界模型驱动 AD RL 的 1 步扩散 | [arXiv:2603.24587](https://arxiv.org/abs/2603.24587) | +| **Xiaomi EV World Model** | 量产车企的世界模型技术栈样本 | [arXiv:2605.18137](https://arxiv.org/abs/2605.18137) | +| **HunyuanWorld-1.0** | 国产开源 3D 可探索世界模型 | [Tencent-Hunyuan/HunyuanWorld-1.0](https://github.com/Tencent-Hunyuan/HunyuanWorld-1.0) | +| **WorldArena 2.0** | 跨模态/平台的具身世界模型评测体系 | [arXiv:2605.17912](https://arxiv.org/abs/2605.17912) | + +--- + +## 8. 数据来源与可复现性 + +### 8.1 检索范围 + +- **时间窗**：2024-03 至 2026-05-19（以 2025–2026 为主，2024 个别奠基性工作保留） +- **arXiv**：5 个主题查询 × 8 篇/主题 = 40 篇（按 `submittedDate` 降序） +- **GitHub**：6 类查询累计 31 个独立仓库（去重，按 stars 降序） + +### 8.2 复现要点 + +- arXiv API 不接受复杂的 `all:"x" AND (all:"y" OR all:"z")` 查询（会返回 HTTP 400），改用 `ti:world+model+` 这种**简单 token 拼接**才能成功。 +- GitHub API 必须用**带引号的短语** `"world model"` + `stars:>50` 过滤，否则会被 `awesome-*` 通用项目刷榜。 +- arXiv 限速 ≥ 3 秒/查询；GitHub 未登录限速 60 次/小时（推荐 `GITHUB_TOKEN`）。 +- 完整脚本参考 [`research/tools/search_info.py`](../tools/search_info.py:1)。 + +### 8.3 数据文件 + +| 文件 | 内容 | +|------|------| +| [`research/data/world_models_arxiv_2025.json`](../data/world_models_arxiv_2025.json:1) | 5 主题 × 8 论文：标题、作者、摘要、arXiv ID、发表日期、分类 | +| [`research/data/world_models_github_2025.json`](../data/world_models_github_2025.json:1) | 6 类查询的 GitHub 仓库元数据（stars/forks/语言/最后更新） | + +--- + +> **生成时间**：2026-05-20 +> **维护者**：Zoo Research Agent +> **下次刷新建议**：每季度更新一次，关注 NeurIPS / CVPR / ICLR / CoRL 大会窗口 \ No newline at end of file