gaojie/worldmodel
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore: initial commit — import worldmodel workspace (plans/, research/)

Browse Source
This commit is contained in:
gaojie
2026-05-20 21:43:57 +08:00
commit bec8a9a4a3
98 changed files with 44128 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
plans/PRISM/.research/arxiv_data.json
+162
View File
@@ -0,0 +1,162 @@
{
"concept-graphs": {
"title": "ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning",
"summary": "For robots to perform a wide variety of tasks, they require a 3D representation of the world that is semantically rich, yet compact and efficient for task-driven perception and planning. Recent approaches have attempted to leverage features from large vision-language models to encode semantics in 3D representations. However, these approaches tend to produce maps with per-point feature vectors, which do not scale well in larger environments, nor do they contain semantic spatial relationships between entities in the environment, which are useful for downstream planning. In this work, we propose ConceptGraphs, an open-vocabulary graph-structured representation for 3D scenes. ConceptGraphs is built by leveraging 2D foundation models and fusing their output to 3D by multi-view association. The resulting representations generalize to novel semantic classes, without the need to collect large 3D datasets or finetune models. We demonstrate the utility of this representation through a number of downstream planning tasks that are specified through abstract (language) prompts and require complex reasoning over spatial and semantic concepts. (Project page: https://concept-graphs.github.io/ Explainer video: https://youtu.be/mRhNkQwRYnc )",
"published": "2023-09-28T17:53:38Z",
"updated": "2023-09-28T17:53:38Z",
"authors": [
"Qiao Gu",
"Alihusein Kuwajerwala",
"Sacha Morin",
"Krishna Murthy Jatavallabhula",
"Bipasha Sen",
"Aditya Agarwal",
"Corban Rivera",
"William Paul",
"Kirsty Ellis",
"Rama Chellappa",
"Chuang Gan",
"Celso Miguel de Melo",
"Joshua B. Tenenbaum",
"Antonio Torralba",
"Florian Shkurti",
"Liam Paull"
],
"primary_cat": "cs.RO",
"journal_ref": "",
"doi": ""
},
"hov-sg": {
"title": "Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation",
"summary": "Recent open-vocabulary robot mapping methods enrich dense geometric maps with pre-trained visual-language features. While these maps allow for the prediction of point-wise saliency maps when queried for a certain language concept, large-scale environments and abstract queries beyond the object level still pose a considerable hurdle, ultimately limiting language-grounded robotic navigation. In this work, we present HOV-SG, a hierarchical open-vocabulary 3D scene graph mapping approach for language-grounded robot navigation. Leveraging open-vocabulary vision foundation models, we first obtain state-of-the-art open-vocabulary segment-level maps in 3D and subsequently construct a 3D scene graph hierarchy consisting of floor, room, and object concepts, each enriched with open-vocabulary features. Our approach is able to represent multi-story buildings and allows robotic traversal of those using a cross-floor Voronoi graph. HOV-SG is evaluated on three distinct datasets and surpasses previous baselines in open-vocabulary semantic accuracy on the object, room, and floor level while producing a 75% reduction in representation size compared to dense open-vocabulary maps. In order to prove the efficacy and generalization capabilities of HOV-SG, we showcase successful long-horizon language-conditioned robot navigation within real-world multi-storage environments. We provide code and trial video data at http://hovsg.github.io/.",
"published": "2024-03-26T16:36:43Z",
"updated": "2024-06-03T17:12:25Z",
"authors": [
"Abdelrhman Werby",
"Chenguang Huang",
"Martin Büchner",
"Abhinav Valada",
"Wolfram Burgard"
],
"primary_cat": "cs.RO",
"journal_ref": "",
"doi": "10.15607/RSS.2024.XX.077"
},
"openscene": {
"title": "OpenScene: 3D Scene Understanding with Open Vocabularies",
"summary": "Traditional 3D scene understanding approaches rely on labeled 3D datasets to train a model for a single task with supervision. We propose OpenScene, an alternative approach where a model predicts dense features for 3D scene points that are co-embedded with text and image pixels in CLIP feature space. This zero-shot approach enables task-agnostic training and open-vocabulary queries. For example, to perform SOTA zero-shot 3D semantic segmentation it first infers CLIP features for every 3D point and later classifies them based on similarities to embeddings of arbitrary class labels. More interestingly, it enables a suite of open-vocabulary scene understanding applications that have never been done before. For example, it allows a user to enter an arbitrary text query and then see a heat map indicating which parts of a scene match. Our approach is effective at identifying objects, materials, affordances, activities, and room types in complex 3D scenes, all using a single model trained without any labeled 3D data.",
"published": "2022-11-28T18:58:36Z",
"updated": "2023-04-06T15:35:13Z",
"authors": [
"Songyou Peng",
"Kyle Genova",
"Chiyu \"Max\" Jiang",
"Andrea Tagliasacchi",
"Marc Pollefeys",
"Thomas Funkhouser"
],
"primary_cat": "cs.CV",
"journal_ref": "",
"doi": ""
},
"3d-llm": {
"title": "3D-LLM: Injecting the 3D World into Large Language Models",
"summary": "Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs. Project Page: : https://vis-www.cs.umass.edu/3dllm/.",
"published": "2023-07-24T17:59:02Z",
"updated": "2023-07-24T17:59:02Z",
"authors": [
"Yining Hong",
"Haoyu Zhen",
"Peihao Chen",
"Shuhong Zheng",
"Yilun Du",
"Zhenfang Chen",
"Chuang Gan"
],
"primary_cat": "cs.CV",
"journal_ref": "",
"doi": ""
},
"ok-robot": {
"title": "OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Robotics",
"summary": "Remarkable progress has been made in recent years in the fields of vision, language, and robotics. We now have vision models capable of recognizing objects based on language queries, navigation systems that can effectively control mobile systems, and grasping models that can handle a wide range of objects. Despite these advancements, general-purpose applications of robotics still lag behind, even though they rely on these fundamental capabilities of recognition, navigation, and grasping. In this paper, we adopt a systems-first approach to develop a new Open Knowledge-based robotics framework called OK-Robot. By combining Vision-Language Models (VLMs) for object detection, navigation primitives for movement, and grasping primitives for object manipulation, OK-Robot offers a integrated solution for pick-and-drop operations without requiring any training. To evaluate its performance, we run OK-Robot in 10 real-world home environments. The results demonstrate that OK-Robot achieves a 58.5% success rate in open-ended pick-and-drop tasks, representing a new state-of-the-art in Open Vocabulary Mobile Manipulation (OVMM) with nearly 1.8x the performance of prior work. On cleaner, uncluttered environments, OK-Robot's performance increases to 82%. However, the most important insight gained from OK-Robot is the critical role of nuanced details when combining Open Knowledge systems like VLMs with robotic modules. Videos of our experiments and code are available on our website: https://ok-robot.github.io",
"published": "2024-01-22T18:42:20Z",
"updated": "2024-02-29T17:20:08Z",
"authors": [
"Peiqi Liu",
"Yaswanth Orru",
"Jay Vakil",
"Chris Paxton",
"Nur Muhammad Mahi Shafiullah",
"Lerrel Pinto"
],
"primary_cat": "cs.RO",
"journal_ref": "",
"doi": "10.15607/RSS.2024.XX.091"
},
"openmask3d": {
"title": "OpenMask3D: Open-Vocabulary 3D Instance Segmentation",
"summary": "We introduce the task of open-vocabulary 3D instance segmentation. Current approaches for 3D instance segmentation can typically only recognize object categories from a pre-defined closed set of classes that are annotated in the training datasets. This results in important limitations for real-world applications where one might need to perform tasks guided by novel, open-vocabulary queries related to a wide variety of objects. Recently, open-vocabulary 3D scene understanding methods have emerged to address this problem by learning queryable features for each point in the scene. While such a representation can be directly employed to perform semantic segmentation, existing methods cannot separate multiple object instances. In this work, we address this limitation, and propose OpenMask3D, which is a zero-shot approach for open-vocabulary 3D instance segmentation. Guided by predicted class-agnostic 3D instance masks, our model aggregates per-mask features via multi-view fusion of CLIP-based image embeddings. Experiments and ablation studies on ScanNet200 and Replica show that OpenMask3D outperforms other open-vocabulary methods, especially on the long-tail distribution. Qualitative experiments further showcase OpenMask3D's ability to segment object properties based on free-form queries describing geometry, affordances, and materials.",
"published": "2023-06-23T17:36:44Z",
"updated": "2023-10-29T14:04:25Z",
"authors": [
"Ayça Takmaz",
"Elisabetta Fedele",
"Robert W. Sumner",
"Marc Pollefeys",
"Federico Tombari",
"Francis Engelmann"
],
"primary_cat": "cs.CV",
"journal_ref": "NeurIPS 2023",
"doi": ""
},
"conceptfusion": {
"title": "ConceptFusion: Open-set Multimodal 3D Mapping",
"summary": "Building 3D maps of the environment is central to robot navigation, planning, and interaction with objects in a scene. Most existing approaches that integrate semantic concepts with 3D maps largely remain confined to the closed-set setting: they can only reason about a finite set of concepts, pre-defined at training time. Further, these maps can only be queried using class labels, or in recent work, using text prompts. We address both these issues with ConceptFusion, a scene representation that is (1) fundamentally open-set, enabling reasoning beyond a closed set of concepts and (ii) inherently multimodal, enabling a diverse range of possible queries to the 3D map, from language, to images, to audio, to 3D geometry, all working in concert. ConceptFusion leverages the open-set capabilities of today's foundation models pre-trained on internet-scale data to reason about concepts across modalities such as natural language, images, and audio. We demonstrate that pixel-aligned open-set features can be fused into 3D maps via traditional SLAM and multi-view fusion approaches. This enables effective zero-shot spatial reasoning, not needing any additional training or finetuning, and retains long-tailed concepts better than supervised approaches, outperforming them by more than 40% margin on 3D IoU. We extensively evaluate ConceptFusion on a number of real-world datasets, simulated home environments, a real-world tabletop manipulation task, and an autonomous driving platform. We showcase new avenues for blending foundation models with 3D open-set multimodal mapping. For more information, visit our project page https://concept-fusion.github.io or watch our 5-minute explainer video https://www.youtube.com/watch?v=rkXgws8fiDs",
"published": "2023-02-14T18:40:26Z",
"updated": "2023-10-23T14:56:15Z",
"authors": [
"Krishna Murthy Jatavallabhula",
"Alihusein Kuwajerwala",
"Qiao Gu",
"Mohd Omama",
"Tao Chen",
"Alaa Maalouf",
"Shuang Li",
"Ganesh Iyer",
"Soroush Saryazdi",
"Nikhil Keetha",
"Ayush Tewari",
"Joshua B. Tenenbaum",
"Celso Miguel de Melo",
"Madhava Krishna",
"Liam Paull",
"Florian Shkurti",
"Antonio Torralba"
],
"primary_cat": "cs.CV",
"journal_ref": "",
"doi": ""
},
"clio": {
"title": "Clio: Real-time Task-Driven Open-Set 3D Scene Graphs",
"summary": "Modern tools for class-agnostic image segmentation (e.g., SegmentAnything) and open-set semantic understanding (e.g., CLIP) provide unprecedented opportunities for robot perception and mapping. While traditional closed-set metric-semantic maps were restricted to tens or hundreds of semantic classes, we can now build maps with a plethora of objects and countless semantic variations. This leaves us with a fundamental question: what is the right granularity for the objects (and, more generally, for the semantic concepts) the robot has to include in its map representation? While related work implicitly chooses a level of granularity by tuning thresholds for object detection, we argue that such a choice is intrinsically task-dependent. The first contribution of this paper is to propose a task-driven 3D scene understanding problem, where the robot is given a list of tasks in natural language and has to select the granularity and the subset of objects and scene structure to retain in its map that is sufficient to complete the tasks. We show that this problem can be naturally formulated using the Information Bottleneck (IB), an established information-theoretic framework. The second contribution is an algorithm for task-driven 3D scene understanding based on an Agglomerative IB approach, that is able to cluster 3D primitives in the environment into task-relevant objects and regions and executes incrementally. The third contribution is to integrate our task-driven clustering algorithm into a real-time pipeline, named Clio, that constructs a hierarchical 3D scene graph of the environment online using only onboard compute, as the robot explores it. Our final contribution is an extensive experimental campaign showing that Clio not only allows real-time construction of compact open-set 3D scene graphs, but also improves the accuracy of task execution by limiting the map to relevant semantic concepts.",
"published": "2024-04-21T15:50:40Z",
"updated": "2024-09-26T20:34:51Z",
"authors": [
"Dominic Maggio",
"Yun Chang",
"Nathan Hughes",
"Matthew Trang",
"Dan Griffith",
"Carlyn Dougherty",
"Eric Cristofalo",
"Lukas Schmid",
"Luca Carlone"
],
"primary_cat": "cs.RO",
"journal_ref": "",
"doi": ""
}
}