gaojie/worldmodel
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore: initial commit — import worldmodel workspace (plans/, research/)

Browse Source
This commit is contained in:
gaojie
2026-05-20 21:43:57 +08:00
commit bec8a9a4a3
98 changed files with 44128 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
plans/PRISM/.research/arxiv_data.json
+162
View File
@@ -0,0 +1,162 @@
{
"concept-graphs": {
"title": "ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning",
"summary": "For robots to perform a wide variety of tasks, they require a 3D representation of the world that is semantically rich, yet compact and efficient for task-driven perception and planning. Recent approaches have attempted to leverage features from large vision-language models to encode semantics in 3D representations. However, these approaches tend to produce maps with per-point feature vectors, which do not scale well in larger environments, nor do they contain semantic spatial relationships between entities in the environment, which are useful for downstream planning. In this work, we propose ConceptGraphs, an open-vocabulary graph-structured representation for 3D scenes. ConceptGraphs is built by leveraging 2D foundation models and fusing their output to 3D by multi-view association. The resulting representations generalize to novel semantic classes, without the need to collect large 3D datasets or finetune models. We demonstrate the utility of this representation through a number of downstream planning tasks that are specified through abstract (language) prompts and require complex reasoning over spatial and semantic concepts. (Project page: https://concept-graphs.github.io/ Explainer video: https://youtu.be/mRhNkQwRYnc )",
"published": "2023-09-28T17:53:38Z",
"updated": "2023-09-28T17:53:38Z",
"authors": [
"Qiao Gu",
"Alihusein Kuwajerwala",
"Sacha Morin",
"Krishna Murthy Jatavallabhula",
"Bipasha Sen",
"Aditya Agarwal",
"Corban Rivera",
"William Paul",
"Kirsty Ellis",
"Rama Chellappa",
"Chuang Gan",
"Celso Miguel de Melo",
"Joshua B. Tenenbaum",
"Antonio Torralba",
"Florian Shkurti",
"Liam Paull"
],
"primary_cat": "cs.RO",
"journal_ref": "",
"doi": ""
},
"hov-sg": {
"title": "Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation",
"summary": "Recent open-vocabulary robot mapping methods enrich dense geometric maps with pre-trained visual-language features. While these maps allow for the prediction of point-wise saliency maps when queried for a certain language concept, large-scale environments and abstract queries beyond the object level still pose a considerable hurdle, ultimately limiting language-grounded robotic navigation. In this work, we present HOV-SG, a hierarchical open-vocabulary 3D scene graph mapping approach for language-grounded robot navigation. Leveraging open-vocabulary vision foundation models, we first obtain state-of-the-art open-vocabulary segment-level maps in 3D and subsequently construct a 3D scene graph hierarchy consisting of floor, room, and object concepts, each enriched with open-vocabulary features. Our approach is able to represent multi-story buildings and allows robotic traversal of those using a cross-floor Voronoi graph. HOV-SG is evaluated on three distinct datasets and surpasses previous baselines in open-vocabulary semantic accuracy on the object, room, and floor level while producing a 75% reduction in representation size compared to dense open-vocabulary maps. In order to prove the efficacy and generalization capabilities of HOV-SG, we showcase successful long-horizon language-conditioned robot navigation within real-world multi-storage environments. We provide code and trial video data at http://hovsg.github.io/.",
"published": "2024-03-26T16:36:43Z",
"updated": "2024-06-03T17:12:25Z",
"authors": [
"Abdelrhman Werby",
"Chenguang Huang",
"Martin Büchner",
"Abhinav Valada",
"Wolfram Burgard"
],
"primary_cat": "cs.RO",
"journal_ref": "",
"doi": "10.15607/RSS.2024.XX.077"
},
"openscene": {
"title": "OpenScene: 3D Scene Understanding with Open Vocabularies",
"summary": "Traditional 3D scene understanding approaches rely on labeled 3D datasets to train a model for a single task with supervision. We propose OpenScene, an alternative approach where a model predicts dense features for 3D scene points that are co-embedded with text and image pixels in CLIP feature space. This zero-shot approach enables task-agnostic training and open-vocabulary queries. For example, to perform SOTA zero-shot 3D semantic segmentation it first infers CLIP features for every 3D point and later classifies them based on similarities to embeddings of arbitrary class labels. More interestingly, it enables a suite of open-vocabulary scene understanding applications that have never been done before. For example, it allows a user to enter an arbitrary text query and then see a heat map indicating which parts of a scene match. Our approach is effective at identifying objects, materials, affordances, activities, and room types in complex 3D scenes, all using a single model trained without any labeled 3D data.",
"published": "2022-11-28T18:58:36Z",
"updated": "2023-04-06T15:35:13Z",
"authors": [
"Songyou Peng",
"Kyle Genova",
"Chiyu \"Max\" Jiang",
"Andrea Tagliasacchi",
"Marc Pollefeys",
"Thomas Funkhouser"
],
"primary_cat": "cs.CV",
"journal_ref": "",
"doi": ""
},
"3d-llm": {
"title": "3D-LLM: Injecting the 3D World into Large Language Models",
"summary": "Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs. Project Page: : https://vis-www.cs.umass.edu/3dllm/.",
"published": "2023-07-24T17:59:02Z",
"updated": "2023-07-24T17:59:02Z",
"authors": [
"Yining Hong",
"Haoyu Zhen",
"Peihao Chen",
"Shuhong Zheng",
"Yilun Du",
"Zhenfang Chen",
"Chuang Gan"
],
"primary_cat": "cs.CV",
"journal_ref": "",
"doi": ""
},
"ok-robot": {
"title": "OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Robotics",
"summary": "Remarkable progress has been made in recent years in the fields of vision, language, and robotics. We now have vision models capable of recognizing objects based on language queries, navigation systems that can effectively control mobile systems, and grasping models that can handle a wide range of objects. Despite these advancements, general-purpose applications of robotics still lag behind, even though they rely on these fundamental capabilities of recognition, navigation, and grasping. In this paper, we adopt a systems-first approach to develop a new Open Knowledge-based robotics framework called OK-Robot. By combining Vision-Language Models (VLMs) for object detection, navigation primitives for movement, and grasping primitives for object manipulation, OK-Robot offers a integrated solution for pick-and-drop operations without requiring any training. To evaluate its performance, we run OK-Robot in 10 real-world home environments. The results demonstrate that OK-Robot achieves a 58.5% success rate in open-ended pick-and-drop tasks, representing a new state-of-the-art in Open Vocabulary Mobile Manipulation (OVMM) with nearly 1.8x the performance of prior work. On cleaner, uncluttered environments, OK-Robot's performance increases to 82%. However, the most important insight gained from OK-Robot is the critical role of nuanced details when combining Open Knowledge systems like VLMs with robotic modules. Videos of our experiments and code are available on our website: https://ok-robot.github.io",
"published": "2024-01-22T18:42:20Z",
"updated": "2024-02-29T17:20:08Z",
"authors": [
"Peiqi Liu",
"Yaswanth Orru",
"Jay Vakil",
"Chris Paxton",
"Nur Muhammad Mahi Shafiullah",
"Lerrel Pinto"
],
"primary_cat": "cs.RO",
"journal_ref": "",
"doi": "10.15607/RSS.2024.XX.091"
},
"openmask3d": {
"title": "OpenMask3D: Open-Vocabulary 3D Instance Segmentation",
"summary": "We introduce the task of open-vocabulary 3D instance segmentation. Current approaches for 3D instance segmentation can typically only recognize object categories from a pre-defined closed set of classes that are annotated in the training datasets. This results in important limitations for real-world applications where one might need to perform tasks guided by novel, open-vocabulary queries related to a wide variety of objects. Recently, open-vocabulary 3D scene understanding methods have emerged to address this problem by learning queryable features for each point in the scene. While such a representation can be directly employed to perform semantic segmentation, existing methods cannot separate multiple object instances. In this work, we address this limitation, and propose OpenMask3D, which is a zero-shot approach for open-vocabulary 3D instance segmentation. Guided by predicted class-agnostic 3D instance masks, our model aggregates per-mask features via multi-view fusion of CLIP-based image embeddings. Experiments and ablation studies on ScanNet200 and Replica show that OpenMask3D outperforms other open-vocabulary methods, especially on the long-tail distribution. Qualitative experiments further showcase OpenMask3D's ability to segment object properties based on free-form queries describing geometry, affordances, and materials.",
"published": "2023-06-23T17:36:44Z",
"updated": "2023-10-29T14:04:25Z",
"authors": [
"Ayça Takmaz",
"Elisabetta Fedele",
"Robert W. Sumner",
"Marc Pollefeys",
"Federico Tombari",
"Francis Engelmann"
],
"primary_cat": "cs.CV",
"journal_ref": "NeurIPS 2023",
"doi": ""
},
"conceptfusion": {
"title": "ConceptFusion: Open-set Multimodal 3D Mapping",
"summary": "Building 3D maps of the environment is central to robot navigation, planning, and interaction with objects in a scene. Most existing approaches that integrate semantic concepts with 3D maps largely remain confined to the closed-set setting: they can only reason about a finite set of concepts, pre-defined at training time. Further, these maps can only be queried using class labels, or in recent work, using text prompts. We address both these issues with ConceptFusion, a scene representation that is (1) fundamentally open-set, enabling reasoning beyond a closed set of concepts and (ii) inherently multimodal, enabling a diverse range of possible queries to the 3D map, from language, to images, to audio, to 3D geometry, all working in concert. ConceptFusion leverages the open-set capabilities of today's foundation models pre-trained on internet-scale data to reason about concepts across modalities such as natural language, images, and audio. We demonstrate that pixel-aligned open-set features can be fused into 3D maps via traditional SLAM and multi-view fusion approaches. This enables effective zero-shot spatial reasoning, not needing any additional training or finetuning, and retains long-tailed concepts better than supervised approaches, outperforming them by more than 40% margin on 3D IoU. We extensively evaluate ConceptFusion on a number of real-world datasets, simulated home environments, a real-world tabletop manipulation task, and an autonomous driving platform. We showcase new avenues for blending foundation models with 3D open-set multimodal mapping. For more information, visit our project page https://concept-fusion.github.io or watch our 5-minute explainer video https://www.youtube.com/watch?v=rkXgws8fiDs",
"published": "2023-02-14T18:40:26Z",
"updated": "2023-10-23T14:56:15Z",
"authors": [
"Krishna Murthy Jatavallabhula",
"Alihusein Kuwajerwala",
"Qiao Gu",
"Mohd Omama",
"Tao Chen",
"Alaa Maalouf",
"Shuang Li",
"Ganesh Iyer",
"Soroush Saryazdi",
"Nikhil Keetha",
"Ayush Tewari",
"Joshua B. Tenenbaum",
"Celso Miguel de Melo",
"Madhava Krishna",
"Liam Paull",
"Florian Shkurti",
"Antonio Torralba"
],
"primary_cat": "cs.CV",
"journal_ref": "",
"doi": ""
},
"clio": {
"title": "Clio: Real-time Task-Driven Open-Set 3D Scene Graphs",
"summary": "Modern tools for class-agnostic image segmentation (e.g., SegmentAnything) and open-set semantic understanding (e.g., CLIP) provide unprecedented opportunities for robot perception and mapping. While traditional closed-set metric-semantic maps were restricted to tens or hundreds of semantic classes, we can now build maps with a plethora of objects and countless semantic variations. This leaves us with a fundamental question: what is the right granularity for the objects (and, more generally, for the semantic concepts) the robot has to include in its map representation? While related work implicitly chooses a level of granularity by tuning thresholds for object detection, we argue that such a choice is intrinsically task-dependent. The first contribution of this paper is to propose a task-driven 3D scene understanding problem, where the robot is given a list of tasks in natural language and has to select the granularity and the subset of objects and scene structure to retain in its map that is sufficient to complete the tasks. We show that this problem can be naturally formulated using the Information Bottleneck (IB), an established information-theoretic framework. The second contribution is an algorithm for task-driven 3D scene understanding based on an Agglomerative IB approach, that is able to cluster 3D primitives in the environment into task-relevant objects and regions and executes incrementally. The third contribution is to integrate our task-driven clustering algorithm into a real-time pipeline, named Clio, that constructs a hierarchical 3D scene graph of the environment online using only onboard compute, as the robot explores it. Our final contribution is an extensive experimental campaign showing that Clio not only allows real-time construction of compact open-set 3D scene graphs, but also improves the accuracy of task execution by limiting the map to relevant semantic concepts.",
"published": "2024-04-21T15:50:40Z",
"updated": "2024-09-26T20:34:51Z",
"authors": [
"Dominic Maggio",
"Yun Chang",
"Nathan Hughes",
"Matthew Trang",
"Dan Griffith",
"Carlyn Dougherty",
"Eric Cristofalo",
"Lukas Schmid",
"Luca Carlone"
],
"primary_cat": "cs.RO",
"journal_ref": "",
"doi": ""
}
}
plans/PRISM/.research/fetch_arxiv.py
+86
View File
@@ -0,0 +1,86 @@
#!/usr/bin/env python3
"""
fetch_arxiv.py — 从 arXiv 原生 API 抓取每个项目的 title / abstract / year。
arXiv 没有 rate limit 烦恼;Semantic Scholar 429 改这里。
"""
from __future__ import annotations
import json
import pathlib
import re
import sys
import time
import urllib.request
import xml.etree.ElementTree as ET
PROJECTS_JSON = pathlib.Path(__file__).parent / "projects.json"
OUT = pathlib.Path(__file__).parent / "arxiv_data.json"
NS = {"a": "http://www.w3.org/2005/Atom",
"arxiv": "http://arxiv.org/schemas/atom"}
UA = "Mozilla/5.0 (research) PRISM-comparison/1.0"
def get(url: str, timeout: int = 30) -> str:
req = urllib.request.Request(url, headers={"User-Agent": UA})
with urllib.request.urlopen(req, timeout=timeout) as r:
return r.read().decode("utf-8", errors="replace")
def fetch_one(arxiv_id: str) -> dict:
"""arXiv id 形如 2309.16650 (可以带 v1 后缀,但去掉更稳)"""
aid = re.sub(r"v\d+$", "", arxiv_id)
url = f"http://export.arxiv.org/api/query?id_list={aid}"
try:
xml = get(url)
except Exception as e:
return {"_error": f"http: {e}"}
try:
root = ET.fromstring(xml)
entry = root.find("a:entry", NS)
if entry is None:
return {"_error": "no entry"}
title = (entry.findtext("a:title", default="", namespaces=NS) or "").strip()
summary = (entry.findtext("a:summary", default="", namespaces=NS) or "").strip()
published = (entry.findtext("a:published", default="", namespaces=NS) or "").strip()
updated = (entry.findtext("a:updated", default="", namespaces=NS) or "").strip()
authors = [
(a.findtext("a:name", default="", namespaces=NS) or "").strip()
for a in entry.findall("a:author", NS)
]
# primary category
prim = entry.find("arxiv:primary_category", NS)
primary = prim.get("term") if prim is not None else None
# journal-ref (= venue if author filled)
jref = entry.findtext("arxiv:journal_ref", default="", namespaces=NS)
# doi
doi = entry.findtext("arxiv:doi", default="", namespaces=NS)
return {
"title": re.sub(r"\s+", " ", title),
"summary": re.sub(r"\s+", " ", summary),
"published": published,
"updated": updated,
"authors": authors,
"primary_cat": primary,
"journal_ref": jref,
"doi": doi,
}
except Exception as e:
return {"_error": f"xml: {e}"}
def main() -> None:
data = json.loads(PROJECTS_JSON.read_text(encoding="utf-8"))
out = {}
for key, v in data.items():
aid = v["meta"]["arxiv"]
sys.stderr.write(f"[arxiv] {v['meta']['name']} ({aid}) ...\n")
out[key] = fetch_one(aid)
time.sleep(3.5) # arXiv 建议 ≥ 3s
OUT.write_text(json.dumps(out, ensure_ascii=False, indent=2),
encoding="utf-8")
sys.stderr.write(f"[ok] {OUT}\n")
if __name__ == "__main__":
main()
plans/PRISM/.research/fetch_projects.py
+155
View File
@@ -0,0 +1,155 @@
#!/usr/bin/env python3
"""
fetch_projects.py — 抓取同档比较项目的元数据(arXiv + Semantic Scholar + GitHub)
输出 JSON 到 stdout 与 projects.json,供后续 comparison.md 编写引用。
"""
from __future__ import annotations
import json
import pathlib
import sys
import time
from typing import Any
import urllib.parse
import urllib.request
OUT = pathlib.Path(__file__).parent / "projects.json"
# 待研究项目列表
PROJECTS = [
{
"key": "concept-graphs",
"name": "ConceptGraphs",
"arxiv": "2309.16650",
"gh": "concept-graphs/concept-graphs",
"homepage": "https://concept-graphs.github.io/",
},
{
"key": "hov-sg",
"name": "HOV-SG",
"arxiv": "2403.17846",
"gh": "hovsg/HOV-SG",
"homepage": "https://hovsg.github.io/",
},
{
"key": "openscene",
"name": "OpenScene",
"arxiv": "2211.15654",
"gh": "pengsongyou/openscene",
"homepage": "https://pengsongyou.github.io/openscene",
},
{
"key": "3d-llm",
"name": "3D-LLM",
"arxiv": "2307.12981",
"gh": "UMass-Foundation-Model/3D-LLM",
"homepage": "https://vis-www.cs.umass.edu/3dllm/",
},
{
"key": "ok-robot",
"name": "OK-Robot",
"arxiv": "2401.12202",
"gh": "ok-robot/ok-robot",
"homepage": "https://ok-robot.github.io/",
},
{
"key": "openmask3d",
"name": "OpenMask3D",
"arxiv": "2306.13631",
"gh": "OpenMask3D/openmask3d",
"homepage": "https://openmask3d.github.io/",
},
{
"key": "conceptfusion",
"name": "ConceptFusion",
"arxiv": "2302.07241",
"gh": "concept-fusion/concept-fusion",
"homepage": "https://concept-fusion.github.io/",
},
{
"key": "clio",
"name": "Clio (MIT-SPARK)",
"arxiv": "2404.13696",
"gh": "MIT-SPARK/Clio",
"homepage": "https://clio-cmu.github.io/",
},
]
UA = "Mozilla/5.0 (research) PRISM-comparison/1.0"
def _get(url: str, headers: dict[str, str] | None = None,
timeout: int = 20) -> str:
h = {"User-Agent": UA, "Accept": "application/json"}
if headers:
h.update(headers)
req = urllib.request.Request(url, headers=h)
try:
with urllib.request.urlopen(req, timeout=timeout) as r:
return r.read().decode("utf-8", errors="replace")
except Exception as e:
return json.dumps({"_error": str(e)})
def fetch_arxiv(arxiv_id: str) -> dict[str, Any]:
"""Semantic Scholar 一次查到 abstract、引用数、年份"""
url = (
"https://api.semanticscholar.org/graph/v1/paper/"
f"arXiv:{arxiv_id}?fields=title,abstract,year,citationCount,"
"authors.name,venue,publicationVenue,referenceCount,influentialCitationCount,"
"openAccessPdf,fieldsOfStudy"
)
raw = _get(url)
try:
return json.loads(raw)
except Exception as e:
return {"_error": f"json: {e}", "_raw": raw[:300]}
def fetch_github(repo: str) -> dict[str, Any]:
url = f"https://api.github.com/repos/{repo}"
raw = _get(url, headers={"Accept": "application/vnd.github+json"})
try:
d = json.loads(raw)
# 只保留关心字段
if "stargazers_count" in d:
return {
"stars": d.get("stargazers_count"),
"forks": d.get("forks_count"),
"subscribers": d.get("subscribers_count"),
"language": d.get("language"),
"license": (d.get("license") or {}).get("spdx_id"),
"open_issues": d.get("open_issues_count"),
"created_at": d.get("created_at"),
"updated_at": d.get("updated_at"),
"pushed_at": d.get("pushed_at"),
"topics": d.get("topics"),
"description": d.get("description"),
"default_branch": d.get("default_branch"),
"archived": d.get("archived"),
"homepage": d.get("homepage"),
}
return d
except Exception as e:
return {"_error": f"json: {e}", "_raw": raw[:300]}
def main() -> None:
out: dict[str, Any] = {}
for p in PROJECTS:
sys.stderr.write(f"[fetch] {p['name']} ({p['arxiv']}) ...\n")
out[p["key"]] = {
"meta": p,
"arxiv_via_s2": fetch_arxiv(p["arxiv"]),
}
time.sleep(1.2) # 礼貌:S2 rate limit
out[p["key"]]["github"] = fetch_github(p["gh"])
time.sleep(0.8)
OUT.write_text(json.dumps(out, ensure_ascii=False, indent=2),
encoding="utf-8")
sys.stderr.write(f"[ok] wrote {OUT}\n")
print(f"saved {OUT}")
if __name__ == "__main__":
main()
plans/PRISM/.research/fetch_readmes.py
+60
View File
@@ -0,0 +1,60 @@
#!/usr/bin/env python3
"""
fetch_readmes.py — 抓取每个项目的 README.md 头部 (前 12 KB)
用于推断:支持的平台、依赖、是否提供数据集、demo 视频链接等。
"""
from __future__ import annotations
import json
import pathlib
import re
import sys
import time
import urllib.request
ROOT = pathlib.Path(__file__).parent
PROJECTS_JSON = ROOT / "projects.json"
OUT = ROOT / "readmes"
UA = "Mozilla/5.0 (research) PRISM-comparison/1.0"
def get_text(url: str) -> str:
req = urllib.request.Request(url, headers={"User-Agent": UA})
try:
with urllib.request.urlopen(req, timeout=20) as r:
return r.read().decode("utf-8", errors="replace")
except Exception as e:
return f"<<ERROR {e}>>"
def find_readme(repo: str, branch: str = "main") -> tuple[str, str]:
"""返回 (raw_url, content)。先试 main,再 master。"""
for br in (branch, "master"):
url = f"https://raw.githubusercontent.com/{repo}/{br}/README.md"
txt = get_text(url)
if not txt.startswith("<<ERROR"):
return url, txt
return url, txt
def main() -> None:
data = json.loads(PROJECTS_JSON.read_text(encoding="utf-8"))
OUT.mkdir(exist_ok=True)
for key, v in data.items():
repo = v["meta"]["gh"]
# 部分项目用了非 main 默认分支
default_branch = v.get("github", {}).get("default_branch") or "main"
sys.stderr.write(f"[readme] {v['meta']['name']} <- {repo}@{default_branch} ...\n")
url, txt = find_readme(repo, default_branch)
out_path = OUT / f"{key}.md"
out_path.write_text(f"<!-- src: {url} -->\n\n{txt[:16000]}\n",
encoding="utf-8")
sys.stderr.write(f" -> {out_path} ({len(txt)} chars)\n")
time.sleep(1.0)
sys.stderr.write(f"[ok] all readmes saved to {OUT}\n")
if __name__ == "__main__":
main()
plans/PRISM/.research/projects.json
+247
View File
@@ -0,0 +1,247 @@
{
"concept-graphs": {
"meta": {
"key": "concept-graphs",
"name": "ConceptGraphs",
"arxiv": "2309.16650",
"gh": "concept-graphs/concept-graphs",
"homepage": "https://concept-graphs.github.io/"
},
"arxiv_via_s2": {
"_error": "HTTP Error 429: "
},
"github": {
"stars": 868,
"forks": 124,
"subscribers": 8,
"language": "Python",
"license": "MIT",
"open_issues": 24,
"created_at": "2023-09-28T23:58:12Z",
"updated_at": "2026-05-14T11:33:15Z",
"pushed_at": "2025-10-16T22:09:00Z",
"topics": [],
"description": "Official code release for ConceptGraphs",
"default_branch": "main",
"archived": false,
"homepage": null
}
},
"hov-sg": {
"meta": {
"key": "hov-sg",
"name": "HOV-SG",
"arxiv": "2403.17846",
"gh": "hovsg/HOV-SG",
"homepage": "https://hovsg.github.io/"
},
"arxiv_via_s2": {
"_error": "HTTP Error 429: "
},
"github": {
"stars": 470,
"forks": 33,
"subscribers": 5,
"language": "Python",
"license": "MIT",
"open_issues": 16,
"created_at": "2024-06-07T13:19:38Z",
"updated_at": "2026-05-14T12:23:03Z",
"pushed_at": "2026-01-19T08:58:18Z",
"topics": [
"3d-scene-graph",
"natural-language-understanding",
"open-vocabulary",
"robot-navigation",
"robot-planning"
],
"description": "[RSS2024] Official implementation of \"Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation\"",
"default_branch": "main",
"archived": false,
"homepage": "https://hovsg.github.io"
}
},
"openscene": {
"meta": {
"key": "openscene",
"name": "OpenScene",
"arxiv": "2211.15654",
"gh": "pengsongyou/openscene",
"homepage": "https://pengsongyou.github.io/openscene"
},
"arxiv_via_s2": {
"_error": "HTTP Error 429: "
},
"github": {
"stars": 820,
"forks": 67,
"subscribers": 17,
"language": "Python",
"license": "Apache-2.0",
"open_issues": 17,
"created_at": "2023-03-18T09:29:23Z",
"updated_at": "2026-05-16T03:28:57Z",
"pushed_at": "2023-10-27T15:57:18Z",
"topics": [
"3d-scene-understanding",
"clip",
"cvpr2023",
"llm",
"matterport3d",
"nuscenes",
"point-cloud-segmentation",
"point-clouds",
"scannet",
"semantic-segmentation"
],
"description": "[CVPR'23] OpenScene: 3D Scene Understanding with Open Vocabularies",
"default_branch": "main",
"archived": false,
"homepage": "https://pengsongyou.github.io/openscene"
}
},
"3d-llm": {
"meta": {
"key": "3d-llm",
"name": "3D-LLM",
"arxiv": "2307.12981",
"gh": "UMass-Foundation-Model/3D-LLM",
"homepage": "https://vis-www.cs.umass.edu/3dllm/"
},
"arxiv_via_s2": {
"_error": "HTTP Error 429: "
},
"github": {
"stars": 1196,
"forks": 74,
"subscribers": 18,
"language": "Python",
"license": "MIT",
"open_issues": 31,
"created_at": "2023-07-24T15:21:57Z",
"updated_at": "2026-05-16T07:35:31Z",
"pushed_at": "2024-06-06T19:14:00Z",
"topics": [],
"description": "Code for 3D-LLM: Injecting the 3D World into Large Language Models",
"default_branch": "main",
"archived": false,
"homepage": ""
}
},
"ok-robot": {
"meta": {
"key": "ok-robot",
"name": "OK-Robot",
"arxiv": "2401.12202",
"gh": "ok-robot/ok-robot",
"homepage": "https://ok-robot.github.io/"
},
"arxiv_via_s2": {
"_error": "HTTP Error 429: "
},
"github": {
"stars": 596,
"forks": 42,
"subscribers": 10,
"language": "Python",
"license": "MIT",
"open_issues": 2,
"created_at": "2024-02-22T22:51:59Z",
"updated_at": "2026-05-16T03:11:25Z",
"pushed_at": "2024-03-04T15:56:19Z",
"topics": [
"home-robots",
"open-vocabulary",
"robotics"
],
"description": "An open, modular framework for zero-shot, language conditioned pick-and-drop tasks in arbitrary homes.",
"default_branch": "main",
"archived": false,
"homepage": "https://ok-robot.github.io"
}
},
"openmask3d": {
"meta": {
"key": "openmask3d",
"name": "OpenMask3D",
"arxiv": "2306.13631",
"gh": "OpenMask3D/openmask3d",
"homepage": "https://openmask3d.github.io/"
},
"arxiv_via_s2": {
"_error": "HTTP Error 429: "
},
"github": {
"stars": 259,
"forks": 19,
"subscribers": 8,
"language": "Python",
"license": "MIT",
"open_issues": 8,
"created_at": "2023-10-31T14:58:50Z",
"updated_at": "2026-04-20T01:15:48Z",
"pushed_at": "2023-12-15T20:57:07Z",
"topics": [],
"description": null,
"default_branch": "main",
"archived": false,
"homepage": null
}
},
"conceptfusion": {
"meta": {
"key": "conceptfusion",
"name": "ConceptFusion",
"arxiv": "2302.07241",
"gh": "concept-fusion/concept-fusion",
"homepage": "https://concept-fusion.github.io/"
},
"arxiv_via_s2": {
"_error": "HTTP Error 429: "
},
"github": {
"stars": 233,
"forks": 17,
"subscribers": 5,
"language": null,
"license": "MIT",
"open_issues": 13,
"created_at": "2023-05-21T11:25:39Z",
"updated_at": "2026-04-23T23:50:16Z",
"pushed_at": "2023-09-23T19:36:58Z",
"topics": [],
"description": "Code release for ConceptFusion [RSS 2023]",
"default_branch": "main",
"archived": false,
"homepage": null
}
},
"clio": {
"meta": {
"key": "clio",
"name": "Clio (MIT-SPARK)",
"arxiv": "2404.13696",
"gh": "MIT-SPARK/Clio",
"homepage": "https://clio-cmu.github.io/"
},
"arxiv_via_s2": {
"_error": "HTTP Error 429: "
},
"github": {
"stars": 239,
"forks": 17,
"subscribers": 11,
"language": "Python",
"license": "BSD-2-Clause",
"open_issues": 10,
"created_at": "2024-04-14T19:29:25Z",
"updated_at": "2026-04-29T01:32:29Z",
"pushed_at": "2025-09-01T14:57:43Z",
"topics": [],
"description": null,
"default_branch": "main",
"archived": false,
"homepage": null
}
}
}
plans/PRISM/.research/readmes/3d-llm.md
+273
View File
@@ -0,0 +1,273 @@
<br />
<p align="center">
<h1 align="center">3D-LLM: Injecting the 3D World into Large Language Models (NeurIPS 2023 Spotlight) </h1>
<p align="center">
<a href="https://evelinehong.github.io">Yining Hong</a>,
<a href="https://haoyuzhen.com">Haoyu Zhen</a>,
<a href="https://peihaochen.github.io">Peihao Chen</a>,
<a href="https://zsh2000.github.io">Shuhong Zheng</a>,
<a href="https://yilundu.github.io">Yilun Du</a>,
<a href="https://zfchenunique.github.io">Zhenfang Chen</a>,
<a href="https://people.csail.mit.edu/ganchuang">Chuang Gan</a>
</p>
<p align="center">
<a href='https://arxiv.org/abs/2307.12981'>
<img src='https://img.shields.io/badge/Paper-PDF-red?style=flat&logo=arXiv&logoColor=red' alt='Paper PDF'>
</a>
<a href='https://vis-www.cs.umass.edu/3dllm/' style='padding-left: 0.5rem;'>
<img src='https://img.shields.io/badge/Project-Page-blue?style=flat&logo=Google%20chrome&logoColor=blue' alt='Project Page'>
</a>
</p>
<p align="center">
<img src="figs/pipeline.png" alt="Logo" width="80%">
</p>
</p>
3D-LLM is the first Large Language Model that could take 3D representations as inputs. It is able to handle both object (e.g., objaverse) and scene data (e.g., scannet & hm3d).
## Installation
Install [salesforce-lavis](https://github.com/salesforce/LAVIS)
```shell
$ conda create -n lavis python=3.8
$ conda activate lavis
$ git clone https://github.com/salesforce/LAVIS.git SalesForce-LAVIS
$ cd SalesForce-LAVIS
$ pip install -e .
$ pip install positional_encodings
```
## Checkpoints
### Pretraining Checkpoints
[Pretrained checkpoints](https://drive.google.com/file/d/1tiis8mOdZGBzmR7vgZtRE4Ni_2FE4nTr/view?usp=drive_link) are released (Please use v2!)
### Finetuning Checkpoints
[Finetuning checkpoints](https://drive.google.com/drive/folders/1RKP1cz6R6H8YziEc4f3MHW9dCCXQChbA?usp=drive_link) for [ScanQA](https://drive.google.com/file/d/1sPynAO8pI_RPR4pwWTrx8weDTdMPsqtW/view?usp=drive_link), [SQA3d](https://drive.google.com/file/d/1Ka9TWv6cs6h-pPaaQG1auIiQma2xbNFk/view?usp=drive_link), and [3DMV_VQA](https://drive.google.com/file/d/1_h2wPPGO64HY5LUcA1bD8DlZx3WCsY8b/view?usp=drive_link) are released.
The results are better than preprint-version paper. We will update the camera-ready paper to the arxiv soon.
## Quick Start: Inference
Download the objaverse subset features [here](https://drive.google.com/file/d/1mJZONfWREfIUAPYXP65D65uS2EoplAfR/view?usp=drive_link). Download the [pretrained checkpoints](https://drive.google.com/drive/folders/1urI2I3S8SgLD8L9brl4ae1Mul_yhCxJe?usp=drive_link). For more details, please refer to `3DLLM_BLIP2-base/DEMO.md`.
```
$ cd 3DLLM_BLIP2-base
$ conda activate lavis
python inference.py # for objects
python inference.py --mode room # for scenes
```
TODO: huggingface auto load checkpoint.
## Finetuning
Finetuning config yaml files that need to be changed are in [this directory](https://github.com/UMass-Foundation-Model/3D-LLM/tree/main/3DLLM_BLIP2-base/lavis/projects/blip2/train)
1. Download the [pretrained checkpoints](https://drive.google.com/file/d/1tiis8mOdZGBzmR7vgZtRE4Ni_2FE4nTr/view?usp=drive_link). Modify the "resume_checkpoint_path" path in the yaml files
2. Download the [questions](https://drive.google.com/drive/folders/14MDiDl6Cch_B27Q0aZgdElhAEOBBpn2o?usp=drive_link), modify the "annotations" path in the yaml files
3. Download the [scannet features](https://drive.google.com/drive/folders/1H1SKZsK_XpMkXntwhY4BqSxsGCRP8cgp?usp=drive_link) or [3dmv-vqa features](https://drive.google.com/drive/folders/1NdFKKn_IZxGezi6fXA60rF1uxTOmhOet?usp=drive_link). Modify the path (both train and val) in lavis/datasets/datasets/threedvqa_datasets.py
4.
```
$ cd 3DLLM_BLIP2-base
$ conda activate lavis
python -m torch.distributed.run --nproc_per_node=8 train.py --cfg-path lavis/projects/blip2/train/<finetune_yaml_file>
```
You can also load the finetuning checkpoints [here](https://drive.google.com/drive/folders/1RKP1cz6R6H8YziEc4f3MHW9dCCXQChbA?usp=drive_link).
5.**Calculating scores**
```
cd calculate_scores
python calculate_score_<task>.py --folder <your result dir> --epoch <your epoch>
```
please also modify the feature and question path in the scripts
TODO: huggingface auto load checkpoint.
## Data
All data will be gradually released in [Google Drive](https://drive.google.com/drive/folders/188Yd7tmiUfyct-dVMpkQ8q_tnqkb-4bo?usp=sharing) and [Huggingface](https://huggingface.co/datasets/ShuhongZheng/3D-LLM) (All files are released in Google Drive first and then Huggingface. Please refer to the Google Drive for file structure)
### Pretraining Data
We are still cleaning the grounding & navigation part. All other pre-training data are released.
#### Object Data
&emsp;Language annotations of object data released [here](https://drive.google.com/file/d/17K1ZGb1HpvPj36a84GefGQK7DOkRp_32/view?usp=sharing).
&emsp;For downloading Objaverse data, please refer to [Objaverse website](https://objaverse.allenai.org/).
&emsp;To get 3D features and point clouds of the Objaverse data, please refer to [Step1](https://github.com/UMass-Foundation-Model/3D-LLM/blob/main/README.md#step1-render-images-from-different-views-of-a-scene) and [Step3](https://github.com/UMass-Foundation-Model/3D-LLM/blob/main/README.md#step3-3d-feature-construction-from-rendered-images) of 3DLanguage Data generation - ChatCaptioner based
&emsp;A small set of objaverse features is released [here](https://drive.google.com/file/d/1mJZONfWREfIUAPYXP65D65uS2EoplAfR/view?usp=drive_link).
&emsp;TODO: We will probably release the whole set of Objaverse 3D features
#### Scene Data
&emsp; [Language data released here](https://drive.google.com/file/d/18AVyAWwMwqP8nyZQfTpEMw1Wfhghjw0b/view?usp=drive_link).
&emsp; 3D features and point clouds (~250G) are released [here](https://drive.google.com/drive/folders/1bJpK00UFRZGH7tdx3PaqDrrTfHNP6B2K?usp=drive_link). However, if you want to explore generating the features yourself, please refer to the Three-step 3D Feature Extraction part [here](https://github.com/UMass-Foundation-Model/3D-LLM#three-step-3d-feature-extraction). Please use v2 to be consistent with the checkpoints (and also result in better performances).
&emsp;chat: 73103. task: 84531
### Finetuning Data
3D features and point clouds of Scannet (used for finetuning ScanQA and SQA3D) are released in [here](https://drive.google.com/drive/folders/1CsEt48jj5uCyelGcXXJBkGH86QYeCE8D?usp=drive_link). 3D features and point clouds of 3DMV-VQA are released [here](https://drive.google.com/drive/folders/1NdFKKn_IZxGezi6fXA60rF1uxTOmhOet?usp=drive_link) (3DMV-VQA data will be further updated for a clearer structure).
All questions can be found [here](https://drive.google.com/drive/folders/14MDiDl6Cch_B27Q0aZgdElhAEOBBpn2o?usp=drive_link).
## 3DLanguage Data Generation
### ChatCaptioner based / Three-step 3D Feature Extraction (Objaverse)
#### Step1: render images from different views of a scene
Follow the instruction in ``3DLanguage_data/ChatCaptioner_based/objaverse_render/README.md`` for installation.
The following code will render images of a objaverse scene (e.g. f6e9ec5953854dff94176c36b877c519). The rendered images will be saved at ``3DLanguage_data/ChatCaptioner_based/objaverse_render/output``.
(Please refer to ``3DLanguage_data/ChatCaptioner_based/objaverse_render/README.md`` for more details about the command)
```shell
$ cd ./3DLanguage_data/ChatCaptioner_based/objaverse_render
$ {path/to/blender} -b -P render.py -noaudio --disable-crash-handler -- --uid f6e9ec5953854dff94176c36b877c519
```
#### Step2: generate caption for this objaverse scene
Installation:
Please follow [ChatCaptioner](https://github.com/Vision-CAIR/ChatCaptioner) to install the environment/
The following code will read the rended images of an objaverse scene (e.g., f6e9ec5953854dff94176c36b877c519) and generate scene caption at ``3DLanguage_data/ChatCaptioner_based/output``
```shell
$ cd ./3DLanguage_data/ChatCaptioner_based
$ python chatcaption.py --specific_scene f6e9ec5953854dff94176c36b877c519
```
#### Step3: 3D feature construction from rendered images
Follow the instruction in ``3DLanguage_data/ChatCaptioner_based/gen_features/README.md`` for extracting 3D features from rendered images.
```shell
$ cd ./3DLanguage_data/ChatCaptioner_based/gen_features
```
### Box-Demonstration-Instruction based
TODO
### Revision based
TODO
## Three-step 3D Feature Extraction (Scene)
This section is for constructing 3D features for scene data. If you already downloaded our released scene data, please skip this section.
### First step
Installation:
Please follow [Mask2Former](https://github.com/facebookresearch/Mask2Former) to install the environment and download the [pretrained weight](https://dl.fbaipublicfiles.com/maskformer/mask2former/coco/instance/maskformer2_swin_large_IN21k_384_bs16_100ep/model_final_e5f453.pkl) to the current directory
if extracting the masks with [Mask2Former](https://openaccess.thecvf.com/content/CVPR2022/papers/Cheng_Masked-Attention_Mask_Transformer_for_Universal_Image_Segmentation_CVPR_2022_paper.pdf).
Please follow [Segment Anything](https://github.com/facebookresearch/segment-anything) to install the environment and download the [pretrained weight](https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth) to the current directory if extracting the masks with [SAM](https://arxiv.org/abs/2304.02643).
Extract masks with Mask2Former:
```shell
$ cd ./three_steps_3d_feature/first_step
$ python maskformer_mask.py --scene_dir_path DATA_DIR_WITH_RGB_IMAGES --save_dir_path DIR_YOU_WANT_TO_SAVE_THE_MASKS
```
Extract masks with Segment Anything:
```shell
$ cd ./three_steps_3d_feature/first_step
$ python sam_mask.py --scene_dir_path DATA_DIR_WITH_RGB_IMAGES --save_dir_path DIR_YOU_WANT_TO_SAVE_THE_MASKS
```
After the first step, we are expected to obtain a directory of masks (specified by ``--save_dir_path``) that contains extracted masks for
multi-view images of the scenes.
### Second step
Note: BLIP features are for LAVIS(BLIP2), CLIP features are for open-flamingo.
Installation: The same as the following ``3D-LLM_BLIP2-based`` section to install [salesforce-lavis](https://github.com/salesforce/LAVIS).
There are four options: (1) Extract CLIP feature with Mask2Former masks; (2) Extract CLIP feature with SAM masks;
(3) Extract BLIP feature with Mask2Former masks; (4) Extract BLIP feature with SAM masks.
Extract 2D CLIP features with Mask2Former masks:
```shell
$ cd ./three_steps_3d_feature/second_step/
$ python clip_maskformer.py --scene_dir_path DATA_DIR_WITH_RGB_IMAGES --mask_dir_path MASK_DIR_FROM_1ST_STEP --save_dir_path DIR_YOU_WANT_TO_SAVE_THE_FEAT
```
For the other options, the scripts are in similar format.
After the second step, we are expected to obtain a directory of features (specified by ``--save_dir_path``) that contains 2D features for
multi-view images of the scenes.
### Third step
#### Direct Reconstruction
Installation:
Please install the [Habitat environment](https://github.com/facebookresearch/habitat-lab/tree/challenge-2022).
Reconstruct 3D feature from multi-view 2D features:
```shell
$ cd ./three_steps_3d_feature/third_step/
$ python sam_mask.py --data_dir_path DATA_DIR_WITH_RGB_IMAGES --depth_dir_path DATA_DIR_WITH_DEPTH_IMAGES --feat_dir_path FEATURE_DIR_FROM_2ND_STEP
```
After the third step, we are expected to obtain two files (``pcd_pos.pt`` and ``pcd_feat.pt``) for each room inside the corresponding RGB directory.
``pcd_pos.pt`` contains the point positions of the 3D point cloud (shape: ``N * 3``). ``pcd_feat.pt`` contains the point features of the 3D point cloud (shape: ``N * n_dim``).
``N`` is the number of sampled points in the point cloud (default: 300000) and ``n_dim`` is the feature dimension (1024 for CLIP feature, 1408 for BLIP feature).
#### GradSLAM (Feature Fusion)
Refer to [Concept Fusion](https://github.com/concept-fusion/concept-fusion).
We will also release our reproduced version of Concept Fusion for our feature generation (we reproduced the paper before their official release).
#### Neural Field
Please refer to [3D-CLR](https://github.com/evelinehong/3D-CLR-Official) repository.
## Pre-training
```shell
$ cd 3DLLM_BLIP2-base
$ conda activate lavis
# use facebook/opt-2.7b:
$ TODO
# use flant5
$ python -m torch.distributed.run --nproc_per_node=8 train.py --cfg-path lavis/projects/blip2/train/pretrain.yaml
```
## 3D-LLM_flamingo-based
TODO.
## Citation
If you find our work useful, please consider citing:
```
@article{3dllm,
author = {Hong, Yining and Zhen, Haoyu and Chen, Peihao and Zheng, Shuhong and Du, Yilun and Chen, Zhenfang and Gan, Chuang},
title = {3D-LLM: Injecting the 3D World into Large Language Models},
journal = {NeurIPS},
year = {2023},
}
```
### Acknowledgements
https://github.com/salesforce/LAVIS
https://github.com/facebookresearch/Mask2Former
https://github.com/facebookresearch/segment-anything
https://github.com/mlfoundations/open_flamingo
https://github.com/concept-fusion/concept-fusion
https://github.com/evelinehong/3D-CLR-Official
plans/PRISM/.research/readmes/clio.md
+333
View File
@@ -0,0 +1,333 @@
# Clio
This repository contains the code for *Clio: Real-time Task-Driven Open-Set 3D Scene Graphs*.
![Clio](./assets/overview-figure.jpg)
Clio is a novel approach for building task-driven 3D scene graphs in real-time with open-set semantics. We draw inspiration from the classical Information Bottleneck principle to form task-relevant clusters of object primitives given a set of natural language tasks — such as ''Read brown textbook'' — and by clustering the scene into task-relevant semantic regions such as “Kitchenette” or “Workspace”. The map defines objects and regions at the correct semantic granularity to support tasks relevant for an agent.
# Table of Contents
* [Paper](#Paper)
* [News](#News)
* [Setup](#Setup)
* [Datasets](#Datasets)
* [Running Clio](#running-clio)
* [Clustering Pre-built Scene Graphs](#clustering-pre-built-scene-graphs)
* [Evaluation](#evaluation)
# Paper
If you find this useful for your research, please consider citing our paper:
* Dominic Maggio, Yun Chang, Nathan Hughes, Matthew Trang, Dan Griffith, Carlyn Dougherty, Eric Cristofalo, Lukas Schmid, Luca Carlone, "**Clio: Real-time Task-Driven Open-Set 3D Scene Graphs**", in IEEE Robotics and Automation Letters, vol. 9, no. 10, pp. 8921-8928, Oct. 2024. [ [IEEE](https://ieeexplore.ieee.org/document/10659066) | [ArXiv](http://arxiv.org/abs/2404.13696) | [Video](https://www.youtube.com/watch?v=m-HJO10qhSQ) ]
```bibtex
@ARTICLE{Maggio2024Clio,
title={Clio: Real-time Task-Driven Open-Set 3D Scene Graphs},
author={Maggio, Dominic and Chang, Yun and Hughes, Nathan and Trang, Matthew and Griffith, Dan and Dougherty, Carlyn and Cristofalo, Eric and Schmid, Lukas and Carlone, Luca},
journal={IEEE Robotics and Automation Letters},
year={2024},
volume={9},
number={10},
pages={8921-8928},
doi={10.1109/LRA.2024.3451395}
}
```
# News
- **[Bayesian Fields](https://arxiv.org/abs/2503.05949)** [Video](https://www.youtube.com/watch?v=DLBHpUP_-dk) – follow-up paper released showing improved results on the Clio datasets using better statistical understanding of CLIP and handling multi-view semantic measurements. Bayesian Fields also shows task-driven clustering with Gaussian Splatting.
- **[Ashita](https://arxiv.org/abs/2504.06553)** – follow-up paper released showing an LLM assisted task-driven reasoning framework that can use higher-level tasks and construct a 3D scene graph for all subtasks.
- Clio work was featured on the front page of **[MIT News](https://news.mit.edu/2024/helping-robots-focus-on-objects-that-matter-0930)** 🎉
# Setup
We recommend that everyone setup Clio by utilizing ROS.
To install ROS, you can follow the instructions [here](http://wiki.ros.org/ROS/Installation) if you haven't already.
> **Note**</br>
> We also provide a python-only implementation of Clio for offline processing of pre-built scene graphs and evaluation.
> If you want to avoid installing ROS and are only interested in this functionality, you can skip ahead to [these](#installing-without-ros) instructions instead.
### Installing with ROS
<details open>
<summary><b>Initial Requirements</b></summary>
Install the following requirements:
```
sudo apt install python3-rosdep python3-catkin-tools python3-vcstool python3-virtualenv
```
If you haven't set up rosdep yet run:
```
sudo rosdep init
rosdep update
```
</details>
<details open>
<summary><b>Getting and Building Clio</b></summary>
To clone and build Clio, first set up your catkin workspace:
```bash
mkdir -p ~/catkin_ws/src
cd ~/catkin_ws
catkin init
catkin config -DCMAKE_BUILD_TYPE=Release
catkin config --skiplist khronos_eval
```
> **Note**</br>
> By default, one of Clio's dependencies, [semantic_inference](https://github.com/MIT-SPARK/semantic_inference.git), will attempt to build against NVIDIA TensorRT.
> This is not required for Clio, and may cause issues when building if you already have CUDA set up on your system.
> You may wish to disable this by running `catkin config -a -DSEMANTIC_INFERENCE_USE_TRT=OFF` before building.
Then, clone the code and build:
```
cd src
git clone git@github.com:MIT-SPARK/Clio.git clio --recursive
vcs import . < clio/install/clio.rosinstall
rosdep install --from-paths . --ignore-src -r -y
cd ..
catkin build
```
> **Note**</br>
> For the rest of these instructions, we assume that you set up your catkin workspace at `~/catkin_ws`.
> If you used a different workspace path, you should substitute where appropriate.
</details>
<details open>
<summary><b>Setting up Open-Set Segmentation</b></summary>
Make a virtual environment and install:
```bash
python3 -m virtualenv --system-site-packages -p /usr/bin/python3 ~/environments/clio_ros
source ~/environments/clio_ros/bin/activate
pip install ~/catkin_ws/src/semantic_inference/semantic_inference[openset]
deactivate
```
> **Warning** :warning:</br>
> `--system-site-packages` is required when creating the environment.
</details>
<details open>
<summary><b>Setting up Clio Python Code</b></summary>
Make a virtual environment and install:
```bash
python3 -m virtualenv --download -p /usr/bin/python3 ~/environments/clio
source ~/environments/clio/bin/activate
pip install -e ~/catkin_ws/src/clio
```
> **Warning** :warning:</br>
> A devel install (i.e., using `-e` when installing Clio) is required.
</details>
### Installing without ROS
> **Warning** :warning:</br>
> This option does not include the open-set segmentation code or the real-time pipeline
First, setup a virtual environment:
```bash
python3 -m virtualenv -p /usr/bin/python3 --download ~/environments/clio
```
Then, clone and install Clio:
```bash
source ~/environments/clio/bin/activate
git clone https://github.com/MIT-SPARK/Clio.git clio --recursive
pip install -e clio
```
> **Note**</br>
> If you forgot to clone with `--recursive` you can run `git submodule update --init --recursive` instead.
# Datasets
Our custom datasets for the *Office*, *Apartment*, *Cubicle*, and *Building* scenes are available for download [here](https://www.dropbox.com/scl/fo/5bkv8rsa2xvwmvom6bmza/AOc8VW71kuZCgQjcw_REbWA?rlkey=wx1njghufcxconm1znidc1hgw&st=c809h8h3&dl=0). Each scene contains RGB images, depth images, a rosbag containing the RGB and depth images along with poses, and the list of tasks with ground truth object labels that was used in our paper. Each scene except *Building* contains a COLMAP dense reconstruction which can optionally be used to separately get a dense mesh view of the scene.
The task list is stored in a yaml file whose keys are the task and values are the ground truth oriented bounding boxes for the relevant objects. The folder structure is:
```
clio_datasets
├── apartment
│ ├── apartment.bag
│ ├── database.db
│ ├── dense
| ├── fused.ply
| |── meshed-poisson.ply
│ ├── depth
│ ├── images
│ ├── region_tasks_apartment.yaml
│ ├── rooms_apartment.yaml
│ ├── sparse
│ └── tasks_apartment.yaml
├── building
│ ├── ...
├── cubicle
│ ├── ...
├── office
│ ├── ...
```
### Pre-built Scene Graphs
Pre-built scene graph files can be downloaded from [here](https://www.dropbox.com/scl/fo/dmft0x6o0joxjxyy2q1qz/AIbgXuR4Y0nxMgsq1VtlWak?rlkey=g3orpk2itv1eyg0r1h7mpfgtt&st=po6btms5&dl=0),
which contain the 3D object primitives with corresponding meshes and semantic embedding vectors that Clio can use to form task-relevant objects.
These scene graph files can be used to test out Clio's Information Bottleneck clustering on a variety of tasks.
See [here](#clustering-pre-built-scene-graphs) for details.
### Pre-generating Open-set Semantics for a Scene
> **Warning** :warning: </br>
> This requires the [semantic_inference](https://github.com/MIT-SPARK/semantic_inference) package and ROS, which is installed by default if you follow the normal setup guide for Clio.
It may be convenient to generate the open-set segmentation and CLIP embeddings for a scene before running Clio.
You can run the following commands for any of the scenes, substituting the appropriate path to the rosbag for the scene.
First, source your `semantic_inference` environment and change to the directory containing the datasets if you haven't already:
```bash
source ~/environments/semantic_inference_ros/bin/activate
cd /path/to/clio/datasets
```
Using the apartment scene as an example, run:
```bash
rosrun semantic_inference_ros make_rosbag --clip-vec --copy \
apartment/apartment.bag -o apartment/apartment_with_semantics.bag \
/dominic/forward/color/image_raw:/dominic/forward/semantic/image_raw
```
to create a new bag, `apartment_with_semantics.bag` that contains the original contents of `apartment.bag` along with the open-set segmentation (under the `/dominic/forward/semantic/image_raw` topic).
# Running Clio
To run Clio on one of the provided [datasets](#datasets), first source your catkin workspace and python environment:
```bash
source ~/catkin_ws/devel/setup.bash
source ~/environments/clio_ros/bin/activate
```
In the following instructions, make sure to substitute the actual path to the datasets in place of `/path/to/datset`.
We'll use the *Office* scene for this example, but any of the datasets should work.
First, start Clio:
```bash
roslaunch clio_ros realsense.launch \
object_tasks_file:=/path/to/datasets/office/tasks_office.yaml \
place_tasks_file:=/path/to/datasets/office/region_tasks_office.yaml
```
If you want to use pre-generated segmentations and semantics instead, you can start Clio with the following:
```bash
roslaunch clio_ros realsense.launch run_segmentation:=false \
object_tasks_file:=/path/to/datasets/office/tasks_office.yaml \
place_tasks_file:=/path/to/datasets/office/region_tasks_office.yaml
```
> **Note**</br>
> Regardless of the `run_segmentation` setting, you should wait until Clio finishes initializing before starting the rosbag. You should see roughly this before proceeding:
> ```
> ...
> [INFO] [1728321782.786728, 0.000000]: '/semantic_inference': finished initializing!
> I1007 17:23:03.652261 3390159 ros_embedding_group.cpp:70] Got embeddings from '/task_server/objects'!
> I1007 17:23:03.654877 3390159 ros_embedding_group.cpp:47] Waiting for embeddings on '/task_server/places'
> I1007 17:23:03.857106 3390159 ros_embedding_group.cpp:70] Got embeddings from '/task_server/places'!
> I1007 17:23:04.047123 3390159 input_module.cpp:76] [Hydra Input] started!
> I1007 17:23:04.047190 3390159 active_window_module.cpp:72] [Active Window] started!
> I1007 17:23:04.047272 3390159 backend_module.cpp:134] [Hydra Backend] started!
> I1007 17:23:04.048808 3390159 graph_builder.cpp:175] [Hydra Frontend] started!
> [ INFO] [1728321784.049510656]: Running...
> ```
In a separate terminal, start the corresponding rosbag for the scene. For the *Office* dataset:
```bash
rosbag play path/to/datasets/office/office.bag --clock
```
You should see something like this after letting the bag play for a little bit:
![cluster](./assets/clio_online_example.png)
# Clustering Pre-built Scene Graphs
You can also try out Clio in an offline setting to perform task-oriented clustering on a pre-built scene graph of object primitives.
To try out Clio in this manner, run the following:
```
cd clio
python clio_batch/object_cluster.py <path-to-scene-graph-json> <path-to-task-yaml> <path-to-output-folder> <path-to-cluster-config>
```
For example, to run Clio just on the pre-built office scene graph (assuming you have made an empty folder first called clio_output), use the following command:
```
python clio_batch/object_cluster.py ~/clio_logs/office/trial_0/large_fine_ps/dsg.json ~/clio_datasets/office/tasks_office.yaml ~/clio_output clio_eval/experiments/configs/cluster/large_delta_00001.yaml
```
To quickly visualize results, you can visualize the estimated Clio bounding boxes (in green) and ground truth bounding boxes (in black) over a COLMAP mesh. To do this on the office scene, run:
```
python clio_eval/visualize_objects.py ~/clio_output/clio_dsg.json ~/clio_datasets/office/tasks_office.yaml ~/clio_datasets/office/dense/meshed-poisson.ply office
```
This will generate a 3D viewer such as this:
![cluster](./assets/sample_clustered.png)
Change to either apartment or cubicle with the following to generate the visualization on those datasets:
```
python clio_eval/visualize_objects.py clio_dsg.json ~/clio_datasets/apartment/tasks_apartment.yaml ~/clio_datasets/apartment/dense/meshed-poisson.ply apartment
python clio_eval/visualize_objects.py clio_dsg.json ~/clio_datasets/cubicle/tasks_cubicle.yaml ~/clio_datasets/cubicle/dense/meshed-poisson.ply cubicle
```
# Evaluation
> **Note**</br>
> To prepare Clio's codebase for release, we updated the versions of Hydra and Khronos that Clio is integrated with and changed how we integrated these codebases.
> An unintended consequence of these changes is that the performance of the real-time version of Clio has decreased compared to the results reported in our paper; this stems from what we believe is a bug associated with the reconstructed object primitives produced by Khronos.
> For a more complete discussion of the problem, please see [this issue](https://github.com/MIT-SPARK/Clio/issues/8).
> We are actively working on isolating the root cause of the issue.
> In the meantime, we've temporarily released a snapshot of the code used to generate the results in our paper [here](https://www.dropbox.com/scl/fi/q4ws6wnh5z9l1jdxztbf9/clio_ws.zip?rlkey=cgjrob8ddkyhof0rosizw7jmw&st=thdh8o5m&dl=0), which includes instructions for how to build and run this version of the code.
> If you plan on including results in a paper that require building new scene graphs with Clio, we politely request that you use this earlier version of Clio for now if possible.
> Please feel free to reach out if you have any questions!
We have included some of the evaluation code used to generate the results in our [paper](http://arxiv.org/abs/2404.13696).
In our paper, we refer to results obtained using the offline version of Clio (see [here](#clustering-pre-built-scene-graphs)) as *Clio-batch*, while the real-time ROS pipline (see [here](#running-clio)) is referred to as *Clio-online*.
We adopt this terminology here to make it easier to refer to the paper results.
### Evaluating Clio-Online
To obtain metrics on the quality of objects obtained by *Clio-Online*, set the paths in [realtime_clustering.yaml](clio_eval/experiments/configs/ablations/realtime_clustering.yaml) to the corresponding local task yaml files and folders containing the scene graphs produced by *Clio-online*. Then run:
```bash
python clio_eval/experiments/evaluate_ablations.py --config clio_eval/experiments/configs/ablations/realtime_clustering.yaml
```
### Ablating Clio-batch
This section is concerned with obtaining a partial ablation study of *Clio-batch*.
A config file for using CLIP model `ViT-L/14` used in the main results of the paper is provided in [3d_clustering.yaml](clio_eval/experiments/configs/ablations/3d_clustering.yaml).
Also note that a config file for Open-CLIP model `ViT-H-14` used to produce results in the appendix of the paper is provided in [3d_clustering_oc.yaml](clio_eval/experiments/configs/ablations/3d_clustering_oc.yaml).
The yaml files allow for running on one or on multiple datasets at a time.
First, set paths to your pre-built scene graph files and yaml files containing the ground truth objects and task labels in `3d_clustering.yaml` and `3d_clustering_oc.yaml`.
Then, to run *Clio-batch* and evaluate the results, run:
```
python clio_eval/experiments/run_3d_object_ablations.py clio_eval/experiments/configs/ablations/<3d_clustering.yaml/3d_clustering_oc.yaml>
```
The provided results will include for each dataset:
- Clio Primitives (labeled as `<dataset>_fine`)
- Khronos (labeled as `<dataset>_khronos`)
- Khronos thresholded (labeled as `<dataset>_khronos_thres`)
- Clio batch (labeled as `<dataset>_d<IB delta stop value>`)
By default, the config files are set to run on all three of the office, apartment, and cubicle datasets. Results from clustering will be saved in a newly created scene graph files for each dataset.
plans/PRISM/.research/readmes/concept-graphs.md
+318
View File
@@ -0,0 +1,318 @@
<!-- src: https://raw.githubusercontent.com/concept-graphs/concept-graphs/main/README.md -->
# ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning
[**Project Page**](https://concept-graphs.github.io/) |
[**Paper**](https://concept-graphs.github.io/assets/pdf/2023-ConceptGraphs.pdf) |
[**ArXiv**](https://arxiv.org/abs/2309.16650) |
[**Video**](https://www.youtube.com/watch?v=mRhNkQwRYnc&feature=youtu.be&ab_channel=AliK)
[Qiao Gu](https://georgegu1997.github.io/)\*,
[Ali Kuwajerwala](https://www.alihkw.com/)\*,
[Sacha Morin](https://sachamorin.github.io/)\*,
[Krishna Murthy Jatavallabhula](https://krrish94.github.io/)\*,
[Bipasha Sen](https://bipashasen.github.io/),
[Aditya Agarwal](https://skymanaditya1.github.io/),
[Corban Rivera](https://www.jhuapl.edu/work/our-organization/research-and-exploratory-development/red-staff-directory/corban-rivera),
[William Paul](https://scholar.google.com/citations?user=92bmh84AAAAJ),
[Kirsty Ellis](https://mila.quebec/en/person/kirsty-ellis/),
[Rama Chellappa](https://engineering.jhu.edu/faculty/rama-chellappa/),
[Chuang Gan](https://people.csail.mit.edu/ganchuang/),
[Celso Miguel de Melo](https://celsodemelo.net/),
[Joshua B. Tenenbaum](http://web.mit.edu/cocosci/josh.html),
[Antonio Torralba](https://groups.csail.mit.edu/vision/torralbalab/),
[Florian Shkurti](http://www.cs.toronto.edu//~florian/),
[Liam Paull](http://liampaull.ca/)
![Splash Figure](./assets/splash-final.png)
# Updates
* The codebase has been significantly refactored in the `ali-dev` [branch](https://github.com/concept-graphs/concept-graphs/tree/ali-dev), which provides a real-time, streamlined re-implementation that supports RGB-D video from iPhone and has a better visualization using [Rerun.io](https://rerun.io/). We also provide this [getting started video tutorial](https://youtu.be/56jEFyrqqpo?si=jo-qto5Gv8qxqEw2). Please check it out!
* The code for real-world mapping and navigation using a Jackal robot is open-sourced [here](https://github.com/sachaMorin/concept_graphs_jackal).
* The code for localization and mapping in AI2Thor is released in the codebase. See [here](https://github.com/concept-graphs/concept-graphs/tree/main?tab=readme-ov-file#ai2thor-related-experiments) for instructions.
## Setup
The env variables needed can be found in `env_vars.bash.template`. When following the setup guide below, you can duplicate that files and change the variables accordingly for easy setup.
### Install the required libraries
We recommend setting up a virtual environment using virtualenv or conda. Our code has been tested with Python 3.10.12. It may also work with other later versions. We also provide the `environment.yml` file for Conda users. In generaly, directly installing conda env using `.yml` file may cause some unexpected issues, so we recommand setting up the environment by the following instructions and only using the `.yml` file as a reference.
Sample instructions for `conda` users.
```bash
conda create -n conceptgraph anaconda python=3.10
conda activate conceptgraph
# Install the required libraries
pip install tyro open_clip_torch wandb h5py openai hydra-core distinctipy
# for yolo
pip install ultralytics
# Install the Faiss library (CPU version should be fine)
conda install -c pytorch faiss-cpu=1.7.4 mkl=2021 blas=1.0=mkl
##### Install Pytorch according to your own setup #####
# For example, if you have a GPU with CUDA 11.8 (We tested it Pytorch 2.0.1)
conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia
# Install Pytorch3D (https://github.com/facebookresearch/pytorch3d/blob/main/INSTALL.md)
# conda install pytorch3d -c pytorch3d # This detects a conflict. You can use the command below, maybe with a different version
conda install https://anaconda.org/pytorch3d/pytorch3d/0.7.4/download/linux-64/pytorch3d-0.7.4-py310_cu118_pyt201.tar.bz2
# Install the gradslam package and its dependencies
# Please clone and install them in separate folders, not within the concept-graphs folder.
git clone https://github.com/krrish94/chamferdist.git
cd chamferdist
pip install .
cd ..
git clone https://github.com/gradslam/gradslam.git
cd gradslam
git checkout conceptfusion
pip install .
```
### Install [Grounded-SAM](https://github.com/IDEA-Research/Grounded-Segment-Anything) package
Follow the instructions on the original [repo](https://github.com/IDEA-Research/Grounded-Segment-Anything#install-without-docker). ConceptGraphs has been tested with the codebase at this [commit](https://github.com/IDEA-Research/Grounded-Segment-Anything/commit/a4d76a2b55e348943cba4cd57d7553c354296223). Grounded-SAM codebase at later commits may require some adaptations.
First checkout the package by
```bash
git clone git@github.com:IDEA-Research/Grounded-Segment-Anything.git
```
Then, install the package Following the commands listed in the original GitHub repo. You can skip the `Install osx` step and the "optional dependencies".
During this process, you will need to set the `CUDA_HOME` to be where the CUDA toolkit is installed.
The CUDA tookit can be set up system-wide or within a conda environment. We tested it within a conda environment, i.e. installing [cudatoolkit-dev](https://anaconda.org/conda-forge/cudatoolkit-dev) using conda.
```bash
# i.e. You can install cuda toolkit using conda
conda install -c conda-forge cudatoolkit-dev
# and you need to replace `export CUDA_HOME=/path/to/cuda-11.3/` by
export CUDA_HOME=/path/to/anaconda3/envs/conceptgraph/
```
You also need to download `ram_swin_large_14m.pth`, `groundingdino_swint_ogc.pth`, `sam_vit_h_4b8939.pth` (and optionally `tag2text_swin_14m.pth` if you want to try Tag2Text) following the instruction [here](https://github.com/IDEA-Research/Grounded-Segment-Anything#label-grounded-sam-with-ram-or-tag2text-for-automatic-labeling).
After installation, set the path to Grounded-SAM as an environment variable
```bash
export GSA_PATH=/path/to/Grounded-Segment-Anything
```
### (Optional) Set up the EfficientSAM variants
Follow the installation instructions on this [page](https://github.com/IDEA-Research/Grounded-Segment-Anything/tree/main/EfficientSAM). The major steps are:
* Install FastSAM codebase following [here](https://github.com/CASIA-IVA-Lab/FastSAM#installation). You don't have to create a new conda env. Just installing it in the same env as the Grounded-SAM is fine.
* Download FastSAM checkpoints [FastSAM-x.pt](https://github.com/CASIA-IVA-Lab/FastSAM#model-checkpoints) and save it to `Grounded-Segment-Anything/EfficientSAM`.
* Download MobileSAM checkpoints [mobile_sam.pt](https://github.com/ChaoningZhang/MobileSAM/blob/master/weights/mobile_sam.pt) and save it to `Grounded-Segment-Anything/EfficientSAM`.
* Download Light HQ-SAM checkpoints [sam_hq_vit_tiny.pth](https://huggingface.co/lkeab/hq-sam/resolve/main/sam_hq_vit_tiny.pth) and save it to `Grounded-Segment-Anything/EfficientSAM`.
### Install this repo
```bash
git clone git@github.com:concept-graphs/concept-graphs.git
cd concept-graphs
pip install -e .
```
### Set up LLaVA (used for scene graph generation)
Follow the instructions on the [LLaVA repo](https://github.com/haotian-liu/LLaVA) to set it up. You also need to prepare the LLaVA checkpoints and save them to `$LLAVA_CKPT_PATH`. We have tested with model checkpoint `LLaVA-7B-v0` and [LLaVA code](https://github.com/haotian-liu/LLaVA) at this [commit](https://github.com/haotian-liu/LLaVA/tree/8fc54a09a6be74b2abd913c468fb3d42ae826194). LLaVA codebase at later commits may require some adaptations.
```bash
# Set the env variables as follows (change the paths accordingly)
export LLAVA_PYTHON_PATH=/path/to/llava
export LLAVA_CKPT_PATH=/path/to/LLaVA-7B-v0
```
## Prepare dataset (Replica as an example)
ConceptGraphs takes posed RGB-D images as input. Here we show how to prepare the dataset using [Replica](https://github.com/facebookresearch/Replica-Dataset) as an example. Instead of the original Replica dataset, download the scanned RGB-D trajectories of the Replica dataset provided by [Nice-SLAM](https://github.com/cvg/nice-slam). It contains rendered trajectories using the mesh models provided by the original Replica datasets.
Download the Replica RGB-D scan dataset using the downloading [script](https://github.com/cvg/nice-slam/blob/master/scripts/download_replica.sh) in [Nice-SLAM](https://github.com/cvg/nice-slam#replica-1) and set `$REPLICA_ROOT` to its saved path.
```bash
export REPLICA_ROOT=/path/to/Replica
export CG_FOLDER=/path/to/concept-graphs/
export REPLICA_CONFIG_PATH=${CG_FOLDER}/conceptgraph/dataset/dataconfigs/replica/replica.yaml
```
ConceptGraphs can also be easily run on other dataset. See `dataset/datasets_common.py` for how to write your own dataloader.
## Run ConceptGraph
The following commands should be run in the `conceptgraph` folder.
```bash
cd conceptgraph
```
### (Optional) Run regular 3D reconstruction for sanity check
The following command runs a 3D RGB reconstruction ([GradSLAM](https://github.com/gradslam/gradslam)) of a replica scene and also visualize it. This is useful for sanity check.
* `--visualize` requires it to be run with GUI.
```bash
SCENE_NAME=room0
python scripts/run_slam_rgb.py \
--dataset_root $REPLICA_ROOT \
--dataset_config $REPLICA_CONFIG_PATH \
--scene_id $SCENE_NAME \
--image_height 480 \
--image_width 640 \
--stride 5 \
--visualize
```
### Extract 2D (Detection) Segmentation and per-resgion features
First, (Detection) Segmentation results and per-region CLIP features are extracted. In the following, we provide two options.
* The first one (ConceptGraphs) uses SAM in the "segment all" mode and extract class-agnostic masks.
* The second one (ConceptGraphs-Detect) uses a tagging model and a detection model to extract class-aware bounding boxes first, and then use them as prompts for SAM to segment each object.
```bash
SCENE_NAME=room0
# The CoceptGraphs (without open-vocab detector)
python scripts/generate_gsa_results.py \
--dataset_root $REPLICA_ROOT \
--dataset_config $REPLICA_CONFIG_PATH \
--scene_id $SCENE_NAME \
--class_set none \
--stride 5
# The ConceptGraphs-Detect
CLASS_SET=ram
python scripts/generate_gsa_results.py \
--dataset_root $REPLICA_ROOT \
--dataset_config $REPLICA_CONFIG_PATH \
--scene_id $SCENE_NAME \
--class_set $CLASS_SET \
--box_threshold 0.2 \
--text_threshold 0.2 \
--stride 5 \
--add_bg_classes \
--accumu_classes \
--exp_suffix withbg_allclasses
```
The above commands will save the detection and segmentation results in `$REPLICA_ROOT/$SCENE_NAME/`.
The visualization of the detection and segmentation can be viewed in `$REPLICA_ROOT/$SCENE_NAME/gsa_vis_none` and `$REPLICA_ROOT/$SCENE_NAME/gsa_vis_ram_withbg_allclasses` respectively.
You can ignore the `There's a wrong phrase happen, this is because of our post-process merged wrong tokens, which will be modified in the future. We will assign it with a random label at this time.` message for now.
### Run the 3D object mapping system
The following command builds an object-based 3D map of the scene, using the image segmentation results from above.
* Use `save_objects_all_frames=True` to save the mapping results at every frame, which can be used for animated visualization by `scripts/animate_mapping_interactive.py` and `scripts/animate_mapping_save.py`.
* Use `merge_interval=20 merge_visual_sim_thresh=0.8 merge_text_sim_thresh=0.8` to also perform overlap-based merging during the mapping process.
```bash
# Using the CoceptGraphs (without open-vocab detector)
THRESHOLD=1.2
python slam/cfslam_pipeline_batch.py \
dataset_root=$REPLICA_ROOT \
dataset_config=$REPLICA_CONFIG_PATH \
stride=5 \
scene_id=$SCENE_NAME \
spatial_sim_type=overlap \
mask_conf_threshold=0.95 \
match_method=sim_sum \
sim_threshold=${THRESHOLD} \
dbscan_eps=0.1 \
gsa_variant=none \
class_agnostic=True \
skip_bg=True \
max_bbox_area_ratio=0.5 \
save_suffix=overlap_maskconf0.95_simsum${THRESHOLD}_dbscan.1_merge20_masksub \
merge_interval=20 \
merge_visual_sim_thresh=0.8 \
merge_text_sim_thresh=0.8
# On the ConceptGraphs-Detect
SCENE_NAME=room0
THRESHOLD=1.2
python slam/cfslam_pipeline_batch.py \
dataset_root=$REPLICA_ROOT \
dataset_config=$REPLICA_CONFIG_PATH \
stride=5 \
scene_id=$SCENE_NAME \
spatial_sim_type=overlap \
mask_conf_threshold=0.25 \
match_method=sim_sum \
sim_threshold=${THRESHOLD} \
dbscan_eps=0.1 \
gsa_variant=ram_withbg_allclasses \
skip_bg=False \
max_bbox_area_ratio=0.5 \
save_suffix=overlap_maskconf0.25_simsum${THRESHOLD}_dbscan.1
```
The above commands will save the mapping results in `$REPLICA_ROOT/$SCENE_NAME/pcd_saves`. It will create two `pkl.gz` files, where the one with `_post` suffix indicates results after some post processing, which we recommend using.`
If you run the above command with `save_objects_all_frames=True`, it will create a folder in `$REPLICA_ROOT/$SCENE_NAME/objects_all_frames`. Then you can run the following command to visualize the mapping process or save it to a video. Also see the relevant files for available key callbacks for viusalization options.
```
python scripts/animate_mapping_interactive.py --input_folder $REPLICA_ROOT/$SCENE_NAME/objects_all_frames/<folder_name>
python scripts/animate_mapping_save.py --input_folder $REPLICA_ROOT/$SCENE_NAME/objects_all_frames/<folder_name>
```
### Visualize the object-based mapping results
```bash
python scripts/visualize_cfslam_results.py --result_path /path/to/output.pkl.gz
```
Then in the open3d visualizer window, you can use the following key callbacks to change the visualization.
* Press `b` to toggle the background point clouds (wall, floor, ceiling, etc.). Only works on the ConceptGraphs-Detect.
* Press `c` to color the point clouds by the object class from the tagging model. Only works on the ConceptGraphs-Detect.
* Press `r` to color the point clouds by RGB.
* Press `f` and type text in the terminal, and the point cloud will be colored by the CLIP similarity with the input text.
* Press `i` to color the point clouds by object instance ID.
### Evaluate semantic segmentation from the object-based mapping results on Replica datasets
First, download the GT point cloud with per-point semantic segmentation labels from this [Google Drive link](https://drive.google.com/file/d/1NhQIM5PCH5L5vkZDSRq6YF1bRaSX2aem/view?usp=sharing). Please refer to [this issue](https://github.com/concept-graphs/concept-graphs/issues/18#issuecomment-1876673985) for a brief description of how they are generated. Unzip the file and record its location in `REPLICA_SEMANTIC_ROOT`.
Then run the following command to evaluate the semantic segmentation results. The results will be saved in the `results` folder, where the mean recall `mrecall` is the mAcc and `fmiou` is the F-mIoU reported in the paper.
```bash
# CoceptGraphs (without open-vocab detector)
python scripts/eval_replica_semseg.py \
--replica_root $REPLICA_ROOT \
--replica_semantic_root $REPLICA_SEMANTIC_ROOT \
--n_exclude 6 \
--pred_exp_name none_overlap_maskconf0.95_simsum1.2_dbscan.1_merge20_masksub
# On the ConceptGraphs-Detect (Grounding-DINO as the object detector)
python scripts/eval_replica_semseg.py \
--replica_root $REPLICA_ROOT \
--replica_semantic_root $REPLICA_SEMANTIC_ROOT \
--n_exclude 6 \
--pred_exp_name ram_withbg_allclasses_overlap_maskconf0.25_simsum1.2_dbscan.1_masksub
```
### Extract object captions and build scene graphs
Ensure that the `openai` package is installed and that your APIKEY is set. We recommend using GPT-4, since GPT-3.5 often produces inconsistent results on this task.
```bash
export OPENAI_API_KEY=<your GPT-4 API KEY here>
```
Also note that if you are using the same [commit](https://github.com/haotian-liu/LLaVA/tree/8fc54a09a6be74b2abd913c468fb3d42ae826194) as we did, you may need to
plans/PRISM/.research/readmes/conceptfusion.md
+218
View File
@@ -0,0 +1,218 @@
<!-- src: https://raw.githubusercontent.com/concept-fusion/concept-fusion/main/README.md -->
# ConceptFusion: Open-set Multimodal 3D Mapping
> Code release for our [RSS 2023](https://roboticsconference.org/) publication
[Project page](https://concept-fusion.github.io/) | [Video explainer](https://www.youtube.com/watch?v=rkXgws8fiDs) | [arXiv](https://arxiv.org/abs/2302.07241)
[Krishna Murthy Jatavallabhula](https://krrish94.github.io/), [Alihusein Kuwajerwala](https://www.alihkw.com/), [Qiao Gu](https://georgegu1997.github.io/), [Mohd Omama](https://scholar.google.com/citations?user=jFH3ShsAAAAJ&hl=en), [Tao Chen](https://taochenshh.github.io/), [Alaa Maalouf](https://www.csail.mit.edu/person/alaa-maalouf), [Shuang Li](https://people.csail.mit.edu/lishuang/), [Ganesh Iyer](https://epiception.github.io/), [Soroush Saryazdi](https://saryazdi.github.io/), [Nikhil Keetha](https://nik-v9.github.io/), [Ayush Tewari](https://ayushtewari.com/), [Joshua B. Tenenbaum](http://web.mit.edu/cocosci/josh.html), [Celso Miguel de Melo](https://celsodemelo.net/), [Madhava Krishna](https://robotics.iiit.ac.in/), [Liam Paull](http://liampaull.ca/), [Florian Shkurti](http://www.cs.toronto.edu//~florian/), [Antonio Torralba](https://groups.csail.mit.edu/vision/torralbalab/)
<p align="center">
<img src="assets/splash.gif" />
</p>
> **Note**: WIP repo with the following key deviations from the ConceptFusion paper
1. Employ the segment anything model (SAM) opposed to Mask2Former to generate mask proposals
2. Remove the mask-to-mask similarity term (uniqueness; Eq. 4) -- needed for Mask2Former, but SAM seems to work okay without
3. (TODO) Add parser and download links for UnCoCo data
## Prerequisites
We recommend setting up a python virtualenv or conda environment to help manage dependencies. Our code has been tested primarily with Python 3.10 (although this should technically work with Python 3.8 with minimal modifications).
Sample instruction for `conda` users.
```
conda create -n conceptfusion python=3.10.8
conda activate conceptfusion
```
**Pytorch**: Install PyTorch using an appropriate Python-CUDA-CuDNN config from the [pytorch webpage](https://pytorch.org/).
**gradslam**: Install the `conceptfusion` branch of [gradlsam](https://gradslam.github.io/) by following these instructions (Note: the `main` branch does not have the feature fusion functionality, and will therefore, not work).
```
git clone https://github.com/gradslam/gradslam.git
cd gradslam
git checkout conceptfusion
pip install -e .
```
**segment-anything**: Install `segment-anything` by following [instructions here](https://github.com/facebookresearch/segment-anything).
**openclip**: Install `openclip` following [instructions here](https://github.com/mlfoundations/open_clip).
**(Optional) OpenAI CLIP**: If interested in using the OpenAI CLIP models, install `clip`.
```
pip install git+https://github.com/openai/CLIP.git
```
Note, however, that our released code isn't set up to use these CLIP models, and may require a few low-effort edits.
## Download and extract datasets
Depending on the dataset you would like to use, download and set it up for gradslam. To extend our (general-enough) dataset class to your own dataset, we recommend looking into the `gradslam` package (again, the `conceptfusion` branch), particularly `gradslam/datasets/` directory. A number of datasets have already been implemented.
### ICL dataset
Download it from [here](https://www.doc.ic.ac.uk/~ahanda/VaFRIC/iclnuim.html).
For the first two "scenes", i.e. living rooms 'lr kt0' and 'lr kt1', we want to download the files from these two links:
- "TUM RGB-D Compatible PNGs"
- "Global Poses [R | t]: Global_RT_Trajectory_GT" files.
### ScanNet and other datasets
**TODO**
## Setup models and download checkpoints
By default, the commandline arguments are setup to run conceptfusion feature extraction (i.e., CLIP features from an `openclip` model). If you would like to use DINO or LSeg features instead, follow these setup instructions. Else, this section may safely be ignored.
### DINO and/or LSeg models
From the dino repo [here](https://github.com/facebookresearch/dino) get the "ViT-B/8 backbone only" checkpoint
From the Lseg-minimal repo [here](https://github.com/krrish94/lseg-minimal#download-pretrained-weights) , get the checkpoint from the onedrive link.
Put these files in the checkpoints folder:
`/your/path/to/concept-fusion/examples/checkpoints/`
Full structure:
```
├── checkpoints
│ ├── dino_vitbase8_pretrain.pth
│ └── lseg_minimal_e200.ckpt
```
Also, clone and setup the `lseg-minimal` and `dino-minimal` repos -- used in running the pretrained networks for feature extraction. (These can be installed anywhere in your env, and do not have to be within the `concept-fusion` directory)
```
cd /path/to/where/you/keep/repos
git clone https://github.com/krrish94/lseg-minimal
cd lseg-minimal
python setup.py build develop
```
> Download the pretrained weights for the model(s) used in the `lseg-minimal` repo (instructions/links in readme)
```
cd /path/to/where/you/keep/repos
git clone https://github.com/krrish94/dino-minimal
cd dino-minimal
python setup.py build develop
```
> The pretrained weights for DINO will automatically be downloaded by this library when you run it for the first time
## Extract ConceptFusion features
To extract pixel-aligned CLIP features from a `GradSLAMDataset`, run
```
cd examples
python extract_conceptfusion_features.py
```
This script can parse any dataset compatible with the `GradSLAMDataset` format. It extracts mask proposals from SAM, computes CLIP features per-mask (and for the full image), and applies the pixel-aligned feature extraction scheme proposed in the paper (with the caveats at the top of this README).
The extracted features are saved in the `saved-feat` directory by default (this can be overridden by passing a `--feat_dir` argument).
## (Optional) Extract DINO and/or LSeg features
To extract features from other models like DINO or LSeg, run `run_feature_fusion_and_save_map.py` with `--mode extract` (and other flags as appropriate; importantly `--checkpoint_path`).
## Fuse extracted features to a 3D map
After extracting features, fuse them to 3D by running
```
python run_feature_fusion_and_save_map.py
```
This script fuses the extracted features into a 3D pointcloud map, and saved this by default to the `saved-map` directory.
## Try out the examples (and interactive demos)
### Interactive demo - click a point on the scene; visualize self similar points
```
python demo_click_query.py --load_path saved-map
```
This script will load the map saved in the `saved-map` directory. An Open3D window will pop up, where you can click on a point (`SHIFT + LEFT_MOUSE_BUTTON`). (while you can technically click multiple points, we discard all but the first clicked point). The script will then plot a similarity heatmap indicating all other scene points and their similarities, visualized in a `jet` colormap (red => higher similarity; blue => lower similarity).
### Interactive demo - text query
```
python demo_text_query.py --load_path saved-map
```
This script will load the map saved in the `saved-map` directory. You may type in a text query on the console (or press `q` to quit), and a similarity map will be displayed (again, using a `jet` colormap).
### TODO
(TODO - add K-means clustering demo)
If interested in running the K-Means clustering demo, you will need to install `fast-pytorch-kmeans`
```
pip install fast-pytorch-kmeans
```
## Outdated instructions
The instructions that follow are outdated, but are retained here, to help understand typical commandline arguments for other datasets such as ScanNet.
### Examples on use run_feature_fusion_and_save_map.py
The fourth step should also work on OpenSeg by setting `--model_type ovseg`.
```
# 0. Change these settings to your own path
SCENE_ID=scene0568_00
SCANNET_ROOT=/home/qiao/data/scannet/scans
DIR_FEAT=/home/qiao/data/scannet/results/${SCENE_ID}-lseg-0-500
DIR_SAVE_MAP=/home/qiao/data/scannet/results/saved-maps-${SCENE_ID}-lseg-0-500
DIR_SAVE_GT=/home/qiao/data/scannet/results/saved-maps-gt-${SCENE_ID}-lseg-0-500
DIR_SAVE_METRICS=/home/qiao/data/scannet/results/metrics-${SCENE_ID}-lseg-0-500
# 1. Extract feature map for each frame and save them to disk
python run_feature_fusion_and_save_map.py --mode extract --model_type lseg --dataconfig_path dataconfigs/scannet/${SCENE_ID}.yaml --dataset_path $SCANNET_ROOT --sequence ${SCENE_ID} --image_height 480 --image_width 640 --frame_start 0 --frame_end 500 --stride 25 --desired_feature_height 240 --desired_feature_width 320 --feat_dir $DIR_FEAT --dir_to_save_map $DIR_SAVE_MAP --checkpoint_path checkpoints/lseg_minimal_e200.ckpt
# 2. Load the saved feature map, fuse them and save the result to disk
python run_feature_fusion_and_save_map.py --mode fusion --model_type lseg --dataconfig_path dataconfigs/scannet/${SCENE_ID}.yaml --dataset_path $SCANNET_ROOT --sequence ${SCENE_ID} --image_height 240 --image_width 320 --frame_start 0 --frame_end 500 --stride 25 --desired_feature_height 240 --desired_feature_width 320 --feat_dir $DIR_FEAT --dir_to_save_map $DIR_SAVE_MAP --checkpoint_path checkpoints/lseg_minimal_e200.ckpt
# 3. Fuse the GT semantic labels, get per-point GT classification and save them to disk
python run_scannet_feature_fusion_and_save_map.py --mode fusion-gt --dataconfig_path dataconfigs/scannet/${SCENE_ID}.yaml --dataset_path $SCANNET_ROOT --sequence ${SCENE_ID} --image_height 240 --image_width 320 --frame_start 0 --frame_end 500 --stride 25 --desired_feature_height 240 --desired_feature_width 320 --feat_dir $DIR_FEAT --dir_to_save_gt $DIR_SAVE_GT --checkpoint_path checkpoints/lseg_minimal_e200.ckpt
# 4. Evaluate the result and compute metrics (4 variants below). Remember to change DIR_SAVE_METRICS to your own path.
## 4-1. Use the text embeddings as the query features
python run_scannet_feature_fusion_and_save_map.py --mode metrics --model_type lseg --dataconfig_path dataconfigs/scannet/${SCENE_ID}.yaml --dataset_path $SCANNET_ROOT --sequence ${SCENE_ID} --image_height 240 --image_width 320 --frame_start 0 --frame_end 500 --stride 25 --desired_feature_height 240 --desired_feature_width 320 --feat_dir $DIR_FEAT --dir_to_save_map $DIR_SAVE_MAP --dir_to_save_gt $DIR_SAVE_GT --dir_to_save_metrics $DIR_SAVE_METRICS --checkpoint_path checkpoints/lseg_minimal_e200.ckpt
## 4-2. Use the feature means (Oracle) as the query features
DIR_SAVE_METRICS=/home/qiao/data/scannet/results/metrics-oracle-${SCENE_ID}-lseg-0-500
python run_scannet_feature_fusion_and_save_map.py --mode metrics --model_type lseg --dataconfig_path dataconfigs/scannet/${SCENE_ID}.yaml --dataset_path $SCANNET_ROOT --sequence ${SCENE_ID} --image_height 240 --image_width 320 --frame_start 0 --frame_end 500 --stride 25 --desired_feature_height 240 --desired_feature_width 320 --feat_dir $DIR_FEAT --dir_to_save_map $DIR_SAVE_MAP --dir_to_save_gt $DIR_SAVE_GT --dir_to_save_metrics $DIR_SAVE_METRICS --checkpoint_path checkpoints/lseg_minimal_e200.ckpt --query_feat oracle
## 4-3. Use the feature means of random 1 points as the query features (multiple runs recommended)
DIR_SAVE_METRICS=/home/qiao/data/scannet/results/metrics-rand1p-${SCENE_ID}-lseg-0-500
python run_scannet_feature_fusion_and_save_map.py --mode metrics --model_type lseg --dataconfig_path dataconfigs/scannet/${SCENE_ID}.yaml --dataset_path $SCANNET_ROOT --sequence ${SCENE_ID} --image_height 240 --image_width 320 --frame_start 0 --frame_end 500 --stride 25 --desired_feature_height 240 --desired_feature_width 320 --feat_dir $DIR_FEAT --dir_to_save_map $DIR_SAVE_MAP --dir_to_save_gt $DIR_SAVE_GT --dir_to_save_metrics $DIR_SAVE_METRICS --checkpoint_path checkpoints/lseg_minimal_e200.ckpt --query_feat random --n_point_query 1
## 4-4. Use the feature means of random 3 points as the query features (multiple runs recommended)
DIR_SAVE_METRICS=/home/qiao/data/scannet/results/metrics-rand3p-${SCENE_ID}-lseg-0-500
python run_scannet_feature_fusion_and_save_map.py --mode metrics --model_type lseg --dataconfig_path dataconfigs/scannet/${SCENE_ID}.yaml --dataset_path $SCANNET_ROOT --sequence ${SCENE_ID} --image_height 240 --image_width 320 --frame_start 0 --frame_end 500 --stride 25 --desired_feature_height 240 --desired_feature_width 320 --feat_dir $DIR_FEAT --dir_to_save_map $DIR_SAVE_MAP --dir_to_save_gt $DIR_SAVE_GT --dir_to_save_metrics $DIR_SAVE_METRICS --checkpoint_path checkpoints/lseg_minimal_e200.ckpt --query_feat random --n_point_query 3
```
For 4-3 and 4-4, it would be better to run the commands multiple times with different seeds and average the results. For example:
```
DIR_SAVE_METRICS=/home/qiao/data/scannet/results/metrics-rand3p-${SCENE_ID}-lseg-0-500
for SEED in {1..10}
do
python run_scannet_feature_fusion_and_save_map.py --mode metrics --model_type lseg --dataconfig_path dataconfigs/scannet/${SCENE_ID}.yaml --dataset_path $SCANNET_ROOT --sequence ${SCENE_ID} --image_height 240 --image_width 320 --frame_start 0 --frame_end 500 --stride 25 --desired_feature_height 240 --desired_feature_width 320 --feat_dir $DIR_FEAT --dir_to_save_map $DIR_SAVE_MAP --dir_to_save_gt $DIR_SAVE_GT --dir_to_save_metrics $DIR_SAVE_METRICS --checkpoint_path checkpoints/lseg_minimal_e200.ckpt --query_feat random --n_point_query 3 --seed $SEED
done
```
plans/PRISM/.research/readmes/hov-sg.md
+314
View File
@@ -0,0 +1,314 @@
<!-- src: https://raw.githubusercontent.com/hovsg/HOV-SG/main/README.md -->
# HOV-SG
[![Static Badge](https://img.shields.io/badge/-arXiv-B31B1B?logo=arxiv)](https://arxiv.org/abs/2403.17846)
[![Static Badge](https://img.shields.io/badge/Project-Page-a)](https://hovsg.github.io/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Static Badge](https://img.shields.io/badge/-Video-FF0000?logo=youtube)](https://hovsg.github.io/static/images/hovsg_rss_final.mp4)
This repository is the official implementation of the paper:
> **Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation**
>
> [Abdelrhman Werby]()&ast;, [Chenguang Huang](http://www2.informatik.uni-freiburg.de/~huang/)&ast;, [Martin Büchner](https://rl.uni-freiburg.de/people/buechner)&ast;, [Abhinav Valada](https://rl.uni-freiburg.de/people/valada), and [Wolfram Burgard](https://www.utn.de/person/wolfram-burgard/). <br>
> &ast;Equal contribution. <br>
>
> *arXiv preprint arXiv:2403.17846*, 2024 <br>
> (Accepted for *Robotics: Science and Systems (RSS), Delft, Netherlands*, 2024.)
<p align="center">
<img src="media/teaser-hovsg-white.png" alt="HOV-SG allows the construction of accurate, open-vocabulary 3D scene graphs for large-scale and multi-story environments and enables robots to effectively navigate in them with language instructions." width="600" />
</p>
## 📰 Major Updates
- **[29 Aug 2024]** **We added `hm3dsem_walks` dataset generation and hierarchical scene graph evaluation code.** <br>
Please review the updated code structure and newly added dependencies for dataset construction. <br><br>
- [01 Jul 2024] Initial release of HOV-SG including mapping and graph construction engine.
## 🏗 Setup
1. Clone and set up the HOV-SG repository
```bash
git clone https://github.com/hovsg/HOV-SG.git
cd HOV-SG
# set up virtual environment and install habitat-sim afterwards separately to avoid errors.
conda env create -f environment.yaml
conda activate hovsg
conda install habitat-sim -c conda-forge -c aihabitat
# set up the HOV-SG python package
pip install -e .
```
### OpenCLIP
HOV-SG uses the Open CLIP model to extract features from RGB-D frames. To download the Open CLIP model checkpoint `CLIP-ViT-H-14-laion2B-s32B-b79K` please refer to [Open CLIP](https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K).
```bash
mkdir checkpoints
wget https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K/resolve/main/open_clip_pytorch_model.bin?download=true -O checkpoints/temp_open_clip_pytorch_model.bin && mv checkpoints/temp_open_clip_pytorch_model.bin checkpoints/laion2b_s32b_b79k.bin
```
Another option is to use the OVSeg fine-tuned Open CLIP model, which is available under [here](https://github.com/facebookresearch/ov-seg):
```bash
pip install gdown
gdown --fuzzy https://drive.google.com/file/d/17C9ACGcN7Rk4UT4pYD_7hn3ytTa3pFb5/view -O checkpoints/ovseg_clip.pth
```
### SAM
HOV-SG uses [SAM](https://github.com/facebookresearch/segment-anything) to generate class-agnostic masks for the RGB-D frames. To download the SAM model checkpoint `sam_v2` execute the following:
```bash
wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth -O checkpoints/sam_vit_h_4b8939.pth
```
## 🖼️ Dataset Preparation
### Habitat Matterport 3D Semantics
HOV-SG takes posed RGB-D sequences as input. In order to produce hierarchical multi-story scenes we make use of the Habitat 3D Semantics dataset ([HM3DSem](https://aihabitat.org/datasets/hm3d-semantics/)).
- Download the [Habitat Matterport 3D Semantics](https://github.com/matterport/habitat-matterport-3dresearch) dataset. More specifically, download through the links corresponding to these filenames: [hm3d-val-habitat-v0.2.tar](https://api.matterport.com/resources/habitat/hm3d-val-habitat-v0.2.tar), [hm3d-val-semantic-annots-v0.2.tar](https://api.matterport.com/resources/habitat/hm3d-val-semantic-annots-v0.2.tar), [hm3d-val-semantic-configs-v0.2.tar]( https://api.matterport.com/resources/habitat/hm3d-val-semantic-configs-v0.2.tar).
<details>
<summary>Make sure that the raw HM3D dataset has the following structure:</summary>
```
├── hm3d
│ ├── hm3d_annotated_basis.scene_dataset_config.json # this file is necessary
│ ├── val
│ │ └── 00824-Dd4bFSTQ8gi
│ │ ├── Dd4bFSTQ8gi.basis.glb
│ │ ├── Dd4bFSTQ8gi.basis.navmesh
│ │ ├── Dd4bFSTQ8gi.glb
│ │ ├── Dd4bFSTQ8gi.semantic.glb
│ │ └── Dd4bFSTQ8gi.semantic.txt
...
...
...
```
</details>
We used the following scenes from the Habitat Matterport 3D Semantics dataset in our evaluation:
<details>
<summary>Show Scenes ID</summary>
1. `00824-Dd4bFSTQ8gi`
2. `00829-QaLdnwvtxbs`
3. `00843-DYehNKdT76V`
4. `00861-GLAQ4DNUx5U`
5. `00862-LT9Jq6dN3Ea`
6. `00873-bxsVRursffK`
7. `00877-4ok3usBNeis`
8. `00890-6s7QHgap2fW`
</details>
1. Our method requires posed input data. Because of that, we recorded trajectories for each sequence we evaluate on. We provide a script (`hovsg/data/hm3dsem/gen_hm3dsem_walks_from_poses.py`) that turns a set of camera poses (`hovsg/data/hm3dsem/metadata/poses`) into a sequence of RGB-D observations using the [habitat-sim](https://github.com/facebookresearch/habitat-sim) simulator. The output includes RGB, depth, poses and frame-wise semantic/panoptic ground truth:
```bash
python data/habitat/gen_hm3dsem_from_poses.py --dataset_dir <hm3dsem_dir> --save_dir data/hm3dsem_walks/
```
2. Secondly, we construct a new hierarchical graph-structured dataset that is called `hm3dsem_walks` that includes ground truth based on all observations recorded. To produce this ground-truth data please execute the following: First, define the following config paths: `main.package_path`, `main.dataset_path`, `main.raw_data_path`, and `main.save_path` under `config/create_graph.yaml`. For each scene, define the `main.scene_id`, `main.split`. Next, execute the following to obtain floor-, region-, and object-level ground truth data per scene. We utilize every recorded frame without skipping (see parameter `dataset.hm3dsem.gt_skip_frames`) and recommend 128 GB of RAM to compile this as the scenes differ in size:
```bash
cd HOV-SG
python hovsg/data/hm3dsem/create_hm3dsem_walks_gt.py
```
To evaluate semantic segmentation cababilities, we used [ScanNet](http://www.scan-net.org/) and [Replica](https://github.com/facebookresearch/Replica-Dataset).
### ScanNet
To get an RGBD sequence for ScanNet, download the ScanNet dataset from the [official website](http://www.scan-net.org/). The dataset contains RGB-D frames compressed as .sens files. To extract the frames, use the [SensReader/python](https://github.com/ScanNet/ScanNet/blob/master/SensReader/python).
We used the following scenes from the ScanNet dataset:
<details>
<summary>Show Scenes ID</summary>
1. `scene0011_00`
2. `scene0050_00`
2. `scene0231_00`
3. `scene0378_00`
4. `scene0518_00`
</details>
### Replica
To get an RGBD sequence for Replica, Instead of the original Replica dataset, download the scanned RGB-D trajectories of the Replica dataset provided by [Nice-SLAM](https://github.com/cvg/nice-slam). It contains rendered trajectories using the mesh models provided by the original Replica datasets.
Download the Replica RGB-D scan dataset using the downloading [script](https://github.com/cvg/nice-slam/blob/master/scripts/download_replica.sh) in [Nice-SLAM](https://github.com/cvg/nice-slam#replica-1).
```bash
wget https://cvg-data.inf.ethz.ch/nice-slam/data/Replica.zip -O data/Replica.zip && unzip data/Replica.zip -d data/Replica_RGBD && rm data/Replica.zip
```
To evaluate against the ground truth semantics labels, you also need also to download the original Replica dataset from the [Replica](https://github.com/facebookresearch/Replica-Dataset) as it contains the ground truth semantics labels as .ply files.
```bash
git clone https://github.com/facebookresearch/Replica-Dataset.git data/Replica-Dataset
chmod +x data/Replica-Dataset/download.sh && data/Replica-Dataset/download.sh data/Replica_original
```
We only used the following scenes from the Replica dataset:
<details>
<summary>Show Scenes ID</summary>
1. `office0`
2. `office1`
3. `office2`
4. `office3`
5. `office4`
6. `room0`
7. `room1`
8. `room2`
</details>
## 📂 Datasets file strutcre
The Data folder should have the following structure:
<details>
<summary>Show data folder structure</summary>
```
├── hm3dsem_walks
│ ├── val
│ │ ├── 00824-Dd4bFSTQ8gi
│ │ │ ├── depth
│ │ │ │ ├── Dd4bFSTQ8gi-000000.png
│ │ │ │ ├── ...
│ │ │ ├── rgb
│ │ │ │ ├── Dd4bFSTQ8gi-000000.png
│ │ │ │ ├── ...
│ │ │ ├── semantic
│ │ │ │ ├── Dd4bFSTQ8gi-000000.png
│ │ │ │ ├── ...
│ │ │ ├── pose
│ │ │ │ ├── Dd4bFSTQ8gi-000000.png
│ │ │ │ ├── ...
| | ├── 00829-QaLdnwvtxbs
| | ├── ..
├── Replica
│ ├── office0
│ │ ├── results
│ │ │ ├── depth0000.png
│ │ │ ├── ...
│ │ | ├── rgb0000.png
│ │ | ├── ...
│ │ ├── traj.txt
│ ├── office1
│ ├── ...
├── ScanNet
│ ├── scans
│ │ ├── scene0011_00
│ │ │ ├── color
│ │ │ │ ├── 0.jpg
│ │ │ │ ├── ...
│ │ │ ├── depth
│ │ │ │ ├── 0.png
│ │ │ │ ├── ...
│ │ │ ├── poses
│ │ │ │ ├── 0.txt
│ │ │ │ ├── ...
│ │ │ ├── internsics
│ │ │ │ ├── intrinsics_color.txt
│ │ │ │ ├── intrinsics_depth.txt
│ │ ├── ..
```
</details>
## :rocket: Run
### Create scene graphs (only for Habitat Matterport 3D Semantics):
```bash
python application/create_graph.py main.dataset=hm3dsem main.dataset_path=data/hm3dsem_walks/val/00824-Dd4bFSTQ8gi/ main.save_path=data/scene_graphs/00824-Dd4bFSTQ8gi
```
<details>
<summary>This will generate a scene graph for the specified RGB-D sequence and save it. The following files are generated:</summary>
```
├── graph
│ ├── floors
│ │ ├── 0.json
│ │ ├── 0.ply
│ │ ├── 1.json
│ │ ├── ...
│ ├── rooms
│ │ ├── 0_0.json
│ │ ├── 0_0.ply
│ │ ├── 0_1.json
│ │ ├── ...
│ ├── objects
│ │ ├── 0_0_0.json
│ │ ├── 0_0_0.ply
│ │ ├── 0_0_1.json
│ │ ├── ...
│ ├── nav_graph
├── tmp
├── full_feats.pt
├── mask_feats.pt
├── full_pcd.ply
├── masked_pcd.ply
```
The `graph` folder contains the generated scene graph hierarchy, the first number in the file name represents the floor number, the second number represents the room number, and the third number represents the object number. The `tmp` folder holds intermediate results obtained throughout graph construction. The `full_feats.pt` and `mask_feats.pt` contain the features extracted from the RGBD frames using the Open CLIP and SAM models. the former contains per point features and the latter contains the features for the object masks. The `full_pcd.ply` and `masked_pcd.ply` contain the point cloud representation of the RGB-D frames and the instance masks of all objects, respectively.
</details>
### Visualize scene graph
```bash
python application/visualize_graph.py graph_path=data/scene_graphs/hm3dsem/00824-Dd4bFSTQ8gi/graph
```
![hovsg_graph_vis](media/hovsg_graph_vis.gif)
### Interactive visualization of scene graphs and natural language queries
#### Setup OpenAI
In order to test graph queries with HOV-SG, you need to setup an OpenAI API account with the following steps:
1. [Sign up an OpenAI account](https://openai.com/blog/openai-api), login your account, and bind your account with at least one payment method.
2. [Get you OpenAI API keys](https://platform.openai.com/account/api-keys), copy it.
3. Open your `~/.bashrc` file, paste a new line `export OPENAI_KEY=<your copied key>`, save the file, and source it with command `source ~/.bashrc`. Another way would be to run `export OPENAI_KEY=<your copied key>` in the teminal where you want to run the query code.
#### Evaluate query against pre-built hierarchical scene graph
```bash
python application/visualize_query_graph.py main.graph_path=data/scene_graphs/hm3dsem/00824-Dd4bFSTQ8gi/graph
```
After launching the code, you will be asked to input the hierarchical query. An example is `chair in the living room on floor 0`. You can see the visualization of the top 5 target objects and the room it lies in.
![hovsg_graph_query](media/hovsg_graph_query.gif)
### Extract feature map for semantic segmentation (only ScanNet and Replica)
```bash
python application/semantic_segmentation.py main.dataset=replica main.dataset_path=Replica/office0 main.save_path=data/sem_seg/office0
```
### Evaluate semantic segmentation (only ScanNet and Replica)
```bash
python application/eval/evaluate_sem_seg.py dataset=replica scene_name=office0 feature_map_path=data/sem_seg/office0
```
### Evaluate predicted scene graphs (only Habitat 3D Semantics)
- Define the scene identifiers and paths of ground truth and the predicted scene graph in the `config/eval_graph.yaml`.
- Run the graph evaluation method:
```bash
python application/eval/evaluate_graph.py
```
## 📔 Abstract
Recent open-vocabulary robot mapping methods enrich dense geometric maps with pre-trained visual-language features. While these maps allow for the prediction of point-wise saliency maps when queried for a certain language concept, largescale environments and abstract queries beyond the object level still pose a considerable hurdle, ultimately limiting languagegrounded robotic navigation. In this work, we present HOVSG, a hierarchical open-vocabulary 3D scene graph mapping approach for language-grounded indoor robot navigation. Leveraging open-vocabulary vision foundation models, we first obtain state-of-the-art open-vocabulary segment-level maps in 3D and subsequently construct a 3D scene graph hierarchy consisting of floor, room, and object concepts, each enriched with openvocabulary features. Our approach is able to represent multistory buildings and allows robotic traversal of those using a cross-floor Voronoi graph. HOV-SG is evaluated on three distinct datasets and surpasses previous baselines in open-vocabulary semantic accuracy on the object, room, and floor level while producing a 75% reduction in representation size compared to dense open-vocabulary maps. In order to prove the efficacy and generalization capabilities of HOV-SG, we showcase successful long-horizon language-conditioned robot navigation within realworld multi-story environments.
If you find our work useful, please consider citing our paper:
```
@article{werby23hovsg,
Author = {Abdelrhman Werby and Chenguang Huang and Martin Büchner and Abhinav Valada and Wolfram Burgard},
Title = {Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation},
Year = {2024},
journal = {Robotics: Science and Systems},
}
```
## 👩‍⚖️ License
For academic usage, the code is released under the [MIT](https://opensource.org/licenses/MIT) license.
For any commercial purpose, please contact the authors.
## 🙏 Acknowledgment
This work was funded by the German Research Foundation
(DFG) Emmy Noether Program grant number 468878300, the
BrainLinks-BrainTools Center of the University of Freiburg,
and an academic grant from NVIDIA.
plans/PRISM/.research/readmes/ok-robot.md
+138
View File
@@ -0,0 +1,138 @@
[![Intro image](https://github.com/ok-robot/ok-robot/assets/156981262/5634372e-9820-4681-bb1a-1c14541c229c)](https://ok-robot.github.io)
# `OK-Robot`
[![arXiv](https://img.shields.io/badge/arXiv-2401.12202-163144.svg?style=for-the-badge)](https://arxiv.org/abs/2401.12202)
![License](https://img.shields.io/github/license/notmahi/bet?color=873a7e&style=for-the-badge)
[![Code Style: Black](https://img.shields.io/badge/Code%20Style-Black-262626?style=for-the-badge)](https://github.com/psf/black)
[![PyTorch](https://img.shields.io/badge/Videos-Website-db6a4b.svg?style=for-the-badge&logo=airplayvideo)](https://ok-robot.github.io/)
[![Discord](https://img.shields.io/discord/1210368035324436551?style=for-the-badge&logo=discord&color=5865F2&label=discord&logoColor=white)](https://discord.gg/wzzZJxqKYC)
**Authors**: [<u>Peiqi Liu</u>*](https://leo20021210.github.io/), [<u>Yaswanth Orru</u>*](https://www.linkedin.com/in/yaswanth-orru/), [<u>Jay Vakil</u>](https://www.linkedin.com/in/jdvakil/), [<u>Chris Paxton</u>](https://cpaxton.github.io/), [<u>Mahi Shafiuallah</u>](https://mahis.life/)<sup>†</sup>, [<u>Lerrel Pinto</u>](https://www.lerrelpinto.com/)<sup>†</sup>
\* equal contribution, † equal advising.
OK-Robot is a zero-shot modular framework that effectively combines the state-of-art navigation and manipulation models to perform pick and place tasks in real homes. It has been tested in 10 real homes on 170+ objects and achieved a total success rate of 58.5%.
https://github.com/ok-robot/ok-robot/assets/156981262/7221264b-2fa3-4008-a757-0604cdc04e2b
## Hardware and software requirements
Hardware required:
* An iPhone Pro with Lidar sensors
* [Hello Robot Stretch](https://hello-robot.com/) with Dex Wrist installed
* A workstation with GPU to run pretrained models
Software required:
* Python 3.9
* Record3D (>1.18.0)
* [CloudCompare](https://www.danielgm.net/cc/release/)
## Installation
* You need to get anygrasp [license and checkpoint](./ok-robot-manipulation/anygrasp_license_registration/README.md).
* [Install](./docs/workspace-installation.md) the necessary environment on workstation to run the navigation and manipulation modules
* [Verify the workspace installation](./docs/installation-verification.md) once the above steps are completed.
* [Install](./docs/robot-installation.md) the necessary packages on robot to be able to properly communicate with backend workstation.
* You might also need to get a [new calibrated URDF](./docs/robot-calibration.md) for accurate robot manipulation.
Once both the robot and workstation are complete. You are good to start the experiments.
## Run Experiments
First [set up the environment](./docs/environment-setup.md) with the tapes, position the robot properly and scan the environment to get a r3d file from Record3D. Place it in `/navigation/r3d/` run following commands.
### On Workstation:
In one terminal run the [Navigation Module](./ok-robot-navigation/).
```
mamba activate ok-robot-env
cd ok-robot-navigation
python path_planning.py debug=False min_height={z coordinates of the ground tapes + 0.1} dataset_path='r3d/{your_r3d_filename}.r3d' cache_path='{your_r3d_filename}.pt' pointcloud_path='{your_r3d_filename}.ply'
```
In another terminal run the [Manipulation module](./ok-robot-manipulation/README.md)
```
mamba activate ok-robot-env
cd ok-robot-manipulation/src
python demo.py --open_communication --debug
```
### On Robot:
Before running anything on the robot, you need to calibrate it by
```
stretch_robot_home.py
```
Our robot codes rely on robot controllers provided by [home-robot](https://github.com/facebookresearch/home-robot). Just like running other home-robot based codes, you need to run two processes synchronously in two terminals.
In one terminal start the home-robot
```
roslaunch home_robot_hw startup_stretch_hector_slam.launch
```
In another terminal run the robot control. More details in [ok-robot-hw](./ok-robot-hw/README.md)
```
cd ok-robot-hw
python run.py -x1 [x1] -y1 [y1] -x2 [x2] -y2 [y2] -ip [your workstation ip]
```
## Citation
If you find this work useful, please consider citing:
```bibtex
@article{liu2024okrobot,
title={OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Robotics},
author={Liu, Peiqi and Orru, Yaswanth and Paxton, Chris and Shafiullah, Nur Muhammad Mahi and Pinto, Lerrel},
journal={arXiv preprint arXiv:2401.12202},
year={2024}
}
```
Our work is reliant on a lot of other publications and open source projects, if you find a particular component useful, please consider citing the original authors as well.
<details>
<summary>List of citations</summary>
```bibtex
@article{fang2023anygrasp,
title={Anygrasp: Robust and efficient grasp perception in spatial and temporal domains},
author={Fang, Hao-Shu and Wang, Chenxi and Fang, Hongjie and Gou, Minghao and Liu, Jirong and Yan, Hengxu and Liu, Wenhai and Xie, Yichen and Lu, Cewu},
journal={IEEE Transactions on Robotics},
year={2023},
publisher={IEEE}
}
@article{minderer2024scaling,
title={Scaling open-vocabulary object detection},
author={Minderer, Matthias and Gritsenko, Alexey and Houlsby, Neil},
journal={Advances in Neural Information Processing Systems},
volume={36},
year={2024}
}
@article{yenamandra2023homerobot,
title={HomeRobot: Open-Vocabulary Mobile Manipulation},
author={Yenamandra, Sriram and Ramachandran, Arun and Yadav, Karmesh and Wang, Austin and Khanna, Mukul and Gervet, Theophile and Yang, Tsung-Yen and Jain, Vidhi and Clegg, Alexander William and Turner, John and others},
journal={arXiv preprint arXiv:2306.11565},
year={2023}
}
```
</details>
## Roadmap
While OK-Robot can do quite a bit by itself, we think there are plenty of room for improvement for a zero-shot, home-dwelling robot. That's why we consider OK-Robot a living release, and will try to occassionally add new features to this. We also encourage you to take a look at the list below, and if you are interested, share your improvements with the community by contributing to this project.
- [x] Create OK-Robot, a shared platform for a zero-shot, open-vocab pick-and-place robot.
- [x] Integrate grasping primitive with AnyGrasp.
- [x] Integrate open-vocabulary navigation with VoxelMap.
- [x] Integrate heuristic based dropping.
- [ ] Improve documentation.
- [ ] Add error detection/recovery from failure while manipulating.
- [ ] Figure out interactive navigation: if an object is not found or a query is ambiguous, ask the end-user.
- [ ] Integrate with an open-source grasp perception model so that we can MIT-license all the dependencies.
plans/PRISM/.research/readmes/openmask3d.md
+193
View File
@@ -0,0 +1,193 @@
<!-- src: https://raw.githubusercontent.com/OpenMask3D/openmask3d/master/README.md -->
<p align="center">
<h1 align="center">OpenMask3D🛋: Open-Vocabulary 3D Instance Segmentation</h1>
<p align="center">
<a href="https://aycatakmaz.github.io/">Ay&#231;a Takmaz</a><sup>1*</sup></span>,
<a href="https://elisabettafedele.github.io/">Elisabetta Fedele</a><sup>1*</sup>
<br>
<a href="https://studios.disneyresearch.com/people/bob-sumner/">Robert W. Sumner</a><sup>1</sup>,
<a href="https://people.inf.ethz.ch/pomarc/">Marc Pollefeys</a><sup>1,2</sup>,
<a href="https://federicotombari.github.io/">Federico Tombari</a><sup>1,3</sup>,
<a href="https://francisengelmann.github.io/">Francis Engelmann</a><sup>1,3</sup>
<br>
<sup>1</sup>ETH Zurich,
<sup>2</sup>Microsoft,
<sup>3</sup>Google <br>
<sup>*</sup>equal contribution
</p>
<h2 align="center">NeurIPS 2023</h2>
<h3 align="center"><a href="https://github.com/OpenMask3D/openmask3d">Code</a> | <a href="https://arxiv.org/abs/2306.13631">Paper</a> | <a href="https://openmask3d.github.io">Project Page</a> </h3>
<div align="center"></div>
</p>
<p align="center">
<a href="">
<img src="https://openmask3d.github.io/static/images/teaser.jpeg" alt="Logo" width="100%">
</a>
</p>
<p align="center">
<strong>OpenMask3D</strong> is a zero-shot approach for 3D instance segmentation with open-vocabulary queries.
Guided by predicted class-agnostic 3D instance masks, our model aggregates per-mask features via multi-view fusion of CLIP-based image embeddings.
</p>
<br>
---
## Setup 🛠
Clone the repository, create conda environment and install the required packages as follows:
```bash
conda create --name=openmask3d python=3.8.5 # create new virtual environment
conda activate openmask3d # activate it
bash install_requirements.sh # install requirements
pip install -e . # install current repository in editable mode
```
Note: If you encounter any issues in the `bash install_requirements.sh` step, we recommend you to run the commands in that script one-by-one, especially for performing the MinkowskiEngine installation manually.
---
## Run the pipeline on a single scene 🛋
In this section we provide some information about how to run the pipeline on a single scene. In particular, we divide this section into four parts:
1. Download **checkpoints**
2. Check the format of **scene's data**
3. Set-up **configurations**
4. **Run** OpenMask3D
### Step 1: Download the checkpoints 📍
Create a folder `resources` in the main directory of the repository. Then, add to this folder the checkpoints for:
* **Mask module network**: use [this link](https://drive.google.com/file/d/1emtZ9xCiCuXtkcGO3iIzIRzcmZAFfI_B/view?usp=sharing) (model trained on ScanNet200 training set) for evaluating on **ScanNet validation scenes**, or [this link](https://drive.google.com/file/d/1rD2Uvbsi89X4lSkont_jUTT7X9iaox9y/view?usp=share_link) for running the model on an **arbitrary scene**.
* **Segment Anything Model** (in our case we used ViT-H): use this [link](https://drive.google.com/file/d/1WHi0hBi0iqMZfk8l3rDXLrW4lEEgHm_y/view?usp=sharing) or the [official repository](https://github.com/facebookresearch/segment-anything#model-checkpoints).
### Step 2: Check the folder structure of the data for your scene 🛢
In order to run OpenMask3D you need to have access to the point cloud of the scene as well to the posed RGB-D frames.
We recommend creating a folder `scene_example` inside the `resources` folder where the data is saved with the following structure ([here](https://drive.google.com/file/d/1UOwBZMCrTMg-_MFwmYkKOrex1YS6Nw-i/view?usp=sharing) we provide a scene as an example).
```
scene_example
├── pose <- folder with camera poses
│ ├── 0.txt
│ ├── 1.txt
│ └── ...
├── color <- folder with RGB images
│ ├── 0.jpg (or .png/.jpeg)
│ ├── 1.jpg (or .png/.jpeg)
│ └── ...
├── depth <- folder with depth images
│ ├── 0.png (or .jpg/.jpeg)
│ ├── 1.png (or .jpg/.jpeg)
│ └── ...
├── intrinsic
│ └── intrinsic_color.txt <- camera intrinsics
└── scene_example.ply <- point cloud of the scene
```
Please note the followings:
* The **point cloud** should be provided as a `.ply` file and the points are expected to be in the z-up right-handed coordinate system.
* The **camera intrinsics** and **camera poses** should be provided in a `.txt` file, containing a 4x4 matrix.
* The **RGB images** and the **depths** can be either in `.png`, `.jpg`, `.jpeg` format; the used format should be specified as explained in **Step 3**.
* The **RGB images** and their corresponding **depths** and **camera poses** should be named as `{FRAME_ID}.extension`, without zero padding for the frame ID, starting from index 0.
### Step 3: Set-up the paths to data and to output folders 🛤
Before running OpenMask3D make sure to fill all the required parameters in [this script](run_openmask3d_single_scene.sh). In particular, if you have followed the structure provided in Step 2, you should adapt only the following fields:
* `SCENE_DIR`: directory to `scene_example`
* `SCENE_INTRINSIC_RESOLUTION`: resolution on which intrinsics are computed
* `IMG_EXTENSION`: extension of RGB pictures. Either `.png`, `.jpg`, `.jpeg`
* `DEPTH_EXTENSION`: extension of depth pictures. Either `.png`, `.jpg`, `.jpeg`
* `DEPTH_SCALE`: factor by which the depth of the sensor should be divided to obtain a measure in terms of meters. It should be set to 1000 for ScanNet depth images and to 6553.5 for Replica depth images. You should set this value based on the scale of your depth maps.
* `MASK_MODULE_CKPT_PATH`: path to the mask module network checkpoint
* `SAM_CKPT_PATH`: path to the Segment Anything Model (SAM) checkpoint
* `OUTPUT_FOLDER_DIRECTORY`: path to the folder in which you wish to save the outputs
* `SAVE_VISUALIZATIONS`: set to true if you wish to save the visualizations of the class-agnostic masks
* `SAVE_CROPS`: set to true if you wish to save the 2D crops of the masks from which the CLIP features are extracted. It can be helpful for debugging and for a qualitative evaluation of the quality of the masks.
* `OPTIMIZE_GPU_USAGE`: set to true if you have some memory constraints and wish to minimize GPU memory footprint. Please note that this version is slower compared to the our default version.
### Step 4: Run OpenMask3D 🚀
Now you can run OpenMask3D by using the following command.
```bash
bash run_openmask3d_single_scene.sh
```
This script first extracts and saves the class-agnostic masks, and then computes the per-mask features. Masks and mask-features are saved into the directory specified by the user at the beginning of [this script](run_openmask3d_single_scene.sh). In particular, the output has the following structure.
```
OUTPUT_FOLDER_DIRECTORY
└── date-time-experiment_name <- folder with the output of a specific experiment
├── crops <- folder with crops (if SAVE_CROPS=true)
├── hydra_outputs <- folder with outputs from hydra (config.yaml files are useful)
├── scene_example_masks.pt <- class-agnostic instance masks - dim. (num_points, num_masks) indicating the masks in which a given point is included
└── scene_example_openmask3d_features.npy <- per-mask features for each object instance - dim. (num_masks, num_features), the mask-feature vecture for each instance mask.
```
Note: For the ScanNet validation, we use available segments on ScanNet and obtain more robust and less noisy masks compared to directly running the mask predictor on the point cloud. Therefore, the results we obtain for a single scene from ScanNet directly using the point cloud can be different then the masks obtained during the overall ScanNet evaluation described in the section below.
---
## Other Configs ⚙️
Other configuration parameters can be modified from [this file](openmask3d/configs/openmask3d_inference.yaml). Here we provide some clarifications of other configuration parameters:
- `multi_level_expansion_ratio`: factor of increment of the crop dimension for using multi-level image crops
- `openmask3d.frequency`: the frequency with which we want to process the frames given in input (e.g. a frequency of 10 takes 1 image in every 10 frames)
- `openmask3d.num_random_rounds` and `openmask3d.num_selected_points`: sets the number of iterations and the number of sampled points for SAM.
---
## Closed-vocabulary 3D instance segmentation evaluation on ScanNet200 📊
In this section we outline the steps to take in order to reproduce our results on the ScanNet200 validation set. In particular, we divide this section into four parts:
1. Download and preprocess the **ScanNet200** dataset
2. Check the format of ScanNet200 dataset
3. Set-up the paths to data and to output folders
2. Run evaluation
### Step 1: Download and pre-process the ScanNet200 dataset 📍
First, you need to download the ScanNet200 dataset as explained [here](https://kaldir.vc.in.tum.de/scannet_benchmark/documentation).
Once you have the dataset, you have to clone the [ScanNet repository](https://github.com/ScanNet/ScanNet) and process the dataset by using the following command.
```
cd class_agnostic_mask_computation
python -m datasets.preprocessing.scannet_preprocessing preprocess \
--data_dir="PATH_TO_ORIGINAL_SCANNET_DATASET" \
--save_dir="data/processed/scannet" \
--git_repo="PATH_TO_SCANNET_GIT_REPO" \
--scannet200=true
```
### Step 2: Check the format of ScanNet200 dataset 🛢
Make sure to have the data in the following form.
```bash
scans <- out folder
├── scene_0011_00
│ ├── data
│ │ ├── intrinsic <- folder with the intrinsics
│ │ └── pose <- folder with the poses
│ ├── data_compressed
│ │ ├── color <- folder with the color images
│ │ └── depth <- folder with the depth images
│ └── scene_0011_00_vh_clean_2.ply <- path to the point cloud/mesh ply file
├── scene0011_01
│ ├── data
│ │ ├── intrinsic
│ │ └── pose
│ ├── data_compressed
│ │ ├── color
│ │ └── depth
│ └── scene_0011_01_vh_clean_2.ply
...
```
### Step 3: Set-up paths to data and to output folders 🛤
Modify the paths and parameters in [this script](run_openmask3d_scannet200_eval.sh), following the instructions provided there.
### Step 4: Run OpenMask3D on ScanNet200 🚀
Now you can compute the per-mask scene features and run the evaluation of OpenMask3D on the whole ScanNet200 dataset by using the following command:
```bash
bash run_openmask3d_scannet200_eval.sh
```
This script first extracts and saves the class-agnostic masks, and then computes the mask features associated with each extracted mask. Afterwards, the evaluation script automatically runs in order to obtain 3D closed-vocabulary semantic instance segmentation scores.
---
## Citation :pray:
```
@inproceedings{takmaz2023openmask3d,
title={{OpenMask3D: Open-Vocabulary 3D Instance Segmentation}},
author={Takmaz, Ay{\c{c}}a and Fedele, Elisabetta and Sumner, Robert W. and Pollefeys, Marc and Tombari, Federico and Engelmann, Francis},
booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
year={2023}
}
```
plans/PRISM/.research/readmes/openscene.md
+250
View File
@@ -0,0 +1,250 @@
<!-- src: https://raw.githubusercontent.com/pengsongyou/openscene/main/README.md -->
<!-- PROJECT LOGO -->
<p align="center">
<h1 align="center"><img src="https://pengsongyou.github.io/media/openscene/logo.png" width="40">OpenScene: 3D Scene Understanding with Open Vocabularies</h1>
<p align="center">
<a href="https://pengsongyou.github.io"><strong>Songyou Peng</strong></a>
·
<a href="https://www.kylegenova.com/"><strong>Kyle Genova</strong></a>
·
<a href="https://www.maxjiang.ml/"><strong>Chiyu "Max" Jiang</strong></a>
·
<a href="https://taiya.github.io/"><strong>Andrea Tagliasacchi</strong></a>
<br>
<a href="https://people.inf.ethz.ch/pomarc/"><strong>Marc Pollefeys</strong></a>
·
<a href="https://www.cs.princeton.edu/~funk/"><strong>Thomas Funkhouser</strong></a>
</p>
<h2 align="center">CVPR 2023</h2>
<h3 align="center"><a href="https://arxiv.org/abs/2211.15654">Paper</a> | <a href="https://youtu.be/jZxCLHyDJf8">Video</a> | <a href="https://pengsongyou.github.io/openscene">Project Page</a></h3>
<div align="center"></div>
</p>
<p align="center">
<a href="">
<img src="https://pengsongyou.github.io/media/openscene/teaser.jpg" alt="Logo" width="100%">
</a>
</p>
<p align="center">
<strong>OpenScene</strong> is a zero-shot approach to perform a series of novel 3D scene understanding tasks using open-vocabulary queries.
</p>
<br>
<!-- TABLE OF CONTENTS -->
<details open="open" style='padding: 10px; border-radius:5px 30px 30px 5px; border-style: solid; border-width: 1px;'>
<summary>Table of Contents</summary>
<ol>
<li>
<a href="#interactive-demo">Interactive Demo</a>
</li>
<li>
<a href="#installation">Installation</a>
</li>
<li>
<a href="#data-preparation">Data Preparation</a>
</li>
<li>
<a href="#run">Run</a>
</li>
<li>
<a href="#applications">Applications</a>
</li>
<li>
<a href="#todo">TODO</a>
</li>
<li>
<a href="#acknowledgement">Acknowledgement</a>
</li>
<li>
<a href="#citation">Citation</a>
</li>
</ol>
</details>
## News :triangular_flag_on_post:
- [2023/10/27] Add the code for LSeg per-pixel feature extraction and multi-view fusion. Check [this repo](https://github.com/pengsongyou/lseg_feature_extraction).
- [2023/03/31] Code is released.
## Interactive Demo
### No GPU is needed! Follow **[this instruction](./demo)** to set up and play with the real-time demo yourself.
<p align="center">
<img src="./media/demo.gif" width="75%" />
</p>
Here we present a **real-time**, **interactive**, **open-vocabulary** scene understanding tool. A user can type in an arbitrary query phrase like **`snoopy`** (rare object), **`somewhere soft`** (property), **`made of metal`** (material), **`where can I cook?`** (activity), **`festive`** (abstract concept) etc, and the correponding regions are highlighted.
## Installation
Follow the [installation.md](installation.md) to install all required packages so you can do the evaluation & distillation afterwards.
## Data Preparation
We provide the **pre-processed 3D&2D data** and **multi-view fused features** for the following datasets:
- ScanNet
- Matterport3D
- nuScenes
- Replica
### Pre-processed 3D&2D Data
You can preprocess the dataset yourself, see the [data pre-processing instruction](scripts/preprocess/README.md).
Alternatively, we have provided the preprocessed datasets. One can download the pre-processed datasets by running the script below, and following the command line instruction to download the corresponding datasets:
```bash
bash scripts/download_dataset.sh
```
The script will download and unpack data into the folder `data/`. One can also download the dataset somewhere else, but link to the corresponding folder with the symbolic link:
```bash
ln -s /PATH/TO/DOWNLOADED/FOLDER data
```
<details>
<summary><strong>List of provided processed data</strong> (click to expand):</summary>
- ScanNet 3D (point clouds with GT semantic labels)
- ScanNet 2D (RGB-D images with camera poses)
- Matterport 3D (point clouds with GT semantic labels)
- Matterport 2D (RGB-D images with camera poses)
- nuScenes 3D (lidar point clouds with GT semantic labels)
- nuScenes 2D (RGB images with camera poses)
- Replica 3D (point clouds)
- Replica 2D (RGB-D images)
- Matterport 3D with top 40 NYU classes
- Matterport 3D with top 80 NYU classes
- Matterport 3D with top 160 NYU classes
</details>
**Note**: 2D processed datasets (e.g. `scannet_2d`) are only needed if you want to do multi-view feature fusion on your own. If so, please follow the [instruction for multi-view fusion](./scripts/feature_fusion/README.md).
### Multi-view Fused Features
To evaluate our OpenScene model or distill a 3D model, one needs to have the multi-view fused image feature for each 3D point (see method in Sec. 3.1 in the paper).
You can run the following to directly download provided fused features:
```bash
bash scripts/download_fused_features.sh
```
<details>
<summary><strong>List of provided fused features</strong> (click to expand):</summary>
- ScanNet - Multi-view fused OpenSeg features, train/val (234.8G)
- ScanNet - Multi-view fused LSeg features, train/val (175.8G)
- Matterport - Multi-view fused OpenSeg features, train/val (198.3G)
- Matterport - Multi-view fused OpenSeg features, test set (66.7G)
- Replica - Multi-view fused OpenSeg features (9.0G)
- Matterport - Multi-view fused LSeg features (coming)
- nuScenes - Multi-view fused OpenSeg features (coming)
- nuScenes - Multi-view fused LSeg features (coming)
</details>
Alternatively, you can also generate multi-view features yourself following the [instruction](./scripts/feature_fusion/README.md).
## Run
When you have installed the environment and obtained the **processed 3D data** and **multi-view fused features**, you are ready to run our OpenScene disilled/ensemble model for 3D semantic segmentation, or distill your own model from scratch.
### Evaluation for 3D Semantic Segmentation with a Pre-defined Labelsets
<p align="center">
<img src="./media/benchmark_screenshot.jpg" width="80%" />
</p>
Here you can evaluate OpenScene features on different dataset (ScanNet/Matterport3D/nuScenes/Replica) that have pre-defined labelsets.
We already include the following labelsets in [label_constants.py](dataset/label_constants.py):
- ScanNet 20 classes (`wall`, `door`, `chair`, ...)
- Matterport3D 21 classes (ScanNet 20 classes + `floor`)
- Matterport top 40, 80, 160 NYU classes (more rare object classes)
- nuScenes 16 classes (`road`, `bicycle`, `sidewalk`, ...)
The general command to run evaluation:
```bash
sh run/eval.sh EXP_DIR CONFIG.yaml feature_type
```
where you specify your experiment directory `EXP_DIR`, and replace `CONFIG.yaml` with the correct config file under [`config/`](./config/). **`feature_type`** corresponds to per-point OpenScene features:
- `fusion`: The 2D multi-view fused features
- `distill`: features from 3D distilled model
- `ensemble`: Our 2D-3D ensemble features
To evaluate with `distill` and `ensemble`, the easiest way is to use a pre-trained 3D distilled model. You can do this by using one of the config files with postfix `_pretrained`.
For example, to evaluate the semantic segmentation on Replica, you can simply run:
```bash
# 2D-3D ensemble
sh run/eval.sh out/replica_openseg config/replica/ours_openseg_pretrained.yaml ensemble
# Run 3D distilled model
sh run/eval.sh out/replica_openseg config/replica/ours_openseg_pretrained.yaml distill
# Evaluate with 2D fused features
sh run/eval.sh out/replica_openseg config/replica/ours_openseg_pretrained.yaml fusion
```
The script will automatically download the pretrained 3D model and run the evaluation for Matterport 21 classes.
You can find all outputs in the `out/replica_openseg`.
For evaluation options, see under `TEST` inside `config/replica/ours_openseg_pretrained.yaml`. Below are important evaluation options that you might want to modify:
- `labelset` (default: None, `scannet`| `matterport` | `matterport40`| `matterport80`|`matterport160`): Evaluate on a specific pre-defined labelset in [label_constants.py](./dataset/label_constants.py). If not specified, same as your 3D point cloud folder name
- `eval_iou` (default: True): whether evaluating the mIoU. Set to `False` if there is no GT labels
- `save_feature_as_numpy` (default: False): save the per-point features as `.npy`
- `prompt_eng` (default: True): input class name X -> "a X in a scene"
- `vis_gt` (default: True): visualize point clouds with GT semantic labels
- `vis_pred` (default: True): visualize point clouds with our predicted semantic labels
- `vis_input` (default: True): visualize input point clouds
If you want to use a 3D model distilled from scratch, specify the `model_path` to the correponding checkpoints `EXP/model/model_best.pth.tar`.
### Distillation
Finally, if you want to distill a new 3D model from scratch, run:
- Start distilling:
```sh run/distill.sh EXP_NAME CONFIG.yaml```
- Resume:
```sh run/resume_distill.sh EXP_NAME CONFIG.yaml```
For available distillation options, please take a look at `DISTILL` inside `config/matterport/ours_openseg.yaml`
### Using Your Own Datasets
1. Follow the [data preprocessing instruction](./scripts/preprocess/README.md), modify codes accordingly to obtain the processed 2D&3D data
2. Follow the [feature fusion instruction](./scripts/feature_fusion/README.md), modify codes to obtain multi-view fused features.
3. You can distill a model on your own, or take our provided 3D distilled model weights (e.g. our 3D model for ScanNet or Matterport3D), and modify the `model_path` accordingly.
4. If you want to evaluate on a specific labelset, change the `labelset` in config.
## Applications
Besides the zero-shot 3D semantic segmentation, we can perform also the following tasks:
- **Open-vocabulary 3D scene understanding and exploration**: query a 3D scene to understand properties that extend beyond fixed category labels, e.g. materials, activity, affordances, room type, abstract concepts...
- **Rare object search**: query a 3D scene database to find rare examples based on their names
- **Image-based 3D object detection**: query a 3D scene database to retrieve examples based on similarities to a given input image
## Acknowledgement
We sincerely thank Golnaz Ghiasi for providing guidance on using OpenSeg model. Our appreciation extends to Huizhong Chen, Yin Cui, Tom Duerig, Dan Gnanapragasam, Xiuye Gu, Leonidas Guibas, Nilesh Kulkarni, Abhijit Kundu, Hao-Ning Wu, Louis Yang, Guandao Yang, Xiaoshuai Zhang, Howard Zhou, and Zihan Zhu for helpful discussion. We are also grateful to Charles R. Qi and Paul-Edouard Sarlin for their proofreading.
We build some parts of our code on top of the [BPNet repository](https://github.com/wbhu/BPNet).
## TODO
- [ ] Support demo for arbitrary scenes
- [ ] Support in-webiste demo
- [x] Support multi-view feature fusion with LSeg
- [x] Add missing multi-view fusion LSeg feature for Matterport & nuScenes
- [x] Add missing multi-view fusion OpenSeg feature for nuScenes
- [x] Multi-view feature fusion code for nuScenes
- [ ] Support the latest PyTorch version
We are very much welcome all kinds of contributions to the project.
## Citation
If you find our code or paper useful, please cite
```bibtex
@inproceedings{Peng2023OpenScene,
title = {OpenScene: 3D Scene Understanding with Open Vocabularies},
author = {Peng, Songyou and Genova, Kyle and Jiang, Chiyu "Max" and Tagliasacchi, Andrea and Pollefeys, Marc and Funkhouser, Thomas},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year = {2023}
```