---
title: "`OK-Robot`"
date: 2026-05-20
draft: false
tags: ["PRISM", "世界模型", "空间记忆", "机器人", "导航", "iOS"]
categories: ["worldmodel"]
---
[](https://ok-robot.github.io)
# `OK-Robot`
[](https://arxiv.org/abs/2401.12202)
[](https://github.com/psf/black)
[](https://ok-robot.github.io/)
[](https://discord.gg/wzzZJxqKYC)
**Authors**: [Peiqi Liu*](https://leo20021210.github.io/), [Yaswanth Orru*](https://www.linkedin.com/in/yaswanth-orru/), [Jay Vakil](https://www.linkedin.com/in/jdvakil/), [Chris Paxton](https://cpaxton.github.io/), [Mahi Shafiuallah](https://mahis.life/)†, [Lerrel Pinto](https://www.lerrelpinto.com/)†
\* equal contribution, † equal advising.
OK-Robot is a zero-shot modular framework that effectively combines the state-of-art navigation and manipulation models to perform pick and place tasks in real homes. It has been tested in 10 real homes on 170+ objects and achieved a total success rate of 58.5%.
https://github.com/ok-robot/ok-robot/assets/156981262/7221264b-2fa3-4008-a757-0604cdc04e2b
## Hardware and software requirements
Hardware required:
* An iPhone Pro with Lidar sensors
* [Hello Robot Stretch](https://hello-robot.com/) with Dex Wrist installed
* A workstation with GPU to run pretrained models
Software required:
* Python 3.9
* Record3D (>1.18.0)
* [CloudCompare](https://www.danielgm.net/cc/release/)
## Installation
* You need to get anygrasp [license and checkpoint](./ok-robot-manipulation/anygrasp_license_registration/README.md).
* [Install](./docs/workspace-installation.md) the necessary environment on workstation to run the navigation and manipulation modules
* [Verify the workspace installation](./docs/installation-verification.md) once the above steps are completed.
* [Install](./docs/robot-installation.md) the necessary packages on robot to be able to properly communicate with backend workstation.
* You might also need to get a [new calibrated URDF](./docs/robot-calibration.md) for accurate robot manipulation.
Once both the robot and workstation are complete. You are good to start the experiments.
## Run Experiments
First [set up the environment](./docs/environment-setup.md) with the tapes, position the robot properly and scan the environment to get a r3d file from Record3D. Place it in `/navigation/r3d/` run following commands.
### On Workstation:
In one terminal run the [Navigation Module](./ok-robot-navigation/).
```
mamba activate ok-robot-env
cd ok-robot-navigation
python path_planning.py debug=False min_height={z coordinates of the ground tapes + 0.1} dataset_path='r3d/{your_r3d_filename}.r3d' cache_path='{your_r3d_filename}.pt' pointcloud_path='{your_r3d_filename}.ply'
```
In another terminal run the [Manipulation module](./ok-robot-manipulation/README.md)
```
mamba activate ok-robot-env
cd ok-robot-manipulation/src
python demo.py --open_communication --debug
```
### On Robot:
Before running anything on the robot, you need to calibrate it by
```
stretch_robot_home.py
```
Our robot codes rely on robot controllers provided by [home-robot](https://github.com/facebookresearch/home-robot). Just like running other home-robot based codes, you need to run two processes synchronously in two terminals.
In one terminal start the home-robot
```
roslaunch home_robot_hw startup_stretch_hector_slam.launch
```
In another terminal run the robot control. More details in [ok-robot-hw](./ok-robot-hw/README.md)
```
cd ok-robot-hw
python run.py -x1 [x1] -y1 [y1] -x2 [x2] -y2 [y2] -ip [your workstation ip]
```
## Citation
If you find this work useful, please consider citing:
```bibtex
@article{liu2024okrobot,
title={OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Robotics},
author={Liu, Peiqi and Orru, Yaswanth and Paxton, Chris and Shafiullah, Nur Muhammad Mahi and Pinto, Lerrel},
journal={arXiv preprint arXiv:2401.12202},
year={2024}
}
```
Our work is reliant on a lot of other publications and open source projects, if you find a particular component useful, please consider citing the original authors as well.
List of citations
```bibtex
@article{fang2023anygrasp,
title={Anygrasp: Robust and efficient grasp perception in spatial and temporal domains},
author={Fang, Hao-Shu and Wang, Chenxi and Fang, Hongjie and Gou, Minghao and Liu, Jirong and Yan, Hengxu and Liu, Wenhai and Xie, Yichen and Lu, Cewu},
journal={IEEE Transactions on Robotics},
year={2023},
publisher={IEEE}
}
@article{minderer2024scaling,
title={Scaling open-vocabulary object detection},
author={Minderer, Matthias and Gritsenko, Alexey and Houlsby, Neil},
journal={Advances in Neural Information Processing Systems},
volume={36},
year={2024}
}
@article{yenamandra2023homerobot,
title={HomeRobot: Open-Vocabulary Mobile Manipulation},
author={Yenamandra, Sriram and Ramachandran, Arun and Yadav, Karmesh and Wang, Austin and Khanna, Mukul and Gervet, Theophile and Yang, Tsung-Yen and Jain, Vidhi and Clegg, Alexander William and Turner, John and others},
journal={arXiv preprint arXiv:2306.11565},
year={2023}
}
```
## Roadmap
While OK-Robot can do quite a bit by itself, we think there are plenty of room for improvement for a zero-shot, home-dwelling robot. That's why we consider OK-Robot a living release, and will try to occassionally add new features to this. We also encourage you to take a look at the list below, and if you are interested, share your improvements with the community by contributing to this project.
- [x] Create OK-Robot, a shared platform for a zero-shot, open-vocab pick-and-place robot.
- [x] Integrate grasping primitive with AnyGrasp.
- [x] Integrate open-vocabulary navigation with VoxelMap.
- [x] Integrate heuristic based dropping.
- [ ] Improve documentation.
- [ ] Add error detection/recovery from failure while manipulating.
- [ ] Figure out interactive navigation: if an object is not found or a query is ambiguous, ask the end-user.
- [ ] Integrate with an open-source grasp perception model so that we can MIT-license all the dependencies.