Abstract
Humans tackle unknown tasks by integrating information from multiple sensory modalities. Existing robotic frameworks struggle to achieve effective multimodal manipulation, especially when sufficient training data is lacking. This study introduces “Panda Act”, a novel robotic manipulation mechanism that leverages large language models (LLMs) and multimodal zero-shot models. The manipulation strategies are generated by LLMs as Python code, which dynamically orchestrates a suite of zero-shot visual and auditory models to fulfil task requirements. This enables robots to execute multimodal manipulations without requiring additional training. Extensive experiments in both simulated and real-world environments demonstrate that this approach excels in task comprehension, zero-shot execution, and adaptability, opening new avenues for enhancing robot adaptability in uncertain environments.
| Original language | English |
|---|---|
| Article number | 31141 |
| Number of pages | 15 |
| Journal | Scientific Reports |
| Volume | 15 |
| Issue number | 1 |
| Early online date | 24 Aug 2025 |
| DOIs | |
| Publication status | E-pub ahead of print - 24 Aug 2025 |
Bibliographical note
Publisher Copyright:© The Author(s) 2025.
Keywords
- LLMs
- Uncertain environments
- Robotic manipulation
- Zero-shot learning
Fingerprint
Dive into the research topics of 'A framework for robotic manipulation tasks based on multiple zero shot models'. Together they form a unique fingerprint.Cite this
- APA
- Author
- BIBTEX
- Harvard
- Standard
- RIS
- Vancouver