Skip to main navigation Skip to search Skip to main content

A framework for robotic manipulation tasks based on multiple zero shot models

  • Yifan Li
  • , Peiyang Jiang
  • , Chengpeng Chai
  • , Xuyang Zhang
  • , Chengguo Liu*
  • *Corresponding author for this work

    Research output: Contribution to journal › Article (Academic Journal) › peer-review

    7 Citations (Scopus)

    Abstract

    Humans tackle unknown tasks by integrating information from multiple sensory modalities. Existing robotic frameworks struggle to achieve effective multimodal manipulation, especially when sufficient training data is lacking. This study introduces “Panda Act”, a novel robotic manipulation mechanism that leverages large language models (LLMs) and multimodal zero-shot models. The manipulation strategies are generated by LLMs as Python code, which dynamically orchestrates a suite of zero-shot visual and auditory models to fulfil task requirements. This enables robots to execute multimodal manipulations without requiring additional training. Extensive experiments in both simulated and real-world environments demonstrate that this approach excels in task comprehension, zero-shot execution, and adaptability, opening new avenues for enhancing robot adaptability in uncertain environments.
    Original languageEnglish
    Article number31141
    Number of pages15
    JournalScientific Reports
    Volume15
    Issue number1
    Early online date24 Aug 2025
    DOIs
    Publication statusE-pub ahead of print - 24 Aug 2025

    Bibliographical note

    Publisher Copyright:
    © The Author(s) 2025.

    Keywords

    • LLMs
    • Uncertain environments
    • Robotic manipulation
    • Zero-shot learning

    Fingerprint

    Dive into the research topics of 'A framework for robotic manipulation tasks based on multiple zero shot models'. Together they form a unique fingerprint.

    Cite this