Preserving and Combining Knowledge in Robotic Lifelong Reinforcement Learning

Yuan Meng1*, Zhenshan Bing1*†, Xiangtong Yao1*, Kejia Chen1,
Kai Huang2†, Yang Gao3, Fuchun Sun4†, Alois Knoll1,
1School of Computation, Information and Technology, Technical University of Munich, Germany
2School of Computer Science and Engineering, Sun Yat-sen University, China
3School of Intelligence Science and Technology, Nanjing University, China
4Department of Computer Science and Technology, Tsinghua University, China
*Indicates Equal Contribution
†To whom correspondence should be addressed; E-mail: zhenshan.bing@tum.de, huangk36@mail.sysu.edu.cn, fcsun@tsinghua.edu.cn
Paper Supplementary Code Data

Simulation and real-world demonstrations of acquired skills during lifelong reinforcemnet learning process. In this study, the agent is sequentially trained on multiple tasks, adhering to a lifelong reinforcement learning process. Our proposed framework employs a Bayesian non-parametric model to manage its prior knowledge space, allowing it to cluster and retain knowledge from the continuous stream of tasks, which significantly enhances overall task success. The video showcases both simulation and real-world demonstrations, illustrating that our framework successfully completes all tasks encountered during the lifelong learning process.

Demonstration of solving the long-horizon task ``clean the table''. Our agent successfully completes the long-horizon task by combining knowledge acquired from the previous lifelong learning process. Unlike recent studies that rely on human demonstrations, our agent demonstrates greater flexibility by learning each subtask independently, without the need for strict conditioning between tasks. This highlights its ability to generalize and adapt to more challenging non-parametric task distributions.

Abstract

Humans can continually accumulate knowledge and develop increasingly complex behaviors and skills throughout their lives, which is a capability known as ``lifelong learning''. Although this lifelong learning capability is considered an essential mechanism that makes up generalized intelligence, recent advancements in artificial intelligence predominantly excel in narrow, specialized domains and generally lack of this lifelong learning capability. Our study introduces a robotic lifelong reinforcement learning framework that addresses this gap by incorporating a non-parametric Bayesian model into the knowledge space. Additionally, we enhance the agent's semantic understanding of tasks by integrating language embeddings into the framework. Our proposed embodied agent can consistently accumulate knowledge from a continuous stream of one-time feeding tasks. Furthermore, our agent can tackle challenging real-world long-horizon tasks by combining and reapplying its acquired knowledge from the original tasks stream. Our findings demonstrate that intelligent embodied agents can exhibit a capability for lifelong learning similar to that of human beings. The proposed framework advances our understanding of the robotic lifelong learning process and may inspire the development of more broadly applicable intelligence.

Supplementary Videos

Paper (Preprint)

BibTeX


      @article{meng2024preserving,
        title={Preserving and combining knowledge in robotic lifelong reinforcement learning},
        author={Meng, Yuan and Bing, Zhenshan and Yao, Xiangtong and Chen, Kejia and Huang, Kai and Gao, Yang and Sun, Fuchun and Knoll, Alois},
        year={2024}
      }