Robots Learn Tasks Sequentially Without Forgetting, Mirroring Human Skill Acquisition

Researchers are increasingly focused on continual reinforcement learning, a challenging field requiring agents to adapt to new tasks without losing previously learned skills. Yannick Denker and Alexander Gepperth, both from Fulda University of Applied Sciences, alongside their colleagues, address this need by presenting CRoSS, a novel and scalable robotic simulation suite. This benchmark utilises realistic physics within the Gazebo simulator and features two robotic platforms, a differential-drive robot and a seven-jointed robotic arm, to provide high task diversity. CRoSS significantly advances the field by offering a readily extensible, containerised environment for controlled studies, alongside baseline performance data for common reinforcement learning algorithms, thereby facilitating reproducible and scalable research into continual learning for robotics. Realistic robotic environments for evaluating continual reinforcement learning agents Scientists have developed a new benchmark suite, the Continual Robotic Simulation Suite (CRoSS), to advance research in continual reinforcement learning. This work addresses a critical need for realistic and scalable robotic environments capable of testing agents’ ability to learn sequentially without catastrophic forgetting. CRoSS introduces two robotic platforms, a two-wheeled differential-drive robot and a seven-joint robotic arm, simulated within the Gazebo simulator, offering high realism and extensibility. The two-wheeled robot is tasked with line-following and object-pushing, with variations in visual and structural parameters generating a large number of distinct challenges. Simultaneously, the robotic arm undertakes goal-reaching scenarios, employing both high-level Cartesian control and low-level joint angle manipulation, mirroring established benchmarks like Continual World but with enhanced physical fidelity. A key innovation within CRoSS is the provision of kinematics-only variants for the robotic arm, enabling significantly faster execution, two orders of magnitude quicker, when sensor data is not required. This design choice facilitates rapid experimentation and scalability without compromising the ability to incorporate realistic sensor input when needed. CRoSS is engineered for ease of use, featuring a containerized setup using Apptainer that ensures reproducibility across different platforms. Researchers have already demonstrated the suite’s capabilities by evaluating standard reinforcement learning algorithms, including Deep Q-Networks and policy gradient methods, establishing a foundation for comparative analysis. The suite’s architecture leverages Gazebo-Transport middleware, enabling seamless communication between sensors, actuators, and agents, and compatibility with the Robot Operating System (ROS). This compatibility paves the way for potential sim-to-real transfer learning, where policies trained in simulation can be deployed on physical robots with minimal adaptation. By combining high task diversity, realistic physics simulation, and a standardized, reproducible setup, CRoSS offers a powerful tool for accelerating progress in continual reinforcement learning research and addressing challenges in real-world robotic applications. The benchmark’s design allows for controlled studies of forgetting, transfer, and scalability, ultimately contributing to the development of more robust and adaptable intelligent agents. Robotic platforms and continual learning benchmark details A 72-qubit superconducting processor forms the foundation of this research, utilized to investigate continual reinforcement learning in realistic robotic simulations. Researchers developed the Continual Robotic Simulation Suite (CRoSS), a benchmark relying on the Gazebo simulator to assess agents learning sequential tasks without catastrophic forgetting. The study employs two robotic platforms: a two-wheeled differential-drive robot equipped with lidar, camera, and bumper sensors, and a seven-joint robotic arm. The differential-drive robot operates within line-following and object-pushing scenarios, with variations in visual and structural parameters generating a diverse range of tasks. For the robotic arm, the work implements two goal-reaching scenarios, one using high-level Cartesian hand position control and the other employing low-level control via joint angles. Additionally, kinematics-only variants of the robotic arm benchmarks were created, bypassing the need for full simulation and accelerating execution speeds by two orders of magnitude. CRoSS leverages Gazebo-Transport middleware for communication between sensors, actuators, and agents, ensuring compatibility with the Robot Operating System (ROS) and facilitating potential sim-to-real transfer. The environment manager adheres to Gymnasium API conventions, enabling seamless integration with existing reinforcement learning pipelines and libraries. To ensure reproducibility, the entire suite is packaged within a containerized setup using Apptainer, allowing out-of-the-box execution on Linux systems. Performance evaluations were conducted using standard reinforcement learning algorithms, including Deep Q-Networks (DQN) and policy gradient methods, to demonstrate the benchmark’s scalability and suitability for continual reinforcement learning research. This methodology allows for controlled studies of continual learning, focusing on forgetting, transfer, and scalability in robotic settings with high realism and extensibility. CRoSS benchmark design and task specifications for continual learning Researchers developed the Continual Robotic Simulation Suite (CRoSS), a new benchmark for continual reinforcement learning utilising realistic robot simulations within Gazebo. The suite features two robotic platforms: a two-wheeled differential-drive robot equipped with lidar, camera, and bumper sensors, and a seven-joint robotic arm. The differential-drive robot is employed in line-following and object-pushing scenarios, generating numerous distinct tasks through variations in visual and structural parameters. The robotic arm is used in goal-reaching scenarios, operating with both high-level cartesian hand position control and low-level joint angle control. For the robotic arm benchmarks, kinematic-only variants were created, bypassing the need for full simulation and achieving execution speeds two orders of magnitude faster. These kinematic variants maintain the same task definitions, reward functions, and termination conditions as the simulated benchmarks, but prioritise computational efficiency. Experiments were conducted on a High-Performance Computing cluster with 40 Linux workstations, each equipped with an Nvidia GTX 3090 graphics card, utilising TensorFlow 2.14 and Apptainer containers to ensure reproducibility. Evaluation of randomly selected tasks in the MPO and MLF benchmarks across default, simplified, and super-simplified settings revealed performance metrics. In the MLF benchmark, the average cumulated score per episode, normalised by episode length, ranged from 1.41 to 1.61 across settings and tasks. Specifically, task 51 achieved a score of 1.45 in the default setting, increasing to 1.61 in the super-simplified setting. MPO results demonstrated scores between 11.3 and 23.9, again varying with task and setting. Task 51 yielded a score of 21.0 in the default setting, rising to 23.5 in the super-simplified setting. These results were obtained by averaging three independent runs, providing a robust assessment of performance. Realistic robotic environments facilitate continual learning evaluation Scientists have developed a new benchmark suite called Continual Robotic Simulation Suite (CRoSS) for continual reinforcement learning research. This suite utilises realistically simulated robots within the Gazebo simulator, encompassing both a two-wheeled differential-drive robot and a seven-jointed robotic arm. The robots are tested across diverse tasks including line-following, object manipulation, and goal-reaching scenarios, with variations in visual and structural parameters to create a broad range of challenges. CRoSS extends existing benchmarks like Continual World by offering higher realism and addressing limitations in control models. The suite includes kinematic-only variants of the robotic arm tasks, enabling faster experimentation without requiring full simulation, and is designed for easy extensibility with almost arbitrary simulated sensors. Standard reinforcement learning algorithms, such as Deep Q-Networks and policy gradient methods, demonstrate significant catastrophic forgetting when sequentially trained on CRoSS tasks, highlighting the need for continual learning techniques. The authors acknowledge that the benchmark focuses on establishing a baseline with standard algorithms, rather than exploring advanced continual learning methods. Algorithms like SAC or PPO were not investigated due to challenges in controlling exploration within the continual learning framework. Future research could explore the application of more sophisticated algorithms and investigate methods to mitigate catastrophic forgetting, potentially through larger replay buffers or novel learning strategies. The availability of a containerized setup and performance data for baseline algorithms promotes reproducibility and facilitates scalable research in continual reinforcement learning for robotics. 👉 More information 🗞 CRoSS: A Continual Robotic Simulation Suite for Scalable Reinforcement Learning with High Task Diversity and Realistic Physics Simulation 🧠 ArXiv: https://arxiv.org/abs/2602.04868 Tags: