Hetero-associative Sequential Memory with Neuromorphic Signals Enables Mobile Manipulator Action Decisions

Researchers are developing increasingly intelligent robots capable of complex interactions with their environment, and a crucial step towards this goal involves equipping them with robust memory systems. Runcong Wang, Fengyi Wang, and Gordon Cheng, from the Institute for Cognitive Systems at the Technical University of Munich, have created a novel memory model that allows robots to learn and recall sequences of actions based on tactile input. Their system encodes sensory information and robot movements into a compact, efficient format, enabling a mobile manipulator to respond appropriately to touch and execute complex grasp sequences. This achievement represents a significant advance in robot control, offering a pathway towards more adaptable, intuitive, and economical robotic systems capable of seamless interaction with the physical world. Researchers have created a novel memory model that allows robots to learn and recall sequences of actions based on tactile input, offering a pathway towards more adaptable, intuitive, and economical robotic systems. This achievement represents a significant advance in robot control, enabling seamless interaction with the physical world. The system utilizes a modern Hopfield network as its foundation, a type of recurrent neural network that functions as an associative memory, storing and recalling patterns, in this case, sequences of robot actions, based on partial or noisy inputs.

Rotary Position Embedding encodes the order of actions in a sequence, representing the position of each action to allow the network to understand temporal relationships. Hyperdimensional Computing uses high-dimensional vectors to represent data, enabling efficient pattern matching and robust information storage. The system integrates data from tactile sensors on the robot’s skin, providing feedback about interaction with the environment and allowing the robot to adapt its actions. Researchers are also exploring Spiking Neural Networks, potentially leading to lower energy consumption and improved robustness., The system operates by encoding a sequence of robot actions into a high-dimensional vector using techniques like Rotary Position Embedding and Hyperdimensional Computing. This encoded sequence is then stored in the Hopfield network. When the robot receives tactile input, the network recalls the corresponding sequence of actions. The robot executes these actions and receives feedback from its tactile sensors, refining its movements and improving performance. Experiments, conducted on a physical robot, demonstrate the system’s capabilities in tasks such as grasping, manipulation, and object placement. The results show improved performance compared to traditional robotic control methods and other machine learning approaches.

This research has significant implications for creating more robust, adaptive, and energy-efficient robots capable of interacting with humans in a more natural and intuitive way.,. Compact Bindings for Robot Tactile Memory Scientists have engineered a novel hetero-associative sequential memory system for mobile manipulators, focusing on compact bindings between robot joint states and tactile observations to drive step-wise action decisions with minimal computational and memory demands.

The team encoded robot joint angles using population place coding, representing each angle with a distribution of active neurons. Skin-measured forces were converted into spike-rate features using an Izhikevich neuron model, a biologically inspired approach to simulating neuronal activity. Both signals were transformed into bipolar binary vectors, a format chosen for efficient storage and retrieval, and bound element-wise to create associations stored within a large-capacity sequential memory., To enhance pattern separation in binary space and incorporate geometric information from touch, researchers introduced 3D rotary positional embeddings, which rotate subspaces based on sensed force direction, effectively encoding the orientation of applied forces. This innovation enables fuzzy retrieval through a softmax weighted recall, allowing the system to respond appropriately even with imprecise or incomplete tactile input by considering temporally shifted action patterns. The system was validated on a Toyota Human Support Robot equipped with robot skin, demonstrating a pseudo-compliance controller that moves a robotic link in response to touch, with movement direction and speed correlating to the amplitude of the applied force. Experiments demonstrated the system’s ability to retrieve multi-joint grasp sequences based on continuing tactile input, showcasing its capacity for complex manipulation tasks.

Results demonstrate successful execution of both single-joint and full-arm behaviors via associative recall, suggesting potential extensions to areas such as imitation learning, motion planning, and multi-modal integration of sensory information.,.

Robotic Action Recall via Tactile Input Scientists have developed a hetero-associative sequential memory system for mobile manipulators that efficiently stores and retrieves robotic action sequences, demonstrating a new approach to robot control and learning.

The team encoded robot joint positions and tactile forces into a compact binary representation, storing associations between these signals to trigger subsequent movements. A key innovation lies in the use of 3D rotary positional embeddings, which enhance the separation of tactile signals and improve the system’s ability to respond appropriately to varying force directions., Experiments conducted on a Toyota Human Support Robot covered in robot skin demonstrate the system’s capabilities, realizing a pseudo-compliance controller that moves a robotic link in response to touch, correlating direction and speed with the amplitude of applied force. The system successfully retrieves multi-joint grasp sequences by continuing tactile input, showcasing its ability to execute complex movements based on touch alone. Tests demonstrate both single-joint and full-arm behaviors executed via associative recall, confirming the system’s effectiveness in controlling robotic movement. The research team achieved efficient storage and retrieval of long sequential patterns in a memory-efficient form, paving the way for applications in imitation learning, motion planning, and multi-modal integration.,.

Tactile Learning Enables Robot Action and Grasping This work presents a hetero-associative sequential memory system that enables robots to learn and execute actions based on tactile input with limited computational resources. The system encodes robot joint positions and tactile forces into a compact binary representation, storing associations between these signals to trigger subsequent movements. A key innovation lies in the use of 3D rotary positional embeddings, which enhance the separation of tactile signals and improve the system’s ability to respond appropriately to varying force directions., Experiments on a mobile manipulator demonstrate the system’s effectiveness in two applications: a pseudo-compliance controller that adjusts robot motion in response to touch, and a tactile-guided grasp execution that coordinates multi-joint movements. The system successfully executed reaching and grasping actions based solely on tactile input, demonstrating a degree of generalization beyond simple memorization of specific sequences. The researchers acknowledge that the system’s performance is dependent on the quality of the initial training data and the accuracy of the tactile sensors. Future work may focus on integrating this memory system with other learning paradigms, such as imitation learning, and exploring. 👉 More information🗞 A Hetero-Associative Sequential Memory Model Utilizing Neuromorphic Signals: Validated on a Mobile Manipulator🧠 ArXiv: https://arxiv.org/abs/2512.07032 Tags: