Prompt-Driven Visuomotor Grasping
A neural model for grasping basic objects with the NICO robot. The initial and target images are fed into the model, along with the shape, color and position of the object to be grasped, and where it should be placed.
We present a biologically inspired neural model for the robotic tasks of object identification, localization, and motor action regression. The model is designed for: 1) Enabling a robot to reach objects in a three-dimensional space as this is a required ability for many real-world robotic applications; 2) Addressing the influence of imbalances in the training data on possible biases in the model’s behavior; 3) Evaluating the model’s performance on the auxiliary tasks of object localization and identification. We examine the effect of training these auxiliary tasks, along with the main task of reaching for an object, to gain a better understanding of the model’s performance and observe possible synergetic effects of learning the three tasks simultaneously.
