Transferring visuomotor learning from simulation to the real world for manipulation tasks in a humanoid robot

Research output: Chapter in Book/Report/Conference proceedingConference contribution


  • Phuong D. H. Nguyen
  • Tobias Fischer
  • Ugo Pattacini
  • Giorgio Metta
  • Yiannis Demiris

Colleges, School and Institutes

External organisations

  • Istituto Italiano di Tecnologia, Genova, Italy
  • Imperial College London


Hand-eye coordination is a requirement for many manipulation tasks including grasping and reaching. However, accurate hand-eye coordination has shown to be especially difficult to achieve in complex robots like the iCub humanoid. In this work, we solve the hand-eye coordination task using a visuomotor deep neural network predictor that estimates the arm’s joint configuration given a stereo image pair of the arm and the underlying head configuration. As there are various unavoidable sources of sensing error on the physical robot, we train the predictor on images obtained from simulation. The images from simulation were modified to look realistic using an image-to-image translation approach. In various experiments, we first show that the visuomotor predictor provides accurate joint estimates of the iCub’s hand in simulation. We then show that the predictor can be used to obtain the systematic error of the robot’s joint measurements on the physical iCub robot. We demonstrate that a calibrator can be designed to automatically compensate this error. Finally, we validate that this enables accurate reaching of objects while circumventing manual finecalibration of the robot.


Original languageEnglish
Title of host publicationIEEE/RSJ Conference on Intelligent Robots and Systems (IROS 2018)
Publication statusPublished - 7 Jan 2019
EventIEEE/RSJ Conference on Intelligent Robots and Systems (IROS 2018) - Madrid, Spain
Duration: 1 Oct 20185 Oct 2018

Publication series

NameIEEE International Workshop on Intelligent Robots and Systems (IROS)
ISSN (Print)2153-0858
ISSN (Electronic)2153-0866


ConferenceIEEE/RSJ Conference on Intelligent Robots and Systems (IROS 2018)