Vision-Guided MPC for Robotic Path Following Using Learned Memory-Augmented Model

Research output: Contribution to journalArticlepeer-review

95 Downloads (Pure)


The control of the interaction between the robot and environment, following a predefined geometric surface path with high accuracy, is a fundamental problem for contact-rich tasks such as machining, polishing, or grinding. Flexible path-following control presents numerous applications in emerging industry fields such as disassembly and recycling, where the control system must adapt to a range of dissimilar object classes, where the properties of the environment are uncertain. We present an end-to-end framework for trajectory-independent robotic path following for contact-rich tasks in the presence of parametric uncertainties. We formulate a combination of model predictive control with image-based path planning and real-time visual feedback, based on a learned state-space dynamic model. For modeling the dynamics of the robot-environment system during contact, we introduce the application of the differentiable neural computer, a type of memory augmented neural network (MANN). Although MANNs have been as yet unexplored in a control context, we demonstrate a reduction in RMS error of ∼ 21.0% compared with an equivalent Long Short-Term Memory (LSTM) architecture. Our framework was validated in simulation, demonstrating the ability to generalize to materials previously unseen in the training dataset.

Original languageEnglish
Article number688275
JournalFrontiers in Robotics and AI
Publication statusPublished - 26 Jul 2021

Bibliographical note

Copyright © 2021 Rastegarpanah, Hathaway and Stolkin.


  • cutting
  • dynamic modeling
  • electric vehicles
  • machine learning
  • predictive control
  • vision


Dive into the research topics of 'Vision-Guided MPC for Robotic Path Following Using Learned Memory-Augmented Model'. Together they form a unique fingerprint.

Cite this