Abstract
Forward biomechanical simulation in HCI holds great promise as a tool for evaluation, design, and engineering of user interfaces. Although reinforcement learning (RL) has been used to simulate biomechanics in interaction, prior work has relied on unrealistic assumptions about the control problem involved, which limits the plausibility of emerging policies. These assumptions include direct torque actuation as opposed to muscle-based control; direct, privileged access to the external environment, instead of imperfect sensory observations; and lack of interaction with physical input devices. In this paper, we present a new approach for learning muscle-actuated control policies based on perceptual feedback in interaction tasks with physical input devices. This allows modelling of more realistic interaction tasks with cognitively plausible visuomotor control. We show that our simulated user model successfully learns a variety of tasks representing different interaction methods, and that the model exhibits characteristic movement regularities observed in studies of pointing. We provide an open-source implementation which can be extended with further biomechanical models, perception models, and interactive environments.
Original language | English |
---|---|
Title of host publication | UIST '22 |
Subtitle of host publication | Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology |
Editors | Maneesh Agrawala, Jacob O. Wobbrock, Eytan Adar, Vidya Setlur |
Place of Publication | New York |
Publisher | Association for Computing Machinery (ACM) |
Number of pages | 14 |
ISBN (Electronic) | 9781450393201 |
DOIs | |
Publication status | Published - 28 Oct 2022 |
Event | 35th Annual ACM Symposium on User Interface Software and Technology, UIST 2022 - Bend, United States Duration: 29 Oct 2022 → 2 Nov 2022 |
Publication series
Name | UIST: User Interface Software and Technology |
---|---|
Publisher | Association for Computing Machinery (ACM) |
Conference
Conference | 35th Annual ACM Symposium on User Interface Software and Technology, UIST 2022 |
---|---|
Country/Territory | United States |
City | Bend |
Period | 29/10/22 → 2/11/22 |
Bibliographical note
Funding Information:A.I. is funded by the Academy of Finland Flagship programme “Finnish Center for Artifcial Intelligence” (FCAI). R.M-S. acknowledges funding support from EPSRC grant EP/R018634/1, Closed-loop Data Science and the Academy of Finland via FCAI. A.O. is supported by Academy of Finland project Human Automata.
Publisher Copyright:
© 2022 Owner/Author.
Keywords
- biomechanical models
- deep reinforcement learning
- simulation models
ASJC Scopus subject areas
- Software
- Human-Computer Interaction
- Computer Graphics and Computer-Aided Design