Abstract
Affordance refers to the perception of possible actions allowed by an object. Despite its relevance to human-computer interaction, no existing theory explains the mechanisms that underpin affordance-formation; that is, how affordances are discovered and adapted via interaction. We propose an integrative theory of affordance-formation based on the theory of reinforcement learning in cognitive sciences. The key assumption is that users learn to associate promising motor actions to percepts via experience when reinforcement signals (success/failure) are present. They also learn to categorize actions (e.g., "rotating"a dial), giving them the ability to name and reason about affordance. Upon encountering novel widgets, their ability to generalize these actions determines their ability to perceive affordances. We implement this theory in a virtual robot model, which demonstrates human-like adaptation of affordance in interactive widgets tasks. While its predictions align with trends in human data, humans are able to adapt affordances faster, suggesting the existence of additional mechanisms.
Original language | English |
---|---|
Title of host publication | CHI '22 |
Subtitle of host publication | CHI Conference on Human Factors in Computing Systems |
Editors | Simone Barbosa, Cliff Lampe, Caroline Appert, David A. Shamma, Steven Drucker, Julie Williamson, Koji Yatani |
Publisher | Association for Computing Machinery (ACM) |
Number of pages | 15 |
ISBN (Print) | 9781450391573 |
DOIs | |
Publication status | Published - 29 Apr 2022 |
Event | CHI '22: CHI Conference on Human Factors in Computing Systems - New Orleans, United States Duration: 29 Apr 2022 → 5 May 2022 |
Publication series
Name | Human factors in computing systems |
---|---|
ISSN (Print) | 1062-9432 |
Conference
Conference | CHI '22 |
---|---|
Country/Territory | United States |
City | New Orleans |
Period | 29/04/22 → 5/05/22 |
Bibliographical note
Funding Information:This project is funded by the Department of Communications and Networking (Aalto University), Finnish Center for Artifcial Intelligence (FCAI), Academy of Finland projects Human Automata (Project ID: 328813) and BAD (Project ID: 318559), and HumaneAI. We thank John Dudley for his support with data visualization and all study participants for their time commitment and valuable insights.
Publisher Copyright:
© 2022 ACM.
Keywords
- Action
- Adaptation
- Affordance
- Design
- Interaction
- Machine Learning
- Modeling
- Motion Planning
- Perception
- Reinforcement Learning
- Robotics
- Theory
ASJC Scopus subject areas
- Human-Computer Interaction
- Computer Graphics and Computer-Aided Design
- Software