Abstract
In this paper, we aim to understand the functionality of 2D sketches by predicting how humans would interact with the objects depicted by sketches in real life. Given a 2D sketch, we learn to predict a tactile saliency map for it, which represents where humans would grasp, press, or touch the object depicted by the sketch. We hypothesize that understanding 3D structure and category of the sketched object would help such tactile saliency reasoning. We thus propose to jointly predict the tactile saliency, depth map and semantic category of a sketch in an end-to-end learning-based framework. To train our model, we propose to synthesize training data by leveraging a collection of 3D shapes with 3D tactile saliency information. Experiments show that our model can predict accurate and plausible tactile saliency maps for both synthetic and real sketches. In addition, we also demonstrate that our predicted tactile saliency is beneficial to sketch recognition and sketch-based 3D shape retrieval, and enables us to establish part-based functional correspondences among sketches.
Original language | English |
---|---|
Title of host publication | MM 2020 - Proceedings of the 28th ACM International Conference on Multimedia |
Publisher | Association for Computing Machinery |
Pages | 3072-3080 |
Number of pages | 9 |
ISBN (Electronic) | 9781450379885 |
ISBN (Print) | 9781450379885 |
DOIs | |
Publication status | Published - 12 Oct 2020 |
Event | 28th ACM International Conference on Multimedia, MM 2020 - Virtual, Online, United States Duration: 12 Oct 2020 → 16 Oct 2020 |
Publication series
Name | Proceedings of the ACM International Conference on Multimedia |
---|
Conference
Conference | 28th ACM International Conference on Multimedia, MM 2020 |
---|---|
Country/Territory | United States |
City | Virtual, Online |
Period | 12/10/20 → 16/10/20 |
Bibliographical note
Funding Information:Manfred Lau acknowledges the Hong Kong Research Grants Council (General Research Fund numbers 11206319 and 11205420).
Publisher Copyright:
© 2020 ACM.
Keywords
- saliency
- sketch
- tactile
ASJC Scopus subject areas
- Software
- Computer Graphics and Computer-Aided Design
- Human-Computer Interaction