Tactile Sketch Saliency

Jianbo Jiao, Ying Cao*, Manfred Lau, Rynson Lau*

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Citations (Scopus)


In this paper, we aim to understand the functionality of 2D sketches by predicting how humans would interact with the objects depicted by sketches in real life. Given a 2D sketch, we learn to predict a tactile saliency map for it, which represents where humans would grasp, press, or touch the object depicted by the sketch. We hypothesize that understanding 3D structure and category of the sketched object would help such tactile saliency reasoning. We thus propose to jointly predict the tactile saliency, depth map and semantic category of a sketch in an end-to-end learning-based framework. To train our model, we propose to synthesize training data by leveraging a collection of 3D shapes with 3D tactile saliency information. Experiments show that our model can predict accurate and plausible tactile saliency maps for both synthetic and real sketches. In addition, we also demonstrate that our predicted tactile saliency is beneficial to sketch recognition and sketch-based 3D shape retrieval, and enables us to establish part-based functional correspondences among sketches.
Original languageEnglish
Title of host publicationMM 2020 - Proceedings of the 28th ACM International Conference on Multimedia
PublisherAssociation for Computing Machinery
Number of pages9
ISBN (Electronic)9781450379885
ISBN (Print)9781450379885
Publication statusPublished - 12 Oct 2020
Event28th ACM International Conference on Multimedia, MM 2020 - Virtual, Online, United States
Duration: 12 Oct 202016 Oct 2020

Publication series

Name Proceedings of the ACM International Conference on Multimedia


Conference28th ACM International Conference on Multimedia, MM 2020
Country/TerritoryUnited States
CityVirtual, Online

Bibliographical note

Funding Information:
Manfred Lau acknowledges the Hong Kong Research Grants Council (General Research Fund numbers 11206319 and 11205420).

Publisher Copyright:
© 2020 ACM.


  • saliency
  • sketch
  • tactile

ASJC Scopus subject areas

  • Software
  • Computer Graphics and Computer-Aided Design
  • Human-Computer Interaction


Dive into the research topics of 'Tactile Sketch Saliency'. Together they form a unique fingerprint.

Cite this