Collaborative learning for hand and object reconstruction with attention-guided graph convolution

Tze Ho Elden Tse, Kwang In Kim, Ales Leonardis, Hyung Jin Chang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

237 Downloads (Pure)

Abstract

Estimating the pose and shape of hands and objects under interaction finds numerous applications including augmented and virtual reality. Existing approaches for hand and object reconstruction require explicitly defined physical constraints and known objects, which limits its application domains. Our algorithm is agnostic to object models, and it learns the physical rules governing hand-object interaction. This requires automatically inferring the shapes and physical interaction of hands and (potentially unknown) objects. We seek to approach this challenging problem by proposing a collaborative learning strategy where two-branches of deep networks are learning from each other. Specifically, we transfer hand mesh information to the object branch and vice versa for the hand branch. The resulting optimisation (training) problem can be unstable, and we address this via two strategies: (i) attention-guided graph convolution which helps identify and focus on mutual occlusion and (ii) unsupervised associative loss which facilitates the transfer of information between the branches. Experiments using four widely-used benchmarks show that our framework achieves beyond state-of-the-art accuracy in 3D pose estimation, as well as recovers dense 3D hand and object shapes. Each technical component above contributes meaningfully in the ablation study.
Original languageEnglish
Title of host publication2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Pages1645-1664
Number of pages11
ISBN (Electronic)978-1-6654-6946-3
ISBN (Print)978-1-6654-6947-0
DOIs
Publication statusPublished - 27 Sept 2022
Event2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) - New Orleans, United States
Duration: 19 Jun 202224 Jun 2022

Publication series

NameProceedings. IEEE Computer Society Conference on Computer Vision and Pattern Recognition
PublisherIEEE
ISSN (Print)1063-6919
ISSN (Electronic)2575-7075

Conference

Conference2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Abbreviated titleCVPR 2022
Country/TerritoryUnited States
CityNew Orleans
Period19/06/2224/06/22

Fingerprint

Dive into the research topics of 'Collaborative learning for hand and object reconstruction with attention-guided graph convolution'. Together they form a unique fingerprint.

Cite this