S2 contact: graph-based network for 3D hand-object contact estimation with semi-supervised learning

Tze Ho Elden Tse, Zhongqun Zhang*, Kwang In Kim, Ales Leonardis, Feng Zheng, Hyung Jin Chang

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contribution

67 Downloads (Pure)

Abstract

Despite the recent efforts in accurate 3D annotations in hand and object datasets, there still exist gaps in 3D hand and object reconstructions. Existing works leverage contact maps to refine inaccurate hand-object pose estimations and generate grasps given object models. However, they require explicit 3D supervision which is seldom available and therefore, are limited to constrained settings, e.g., where thermal cameras observe residual heat left on manipulated objects. In this paper, we propose a novel semi-supervised framework that allows us to learn contact from monocular images. Specifically, we leverage visual and geometric consistency constraints in large-scale datasets for generating pseudo-labels in semi-supervised learning and propose an efficient graph-based network to infer contact. Our semi-supervised learning framework achieves a favourable improvement over the existing supervised learning methods trained on data with ‘limited’ annotations. Notably, our proposed model is able to achieve superior results with less than half the network parameters and memory access cost when compared with the commonly-used PointNet-based approach. We show benefits from using a contact map that rules hand-object interactions to produce more accurate reconstructions. We further demonstrate that training with pseudo-labels can extend contact map estimations to out-of-domain objects and generalise better across multiple datasets. Project page is available.
Original languageEnglish
Title of host publicationComputer Vision – ECCV 2022
Subtitle of host publication17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part I
EditorsShai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, Tal Hassner
PublisherSpringer
Pages568–584
Number of pages17
Edition1
ISBN (Electronic)9783031197697
ISBN (Print)9783031197680
DOIs
Publication statusPublished - 23 Oct 2022
Event17th European Conference on Computer Vision, ECCV 2022 - Tel Aviv, Israel
Duration: 23 Oct 202227 Oct 2022

Publication series

NameLecture Notes in Computer Science
PublisherSpringer
Volume13661
ISSN (Print)0302-9743
ISSN (Electronic)611-3349

Conference

Conference17th European Conference on Computer Vision, ECCV 2022
Abbreviated titleECCV 2022
Country/TerritoryIsrael
CityTel Aviv
Period23/10/2227/10/22

Fingerprint

Dive into the research topics of 'S2 contact: graph-based network for 3D hand-object contact estimation with semi-supervised learning'. Together they form a unique fingerprint.

Cite this