SeqHAND: RGB-Sequence-Based 3D Hand Pose and Shape Estimation

John Yang, Hyung Jin Chang, Seungeui Lee, Nojun Kwak

Research output: Chapter in Book/Report/Conference proceedingConference contribution


3D hand pose estimation based on RGB images has been studied for a long time. Most of the studies, however, have performed frame-by-frame estimation based on independent static images. In this paper, we attempt to not only consider the appearance of a hand but incorporate the temporal movement information of a hand in motion into the learning framework, which leads to the necessity of a large-scale dataset with sequential RGB hand images. We propose a novel method that generates a synthetic dataset that mimics natural human hand movements
by re-engineering annotations of an extant static hand pose dataset into pose-flows. With the generated dataset, we train a newly proposed recurrent framework, exploiting visuo-temporal features from sequential synthetic hand images and emphasizing smoothness of estimations with
temporal consistency constraints. Our novel training strategy of detaching the recurrent layer of the framework during domain finetuning from synthetic to real allows preservation of the visuo-temporal features learned from sequential synthetic hand images. Hand poses that are sequentially estimated consequently produce natural and smooth hand movements which lead to more robust estimations. Utilizing temporal information for 3D hand pose estimation significantly enhances general pose estimations by outperforming state-of-the-art methods in our experiments on hand pose estimation benchmarks.
Original languageEnglish
Title of host publicationComputer Vision - ECCV 2020
Subtitle of host publication16th European Conference, Glasgow , UK, August 23-28 2020, Proceedings
Number of pages17
Publication statusAccepted/In press - 3 Jul 2020
Event16th European Conference on Computer Vision (ECCV2020) - Virtual Event
Duration: 23 Aug 202028 Aug 2020

Publication series

NameLecture Notes in Computer Science
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Conference16th European Conference on Computer Vision (ECCV2020)
CityVirtual Event


  • 3D Hand Pose Estimations
  • Pose- ow Generation
  • Syntheticto- real domain gap reduction
  • Synthetic hand motion dataset


Dive into the research topics of 'SeqHAND: RGB-Sequence-Based 3D Hand Pose and Shape Estimation'. Together they form a unique fingerprint.

Cite this