3D Finger CAPE: clicking action and position estimation under self-occlusions in egocentric viewpoint

Youngkyoon Jang, Seung-tak Noh, Hyung Jin Chang, Tae-kyun Kim, Woontack Woo

Research output: Contribution to journalArticlepeer-review

56 Citations (Scopus)


In this paper we present a novel framework for simultaneous detection of click action and estimation of occluded fingertip positions from egocentric viewed single-depth image sequences. For the detection and estimation, a novel probabilistic inference based on knowledge priors of clicking motion and clicked position is presented. Based on the detection and estimation results, we were able to achieve a fine resolution level of a bare hand-based interaction with virtual objects in egocentric viewpoint. Our contributions include: (i) a rotation and translation invariant finger clicking action and position estimation using the combination of 2D image-based fingertip detection with 3D hand posture estimation in egocentric viewpoint. (ii) a novel spatio-temporal random forest, which performs the detection and estimation efficiently in a single framework. We also present (iii) a selection process utilizing the proposed clicking action detection and position estimation in an arm reachable AR/VR space, which does not require any additional device. Experimental results show that the proposed method delivers promising performance under frequent self-occlusions in the process of selecting objects in AR/VR space whilst wearing an egocentric-depth camera-attached HMD.
Original languageEnglish
Pages (from-to)501-510
Number of pages10
JournalIEEE transactions on visualization and computer graphics
Issue number4
Early online date20 Jan 2015
Publication statusPublished - 18 Apr 2015


Dive into the research topics of '3D Finger CAPE: clicking action and position estimation under self-occlusions in egocentric viewpoint'. Together they form a unique fingerprint.

Cite this