Skip to main navigation Skip to search Skip to main content

High-Fidelity Eye Animatable Neural Radiance Fields for Human Face

Research output: Working paper/Preprint › Preprint

4 Downloads (Pure)

Abstract

Face rendering using neural radiance fields (NeRF) is a rapidly developing research area in computer vision. While recent methods primarily focus on controlling facial attributes such as identity and expression, they often overlook the crucial aspect of modeling eyeball rotation, which holds importance for various downstream tasks. In this paper, we aim to learn a face NeRF model that is sensitive to eye movements from multi-view images. We address two key challenges in eye-aware face NeRF learning: how to effectively capture eyeball rotation for training and how to construct a manifold for representing eyeball rotation. To accomplish this, we first fit FLAME, a well-established parametric face model, to the multi-view images considering multi-view consistency. Subsequently, we introduce a new Dynamic Eye-aware NeRF (DeNeRF). DeNeRF transforms 3D points from different views into a canonical space to learn a unified face NeRF model. We design an eye deformation field for the transformation, including rigid transformation, e.g., eyeball rotation, and non-rigid transformation. Through experiments conducted on the ETH-XGaze dataset, we demonstrate that our model is capable of generating high-fidelity images with accurate eyeball rotation and non-rigid periocular deformation, even under novel viewing angles. Furthermore, we show that utilizing the rendered images can effectively enhance gaze estimation performance.
Original languageEnglish
PublisherarXiv
Number of pages14
DOIs
Publication statusPublished - 12 Sept 2023

Bibliographical note

BMVC2023 Oral

Keywords

  • cs.CV

Fingerprint

Dive into the research topics of 'High-Fidelity Eye Animatable Neural Radiance Fields for Human Face'. Together they form a unique fingerprint.
  • High-Fidelity Eye Animatable Neural Radiance Fields for Human Face

    Wang, H., Zhang, Z., Cheng, Y. & Chang, H. J., 24 Nov 2023, 34th British Machine Vision Conference 2023, {BMVC} 2023, Aberdeen, UK, November 20-24, 2023. BMVA , 14 p. 105

    Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

    Open Access

Cite this