Abstract
Self-supervised Object Segmentation (SOS) aims to segment objects without any annotations. Under conditions of multi-camera inputs, the structural, textural and geometrical consistency among each view can be leveraged to achieve fine-grained object segmentation. To make better use of the above information, we propose Surface representation based Self-supervised Object Segmentation (Surface-SOS), a new framework to segment objects for each view by 3D surface representation from multi-view images of a scene. To model high-quality geometry surfaces for complex scenes, we design a novel scene representation scheme, which decomposes the scene into two complementary neural representation modules respectively with a Signed Distance Function (SDF). Moreover, Surface-SOS is able to refine single-view segmentation with multi-view unlabeled images, by introducing coarse segmentation masks as additional input. To the best of our knowledge, Surface-SOS is the first self-supervised approach that leverages neural surface representation to break the dependence on large amounts of annotated data and strong constraints. These constraints typically involve observing target objects against a static background or relying on temporal supervision in videos. Extensive experiments on standard benchmarks including LLFF, CO3D, BlendedMVS, TUM and several real-world scenes show that Surface-SOS always yields finer object masks than its NeRF-based counterparts and surpasses supervised single-view baselines remarkably.
Original language | English |
---|---|
Pages (from-to) | 2018 - 2031 |
Journal | IEEE Transactions on Image Processing |
Volume | 33 |
DOIs | |
Publication status | Published - 12 Mar 2024 |
Bibliographical note
Funding Agency:Outstanding Talents Training Fund in Shenzhen
10.13039/501100017610-Shenzhen Science and Technology Program-Shenzhen Cultivation of Excellent Scientific and Technological Innovation Talents Project (Grant Number: RCJC20200714114435057)
10.13039/501100017610-Shenzhen Science and Technology Program-Shenzhen Hong Kong Joint (Grant Number: SGDX20211123144400001)
10.13039/501100001809-National Natural Science Foundation of China (Grant Number: U21B2012)
10.13039/501100007937-Migu Cultural Technology Company Ltd., (Migu)-Peking University Meta Vision Technology Innovation Laboratory
10.13039/501100000288-Royal Society (Grant Number: IES\R3\223050 and SIF\R1\231009)
Keywords
- multi-view object segmentation
- neural surface representation
- Self-supervised learning
ASJC Scopus subject areas
- Software
- Computer Graphics and Computer-Aided Design