TY - GEN
T1 - Diffuse3D
T2 - 2023 IEEE/CVF International Conference on Computer Vision (ICCV)
AU - Jiang, Yutao
AU - Zhou, Yang
AU - Liang, Yuan
AU - Liu, Wenxi
AU - Jiao, Jianbo
AU - Quan, Yuhui
AU - He, Shengfeng
PY - 2024/1/15
Y1 - 2024/1/15
N2 - This paper aims to resolve the challenging problem of wide-angle novel view synthesis from a single image, a.k.a. wide-angle 3D photography. Existing approaches rely on local context and treat them equally to inpaint occluded RGB and depth regions, which fail to deal with large-region occlusion (i.e., observing from an extreme angle) and foreground layers might blend into background inpainting. To address the above issues, we propose Diffuse3D which employs a pre-trained diffusion model for global synthesis, while amending the model to activate depth-aware inference. Our key insight is to alter the convolution mechanism in the denoising process. We inject depth information into the denoising convolution operation with bilateral kernels, i.e., a depth kernel and a spatial kernel, to consider layered correlations among pixels. In this way, foreground regions are overlooked in background inpainting and only pixels close in depth are leveraged. On the other hand, we propose a global-local balancing approach to maximize both contextual understandings. Extensive experiments demonstrate that our approach outperforms state-of-the-art methods in novel view synthesis, especially in wide-angle scenarios. More importantly, our method does not require any training and is a plug-and-play module that can be integrated with any diffusion model. Our code can be found at https://github.com/yutaojiang1/Diffuse3D.
AB - This paper aims to resolve the challenging problem of wide-angle novel view synthesis from a single image, a.k.a. wide-angle 3D photography. Existing approaches rely on local context and treat them equally to inpaint occluded RGB and depth regions, which fail to deal with large-region occlusion (i.e., observing from an extreme angle) and foreground layers might blend into background inpainting. To address the above issues, we propose Diffuse3D which employs a pre-trained diffusion model for global synthesis, while amending the model to activate depth-aware inference. Our key insight is to alter the convolution mechanism in the denoising process. We inject depth information into the denoising convolution operation with bilateral kernels, i.e., a depth kernel and a spatial kernel, to consider layered correlations among pixels. In this way, foreground regions are overlooked in background inpainting and only pixels close in depth are leveraged. On the other hand, we propose a global-local balancing approach to maximize both contextual understandings. Extensive experiments demonstrate that our approach outperforms state-of-the-art methods in novel view synthesis, especially in wide-angle scenarios. More importantly, our method does not require any training and is a plug-and-play module that can be integrated with any diffusion model. Our code can be found at https://github.com/yutaojiang1/Diffuse3D.
KW - Photography
KW - Training
KW - Computer vision
KW - Three-dimensional displays
KW - Image resolution
KW - Correlation
KW - Convolution
UR - https://jianbojiao.com/pdfs/iccv23_diffuse3d.pdf
U2 - 10.1109/ICCV51070.2023.00826
DO - 10.1109/ICCV51070.2023.00826
M3 - Conference contribution
SN - 9798350307191
T3 - International Conference on Computer Vision (ICCV)
SP - 8964
EP - 8974
BT - 2023 IEEE/CVF International Conference on Computer Vision (ICCV)
PB - IEEE
Y2 - 1 October 2023 through 6 October 2023
ER -