This paper presents a novel approach to inpainting 3D regions of a scene, given masked multi-view images, by distilling a 2D diffusion model into a learned 3D scene representation (e.g. a NeRF). Unlike 3D generative methods that explicitly condition the diffusion model on camera pose or multi-view information, our diffusion model is conditioned only on a single masked 2D image. Nevertheless, we show that this 2D diffusion model can still serve as a generative prior in a 3D multi-view reconstruction problem where we optimize a NeRF using a combination of score distillation sampling and NeRF reconstruction losses. Predicted depth is used as additional supervision to encourage accurate geometry. We compare our approach to 3D inpainting methods that focus on object removal. And because our method can generate content to fill any 3D masked region, we additionally demonstrate 3D object completion, 3D object replacement, and 3D scene completion.
There's excellent related work that was introduced around the same time as ours.
NeRFiller introduces an approach to completing missing regions of a NeRF.
Flow-Guided Diffusion for Video Inpainting presents a method for inpainting videos.
@article{prabhu2023inpaint,
author = {Prabhu, Kira and Wu, Jane and Tsai, Lynn and Hedman, Peter and Goldman, Dan B and Poole, Ben and Broxton, Michael},
title = {Inpaint3D: 3D Scene Content Generation using 2D Inpainting Diffusion},
journal = {arXiv preprint arXiv:2312.03869},
year = {2023},
}