IllumiNeRF
3D Relighting Without Inverse Rendering

NeurIPS 2024

Xiaoming Zhao1,3, Pratul P. Srinivasan2, Dor Verbin2, Keunhong Park1, Ricardo Martin-Brualla1, Philipp Henzler1
1Google Research, 2Google DeepMind, 3University of Illinois Urbana-Champaign
arXiv Code (for Quantitative Results)
Main Page Stanford-ORB Gallery TensoIR Gallery CAT3D Gallery
Full Stanford-ORB Benchmark Full TensoIR Benchmark

Abstract

Existing methods for relightable view synthesis --- using a set of images of an object under unknown lighting to recover a 3D representation that can be rendered from novel viewpoints under a target illumination --- are based on inverse rendering, and attempt to disentangle the object geometry, materials, and lighting that explain the input images. Furthermore, this typically involves optimization through differentiable Monte Carlo rendering, which is brittle and computationally-expensive. In this work, we propose a simpler approach: we first relight each input image using an image diffusion model conditioned on lighting and then reconstruct a Neural Radiance Field (NeRF) with these relit images, from which we render novel views under the target lighting. We demonstrate that this strategy is surprisingly competitive and achieves state-of-the-art results on multiple relighting benchmarks.

How It Works


  1. Given a set of images and camera poses in (a), we run NeRF to extract the 3D geometry as in (b);
  2. Based on this geometry and a target light shown in (c), we create radiance cues for each given input view as in (d);
  3. Next, we independently relight each input image using a Relighting Diffusion Model illustrated in (e) and sample S possible solutions for each given image displayed in (f);
  4. Finally, we distill the relit set of images into a 3D representation through a Latent NeRF optimization as in (g) and (h).

3D Consistent Relighting

  • On the top: we show renderings from our final latent NeRF;
  • On the bottom: we show a diffusion sample from the nearest training view corresponding to each rendered frame on the top.

Related Works

Check out the following concurrent works which also introduce a (single-image) relighting diffusion model.

BibTeX

@inproceedings{zhao2024illuminerf,
    author    = {Xiaoming Zhao and Pratul P. Srinivasan and Dor Verbin and Keunhong Park and Ricardo Martin Brualla and Philipp Henzler},
    title     = {{IllumiNeRF: 3D Relighting Without Inverse Rendering}},
    booktitle = {NeurIPS},
    year      = {2024},
}