Strata-NeRF: Neural Radiance Fields for Stratified Scenes

1Vision and AI Lab, IISc Bangalore, 2Samsung R&D Institute, Bangalore, 3Brown University

Abstract

Neural Radiance Field (NeRF) approaches learn the un- derlying 3D representation of a scene and generate photo- realistic novel views with high fidelity. However, most pro- posed settings concentrate on modelling a single object or a single level of a scene. However, in the real world, we may capture a scene at multiple levels, resulting in a lay- ered capture. For example, tourists usually capture a mon- ument’s exterior structure before capturing the inner struc- ture. Modelling such scenes in 3D with seamless switch- ing between levels can drastically improve immersive ex- periences. However, most existing techniques struggle in modelling such scenes. We propose Strata-NeRF, a single neural radiance field that implicitly captures a scene with multiple levels. Strata-NeRF achieves this by condition- ing the NeRFs on Vector Quantized (VQ) latent represen- tations which allow sudden changes in scene structure. We evaluate the effectiveness of our approach in multi-layered synthetic dataset comprising diverse scenes and then fur- ther validate its generalization on the real-world RealEstate 10k dataset. We find that Strata-NeRF effectively captures stratified scenes, minimizes artifacts, and synthesizes high- fidelity views compared to existing approaches.

Video

Novel Views from RealEstate10K

Double click on the videos to view full-screen.

7 Rutledge Ave Highland Mills

Mip-NeRF 360

Strata-NeRF

Novel Views from Synthetic Dataset

Comparison between Ground Truth, Mip-NeRF 360 and Strata-NeRF (Ours).

Cube-Sphere-Monkey

Level 0

Level 1

Level 2

Dragon in Pyramid

Level 0

Level 1

Buddhist Temple

Level 0

Level 1

Level 2

Coffee Shop

Level 0

Level 1

Level 2

Acknowledgement

This work was supported by Samsung R&D Institute India, Bangalore, PMRF and Kotak IISc AIML Centre (KIAC). Srinath Sridhar was partly supported by NSF grant CNS-2038897