Fourier PlenOctrees for Dynamic Radiance Field Rendering in Real-time

Implicit neural representations such as Neural Radiance Field (NeRF) have largely focused on modeling static objects captured under multi-view settings where real-time rendering can be achieved with smart data structures, e.g., PlenOctree. In this paper, we present a novel Fourier PlenOctree (FPO) technique to tackle efficient neural modeling and real-time rendering of dynamic scenes captured under the free-view video (FVV) setting. The key idea in our FPO is an ingenious combination of generalized NeRF, PlenOctree representation, volumetric fusion and Fourier transform. For accelerate FPO construction, we present a novel coarse-to-fine fusion scheme that leverages generalizable NeRF technique to generate the tree via spatial blending. To tackle dynamic scenes, we tailor the implicit network to model the Fourier coefficients of time-varying density and color attributes. Finally we construct the FPO and train the Fourier coefficients directly on the leaves of a union PlenOctree structure of the dynamic sequence. We show the resulting FPO enables compact memory overload to handle dynamic objects and supports efficient fine-tuning. Extensive experiments show that the proposed method is 3000 times faster than original NeRF and achieves over an order of magnitude acceleration over SOTA while preserving high visual quality for free-viewpoint rendering of unseen dynamic scenes. Project Page | Paper