What are Neural Radiance Fields?

Neural Radiance Fields: A Quick Summary

Neural Radiance Fields, also known as NeRFs, are a new solution to perform inverse rendering, creating realistic and accurate visualizations.

Radiance Fields, used and represented in computer graphics, are made possible with a neural network that creates highly realistic 3D representations from a set of 2D images taken from different angles. These representations can be viewed from arbitrary angles in the scenes, where details such as reflections can change based on the viewing perspective.

As the name suggests, the neural network gets quite a lot done during the generation process. It takes a 5D input: the 3D spatial location in terms of the x, y, and z axis, as well as the viewing direction θ and φ. With these inputs, the algorithm will process the data and output the volume density and view-dependent emitted radiance at given spatial location.

For more information, please visit our good friend Michael Rubloff's dedicated website to learn more about NeRFs.

What is So Special about NeRFs?

Despite photogrammetry being the most tried and true method when it comes to 3D scanning, NeRFs excel in producing highly detailed and realistic 3D representations by learning and representing volumetric scenes with continuous radiance fields, capturing fine details and complex lighting effects that photogrammetry may miss. It generates photorealistic novel views from arbitrary camera positions, thanks to its ability to accurately interpolate between viewpoints. NeRF handles complex lighting scenarios, such as reflections, refractions, and shadows, more effectively by learning the light transport within the scene.

Radiance Field's Technological Diagram

In KIRI Engine's case, the alteration and wonder child of NeRFs: NSR, effectively solves the limitation of NeRFs not being directly usable as surface meshes within popular 3D editor applications such as Blender and Unreal Engine. The technology effectively uses neural network capabilities derived from NeRFs to accurately reconstruct shiny/reflective/transparent objects into detailed 3D models.

Neural Surface Reconstruction

Neural Surface Reconstruction, or NSR, is a proprietary technology developed by KIRI Innovations specifically to combat the limitations of traditional photogrammetry. It's made possible with Nvidia's Neuroangelo technology.

Neuralangelo RGB, Surface Reconstruction, and Surface Normals

Despite being the reliable method for producing realistic meshes, the technology of photogrammetry inherently struggles with surfaces that are shiny, reflective, translucent, or transparent. This is because photogrammetry is a technique that analyses the unique feature points on given objects through a set of photos taken around a given object. By being reflective/shiny/translucent/transparent, surfaces pose a great variation in their unique features, making them difficult to stay consistent enough for any photogrammetrical algorithms to analyze.

Since its original establishment, which aims to create affordable 3D scanning solutions for 3D creators around the world, KIRI Innovations has long researched effective ways to combat its own limitations. The importance of this research is very apparent: a dedicated 3D scanning platform should be able to scan objects of all surface characteristics.

Neural Surface Reconstruction was the effective solution concluding such research. This is a technology that, like photogrammetry, analyzes a series of photos captured from various angles around an object. However, instead of relying on the identification of common features in overlapping photos, NSR employs a Multilayer Perceptron, also known as a Neural Network, to anticipate the appearance of the object from any viewpoint and direction based on the available set of overlapping photos. In turn, the algorithm is able to reconstruct the photo set's respective surface mesh.

How does KIRI Use Neural Surface Reconstruction?

KIRI Engine has a dedicated scanning method that features Neural Surface Reconstruction technology, called Featureless Object Scan. As the name suggests, this scanning method effectively scans objects with 'featureless' surfaces, making it the ideal scanning method for objects such as toy cars, glasses, plastic bottles, etc.

KIRI Engine's Featureless Object Scan Interface

This scanning method prefers to use the video input format to generate 3D meshes. The continuous framing of the video allows the algorithm to analyze and anticipate the surfaces alongside the background and will proceed to reconstruct a mesh based on the object captured. Utilizing this technology, the input footage should be a video orbiting slowly around the object with three essential layers: top, middle, and bottom.

In essence, having NSR technology built into the KIRI Engine allows its users drastically increased versatility in 3D content creation, as it grants users to choose the most optimal scanning method for respective types of objects.