AMD is trying to replace ray tracing with neural networks – AI learns how to produce frames with realistic lighting

AMD technicians shared their AI-driven frame-by-frame image generation method to instantly replace global illumination. This method is called “time-stable generated illumination using a one-step diffusion model”. The idea is to simulate global lighting through image generation, rather than computationally intensive methods like ray tracing.

Image source: AMD

Earlier this year, AMD’s Jack Huynh announced a new set of FSR Diamond technologies. It includes what is described in the article “Next Generation Neural Rendering”as well as improved scaling and machine learning-based multi-frame generation, ray regeneration and path tracing.

“In our work, we try to solve the global illumination problem as an image generation problem, — explained SungYe Kim, one of the developers. — Global illumination makes a scene look more realistic and is also the most expensive part of the rendering process. ” The proposed method uses a framework with direct illumination from the light source to the object without further reflections – as an initial hint for a stable diffusion 2.1 Turbo model.

The model is trained on a synthetic dataset containing 31,000 frames of pre-rendered internal scenes. Each frame contains 1 to 30 light sources and moving objects. So, according to the developers, “The model can see a variety of lighting conditions”. Training involved running the dataset 50 times on an AMD Instinct MI210 GPU.

Use sparse lighting, brightness, material and geometry signals to ensure generation is consistent with room and lighting parameters. The way frames are processed forces the model to focus specifically on lighting. Researchers also use so-called temporal VAE (TVAE) to improve consistency. It uses motion vectors to align the current frame with previous frames and stores a history of previous results to ensure there are no flickers or other visual artifacts.

The developers claim their approach “Restore soft shadows and reflected light”while other similar technologies “Creating illusions” and “Twisted Geometry”. The model reportedly runs at 0.29 seconds per frame on an RTX 3090 graphics card, which is about 3.45 frames per second at a resolution of 512 x 512 pixels and about 8-9 GB of video memory loaded. Considering this is just the global lighting stage and not the entire frame rendering pipeline, it seems like we’re still a long way off from gamers actually using this approach.