Foveated Rendering - A New Way To See Inside VR

Foveated rendering is a technique of rendering that developers are looking at as an alternative for current generation rendering techniques.

Foveated rendering can be defined as a type of rendering, where an eye tracker (in a virtual reality headset) is used to reduce the rendering workload by significantly reducing the quality of the output images, that are present in the peripheral vision of the user. The peripheral vision zone is the area where the fovea is not focussing on.

What is the fovea, you ask? Well, the fovea is an integral part of the human eye. It is a centrally located depression region in the eye, which is comprised of densely packed photoreceptor cells. Almost half of the nerve fibres carry information from the fovea, while the other fifty per cent carry information from the other parts of the retina. It is also responsible for the sharp central (foveal) vision and is the only area of the eye where perfect vision is achieved, and the distinction of fine detail and colour is possible.

Some advantages of foveated rendering are:

1. Performance load is decreased while rendering
2. Reduces on-screen pixel-to-be-rendered count, per frame
3. Allows high-resolution images at gaze point (even with hardware and bandwidth limitations)
4. Improves latency in rendering pipelines
5. Provides a more natural human-like vision inside Virtual Reality Environments

The current implementation of foveated rendering environments is quite interesting. An eye tracker identifies the gaze and focuses point of the user, keeps track of it, and then an algorithm basically blurs out the peripheral images. This reduces the number of pixels that need to be rendered on the screen per frame, which greatly increases performance. An example can be seen:

There are several problems that come with foveated rendering, however. Some of the basic ones involve during the eye-tracking phase. The human eye exhibits six different types of motion when looking at and focussing on a particular object, namely:

1. Saccades - Relocation of the fovea to important features in a scene, while stopping momentarily to focus on those features (these shortstops are called fixations)

1. Smooth pursuit - Slow rotation that occurs when tracking a moving subject in the human field of vision (cannot exceed 30 degrees per second)
2. Vestibulo-Occular Reflex - Involuntary eye-movements that happen to counter the head movement to provide image stabilization
3. Optokinetics - Slow and fast-paced eye movements
4. Convergence and Divergence
5. Microsaccades

These are just the initial problems that need to be addressed for the tracking to be done properly. There are also several issues in the rendering pipeline, that come in effect during the later stages.

In short, foveated rendering is a new and up-and-coming subject matter, that if done properly has a lot of potential, in not only delivering quality experiences but also having the ability to not be hardware limited.

Some related links:

Oculus Explains VR Foveated Rendering
Types of Foveated Rendering
Video by Facebook Reality Labs
NVidia Deepfocus Algorithm