Foveated images with adaptive exposure

Patent metadata

Field	Value
Publication number	US-12425743-B2
Application number	US-202318316710-A
Country	US
Kind code	B2
Filing date	May 12, 2023
Priority date	May 12, 2023
Publication date	Sep 23, 2025
Grant date	Sep 23, 2025

Abstract

Systems and methods are disclosed that address the need for adaptive exposure within high dynamic range (HDR) images. Solutions can leverage recent advances in the use of virtual reality (VR) headsets and Augmented Reality (AR) displays equipped with infrared (IR) eye tracking devices. A gaze vector determined by the eye tracking device identifies one or more fixation points on the image that corresponds to an area where there exists a faulty exposure. The exposure around the fixation point can be adaptively corrected using image processing techniques. Using spatial adaptive exposure, the resulting image, a type of foveated image, can be rendered on a low dynamic range (LDR) display with sufficient detail.

First claim

The invention claimed is:

1. A method, comprising:
identifying image data having an underexposed area or an overexposed area;
determining a fixation point in the image data based on eye motion of an observer;
superimposing a probability distribution onto a plurality of pixels within a radius of the fixation point by setting a mean of the probability distribution to the fixation point;
for the pixels in the plurality of pixels within the radius:
computing pixel intensities for the plurality of pixels within the radius of the fixation point, and
adjusting an exposure setting of the pixels as a function of a location of the pixels relative to the fixation point, the pixel intensities, the probability distribution, and a dark threshold value or a light threshold value; and

causing display of the image data to a user.

2. The method of claim 1 , wherein the dark threshold value is within about 10% of a top intensity.

3. The method of claim 1 , wherein the light threshold value is within about 10% of a bottom intensity.

4. The method of claim 1 , wherein the plurality of pixels within the radius of the fixation point encompasses less than 10% of the image data.

5. The method of claim 1 , wherein the image data is derived from a synthetic 3D scene.

6. The method of claim 1 , wherein the probability distribution is a Gaussian distribution.

7. The method of claim 1 , wherein the radius is defined according to the probability distribution.

8. A system, comprising:
a display;
a world-facing camera attached to the display, the world-facing camera configured to produce a image data;
a gaze tracking device attached to the display, the gaze tracking device configured to superimpose a probability distribution onto a plurality of pixels within a radius of a fixation point by setting a mean of the probability distribution to the fixation point; and
a graphics processing unit (GPU), communicatively coupled to the world-facing camera, the gaze tracking device, and the display, the GPU configured to adaptively correct exposures of the plurality of pixels to produce a foveated image for projection onto the display by computing a pixel intensity a pixel of the plurality of pixels and adjusting an exposure setting of the pixel as a function of a location of the pixel relative to the fixation point, the pixel intensity, the probability distribution, and a dark threshold value or a light threshold value.

9. The system of claim 8 , wherein the display is a low dynamic range display.

10. The system of claim 8 , wherein the display is a head-mounted display suitable for augmented reality and virtual reality use.

11. The system of claim 8 , wherein the image data is a high dynamic range image.

12. The system of claim 8 , wherein the image data has at least 10-bit resolution and the display has 8-bit resolution.

13. The system of claim 8 , wherein the gaze tracking device is an infrared gaze tracking device configured to track a user's eye gaze at a frequency of 120 Hz.

14. The system of claim 8 , wherein the foveated image is an image of a user's field of view.

15. A headset, comprising:
a wearable display;
a world-facing camera attached to the wearable display;
a gaze tracking device attached to the wearable display; and
a graphics processing unit (GPU) coupled to the world-facing camera, the gaze tracking device, and the wearable display, the GPU configured to:
receive image data and an area of interest defined according to a fixation point and a probability distribution superimposed onto a plurality of pixels of the image data within a radius of the fixation point by setting a mean of the probability distribution to the fixation point;
adjust an exposure of a portion of the image data corresponding to the area of interest, to produce a foveated image by computing a pixel intensity a pixel of the plurality of pixels and adjusting an exposure setting of the pixel as a function of a location of the pixel relative to the fixation point, the pixel intensity, the probability distribution, and a dark threshold value or a light threshold value; and
project the foveated image onto the wearable display.

16. The headset of claim 15 , wherein the world-facing camera is configured to capture the image data as 360-degree image data and transmit the image data to the GPU.

17. The headset of claim 15 , wherein the world-facing camera is configured to capture the image data as image data of a scene and transmit the image data to the GPU.

18. The headset of claim 17 , wherein the gaze tracking device is configured to identify the area of interest in the scene, the area of interest indicating a faulty exposure in the image of the scene and transmit the area of interest to the GPU.

19. The headset of claim 18 , wherein the area of interest is identified by a substantially stationary gaze vector detected by the gaze tracking device.

20. The headset of claim 18 , wherein the GPU is configured to adjust a contrast value of a portion of the display identified by the gaze tracking device.

21. The headset of claim 15 , further comprising an inertial measurement unit (IMU) configured to measure a position of the wearable display and transmit the position to the GPU.

Assignees

• Google Llc

Inventors

• Du Ruofei
• Olwal Alex

Classifications

• G06T2207/20208

  High dynamic range [HDR] image processing · CPC title

• G06T5/20

  using local operators · CPC title

• G06T1/20

  Processor architectures; Processor configuration, e.g. pipelining · CPC title

• G06F3/013 Primary

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G02B27/0172

  characterised by optical features · CPC title