Modifying rendered image data based on ambient light from a physical environment

Patent metadata

Field	Value
Publication number	US-11545108-B2
Application number	US-202117158402-A
Country	US
Kind code	B2
Filing date	Jan 26, 2021
Priority date	Feb 3, 2020
Publication date	Jan 3, 2023
Grant date	Jan 3, 2023

Abstract

A method includes obtaining rendered image data that includes a representation of an object for display using a see-through display. The see-through display permits ambient light from a physical environment through the see-through display. The method includes sensing a plurality of light superposition characteristic values associated with the ambient light that quantifies the ambient light. The method includes determining a plurality of display correction values associated with the electronic device based on the plurality of light superposition characteristic values and predetermined display characteristics of the representation of the object. The method includes generating, from the rendered image data, display data for the see-through display in accordance with the plurality of display correction values in order to satisfy the predetermined display characteristics of the representation of the object within a performance threshold.

First claim

What is claimed is:

1. A method comprising:
at an electronic device including one or more processors, a non-transitory memory, and a see-through display:
obtaining rendered image data that includes a representation of an object for display using the see-through display, wherein the see-through display permits ambient light from a physical environment through the see-through display, and wherein the representation of the object is characterized by predetermined display characteristics;
sensing a plurality of light superposition characteristic values associated with the ambient light from the physical environment, wherein the plurality of light superposition characteristic values quantifies the ambient light arriving at the see-through display;
determining a plurality of display correction values associated with the electronic device based on the plurality of light superposition characteristic values and the predetermined display characteristics of the representation of the object; and
modifying the rendered image data in order to generate display data for the see-through display in accordance with the plurality of display correction values in order to satisfy the predetermined display characteristics of the representation of the object within a performance threshold.

2. The method of claim 1 , further comprising:
transmitting, via the one or more processors, the display data to the see-through display; and
displaying the display data on the see-through display.

3. The method of claim 1 , wherein sensing the plurality of light superposition characteristic values is performed by an image sensor.

4. The method of claim 1 , wherein sensing the plurality of light superposition characteristic values is performed by an ambient light sensor.

5. The method of claim 1 , wherein sensing the plurality of light superposition characteristic values includes determining scene content information associated with the physical environment.

6. The method of claim 1 , wherein the plurality of display correction values includes grid data.

7. The method of claim 6 , further comprising determining interpolated grid values by interpolating the grid data, wherein generating the display data includes adjusting the rendered image data according to the interpolated grid values.

8. The method of claim 1 , wherein the plurality of display correction values includes a combination of global luminance values associated with the physical environment and chromaticity values associated with the physical environment.

9. The method of claim 1 , wherein the plurality of display correction values includes a combination of local luminance values associated with the physical environment and chromaticity values associated with the physical environment.

10. The method of claim 1 , wherein the plurality of display correction values includes a combination of spatial luminance values associated with the physical environment and chromaticity values associated with the physical environment.

11. The method of claim 1 , wherein the display data is generated as a further function of scene content information associated with the rendered image data.

12. A system comprising:
a see-through display, wherein the see-through display permits ambient light from a physical environment through the see-through display;
a sensor subsystem to sense a plurality of light superposition characteristic values associated with the ambient light from the physical environment, wherein the plurality of light superposition characteristic values quantifies the ambient light arriving at the see-through display;
a rendering subsystem to generate rendered image data that includes a representation of an object, wherein the representation of the object is characterized by predetermined display characteristics;
a correction values generator to determine a plurality of display correction values associated with the system based on the plurality of light superposition characteristic values and the predetermined display characteristics of the representation of the object; and
a display pipeline to:
obtain the rendered image data that includes the representation of the object for display using the see-through display, and
modify the rendered image data in order to generate display data for the see-through display in accordance with the plurality of display correction values in order to satisfy the predetermined display characteristics of the representation of the object within a performance threshold.

13. The system of claim 12 , further comprising a head and body pose identifier to:
obtain head and body pose data, and
provide the head and body pose data to the display pipeline in order to generate the display data.

14. The system of claim 12 , wherein the sensor subsystem includes an image sensor or an ambient light sensor.

15. The system of claim 12 , wherein the correction values generator includes a grid data generator to generate grid data, wherein the plurality of display correction values includes the grid data.

16. The system of claim 15 , wherein the display pipeline includes a grid data interpolator to interpolate the grid data, and wherein the display pipeline generates the display data as a function of the interpolated grid data.

17. The system of claim 12 , wherein the correction values generator includes a color correction manager and a tint manager to collectively determine a combination of global luminance values associated with the physical environment and chromaticity values associated with the physical environment, wherein the combination of the global luminance values and the chromaticity values are included in the plurality of display correction values.

18. The system of claim 12 , wherein the correction values generator includes a color correction manager and a tint manager to collectively determine a combination of local luminance values associated with the physical environment and chromaticity values associated with the physical environment, wherein the combination of the local luminance values and the chromaticity values are included in the plurality of display correction values.

19. The system of claim 12 , wherein the correction values generator includes a color correction manager and a tint manager to collectively determine a combination of spatial luminance values associated with the physical environment and chromaticity values associated with the physical environment, wherein the combination of the spatial luminance values and the chromaticity values are included in the plurality of display correction values.

20. An electronic device comprising:
one or more processors;
a non-transitory memory;
a see-through display; and
one or more programs, wherein the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors, the one or more programs including instructions for:
obtaining rendered image data that includes a representation of an object for display using the see-through display, wherein the see-through display permits ambient light from a physical environment through the see-through display, and wherein the representation of the object is characterized by predetermined display characteristics;
sensing a plurality of light superposition characteristic values associated with the ambient light from the physical environment, wherein the plurality of light superposition characteristic values quantifies the ambient light arriving at the see-through display;
determining a plurality of display correcti

Assignees

• Apple Inc

Inventors

• Hazra Siddharth Shankar
• Habel Ralf
• Grundhoefer Anselm
• Movshovich Aleksandr M
• Khan Moinul
• Hussein Marwan

Classifications

• G09G2360/14

  Detecting light within display terminals, e.g. using a single or a plurality of photosensors · CPC title

• G09G5/10 Primary

  Intensity circuits · CPC title

• G02B27/017

  Head mounted · CPC title

• G02B2027/0118

  comprising devices for improving the contrast of the display / brillance control visibility · CPC title

• G06T15/005 Primary

  General purpose rendering architectures · CPC title