Foveated image rendering for head-mounted display devices

Patent metadata

Field	Value
Publication number	US-11682106-B2
Application number	US-202117201973-A
Country	US
Kind code	B2
Filing date	Mar 15, 2021
Priority date	Dec 29, 2017
Publication date	Jun 20, 2023
Grant date	Jun 20, 2023

Abstract

Examples disclosed herein obtain first image data and the second image data for a foveated image frame to be displayed on a display, the first image data to have a first resolution and the second image data to have a second resolution lower than the first resolution. Disclosed examples also up-sample the second image data based on first metadata to generate up-sampled second image data, the up-sampled second image data to have the first resolution, and combine the first image data and the up-sampled second image data based on second metadata. Disclosed examples further perform, based on third metadata, a combination of at least two different filter operations on an overlap region including a portion of the first image data and a portion of the up-sampled second image data to generate the foveated image frame, the third metadata to specify a width in pixels of the overlap region.

First claim

What is claimed is:

1. A head-mounted display system comprising:
a display;
a host device to provide first image data and second image data for a foveated image frame to be displayed on the display, the first image data to have a first resolution and the second image data to have a second resolution lower than the first resolution; and
a display controller to:
obtain the first image data and the second image data from the host device;
up-sample the second image data based on first metadata from the host device to generate up-sampled second image data, the up-sampled second image data to have the first resolution;
combine the first image data and the up-sampled second image data based on second metadata from the host device; and
perform, based on third metadata from the host device, a combination of at least two different filter operations on a filter overlap region specified in the third metadata for the at least two different filter operations to generate the foveated image frame, the filter overlap region to include a portion of the first image data and a portion of the up-sampled second image data, the filter overlap region corresponding to a rectangular strip with a width specified in the third metadata.

2. The head-mounted display system of claim 1 , wherein the display controller is to render the foveated image frame on the display.

3. The head-mounted display system of claim 1 , wherein the first metadata is to specify at least one of the second resolution of the second image data or a down-sizing ratio corresponding to the second resolution, and the second metadata is to specify a position of the first image data in the foveated image frame.

4. The head-mounted display system of claim 1 , wherein the first resolution of the first image data is to correspond to a resolution of the display.

5. The head-mounted display system of claim 1 , wherein the third metadata is to specify the width of the rectangular strip in pixels.

6. The head-mounted display system of claim 1 , wherein the two different filter operations include at least two of a data smoothing, an averaging operation and a blurring operation.

7. The head-mounted display system of claim 1 , further including a data interface to communicatively couple the host device and the display controller, and the display controller is to obtain the first image data and the second image data from the host device via the data interface.

8. At least one non-transitory computer readable medium comprising computer readable instructions that, when executed, cause at least one or more processors to at least:
obtain first image data and second image data for a foveated image frame to be displayed on a display, the first image data to have a first resolution and the second image data to have a second resolution lower than the first resolution;
up-sample the second image data based on first metadata to generate up-sampled second image data, the up-sampled second image data to have the first resolution;
combine the first image data and the up-sampled second image data based on second metadata; and
perform, based on third metadata, a combination of at least two different filter operations on a filter overlap region to generate the foveated image frame, the filter overlap region for the at least two different filter operations to be specified in the third metadata, the filter overlap region to include a portion of the first image data and a portion of the up-sampled second image data, the filter overlap region corresponding to a rectangular strip with a width specified in the third metadata.

9. The at least one non-transitory computer readable medium of claim 8 , wherein the instructions cause the one or more processors to render the foveated image frame on the display.

10. The at least one non-transitory computer readable medium of claim 8 , wherein the first metadata is to specify at least one of the second resolution of the second image data or a down-sizing ratio corresponding to the second resolution, and the second metadata is to specify a position of the first image data in the foveated image frame.

11. The at least one non-transitory computer readable medium of claim 8 , wherein the first resolution of the first image data is to correspond to a resolution of the display.

12. The at least one non-transitory computer readable medium of claim 8 , wherein the third metadata is to specify the width of the rectangular strip data in pixels.

13. The at least one non-transitory computer readable medium of claim 8 , wherein the two different filter operations include at least two of a data smoothing, an averaging operation and a blurring operation.

14. The at least one non-transitory computer readable medium of claim 8 , wherein the instructions cause the one or more processors to obtain the first metadata and the second metadata from a metadata packet before the one or more processors obtain the first image data and the second image data.

15. A method comprising:
providing, with a first device, first image data and second image data for a foveated image frame to be displayed on a display, the first image data to have a first resolution and the second image data to have a second resolution lower than the first resolution;
up-sampling, with a second device, the second image data based on first metadata to generate up-sampled second image data, the up-sampled second image data to have the first resolution;
combining, with the second device, the first image data and the up-sampled second image data based on second metadata; and
performing, with the second device and based on third metadata, a combination of at least two different filter operations on a filter overlap region to generate the foveated image frame, the filter overlap region for the at least two different filter operations specified in the third metadata, the filter overlap region including a portion of the first image data and a portion of the up-sampled second image data, the filter overlap region corresponding to a rectangular strip with a width specified in the third metadata.

16. The method of claim 15 , further including rendering the foveated image frame on the display, wherein the first resolution of the first image data is to correspond to a resolution of the display.

17. The method of claim 15 , wherein the first metadata is to specify at least one of the second resolution of the second image data or a down-sizing ratio corresponding to the second resolution, and the second metadata is to specify a position of the first image data in the foveated image frame.

18. The method of claim 15 , wherein the third metadata is to specify the width of the rectangular strip in pixels.

19. The method of claim 15 , wherein the two different filter operations include at least two of a data smoothing, an averaging operation and a blurring operation.

20. The method of claim 15 , further including obtaining, at the second device, the first metadata, the second metadata and the third metadata from the first device.

Assignees

• Intel Corp

Inventors

• Steyskal Aaron
• Ansari Nausheen
• Suzuki Nobuyuki
• Zhuang Zhiming
• Kuwahara Atsuo
• Smith Gary K

Classifications

• G06T3/4038 Primary

  Image mosaicing, e.g. composing plane images from plane sub-images · CPC title

• G06T11/60

  Creating or editing images; Combining images with text · CPC title

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• H04L67/131 Primary

  Protocols for games, networked simulations or virtual reality · CPC title

• G06T1/20

  Processor architectures; Processor configuration, e.g. pipelining · CPC title