Foveated image rendering for head-mounted display devices

Patent metadata

Field	Value
Publication number	US-10949947-B2
Application number	US-201715858678-A
Country	US
Kind code	B2
Filing date	Dec 29, 2017
Priority date	Dec 29, 2017
Publication date	Mar 16, 2021
Grant date	Mar 16, 2021

Abstract

Example methods, apparatus, systems and articles of manufacture (e.g., non-transitory physical storage media) to implement foveated image rendering for head-mounted displays device are disclosed herein. Example head-mounted display devices disclosed herein include a frame buffer to store first and second image data for an image frame, the first image data having a first resolution and the second image data having a second resolution lower than the first resolution, the first image data and the second image data obtained from a host device via a data interface. Disclosed example head-mounted display devices also include a device controller to up-sample the second image data based on first metadata from the host device to generate up-sampled second image data having the first resolution, and combine the first image data and the up-sampled second image data based on second metadata from the host device to render a foveated image frame on a display.

First claim

What is claimed is:

1. A head-mounted display device comprising:
a display;
a data interface;
a frame buffer; and
a device controller to:
store first image data for a foveated image frame from a host device in the frame buffer before second image data for the foveated image frame is obtained from the host device, the first image data to have a first resolution and the second image data to have a second resolution lower than the first resolution;
as the second image data is obtained from the host device via the data interface, up-sample the second image data based on first metadata from the host device to generate up-sampled second image data to combine with the first image data, the up-sampled second image data to have the first resolution;
combine the first image data from the frame buffer and the up-sampled second image data based on second metadata from the host device to form the foveated image frame; and
apply, based on third metadata from the host device, a combination of at least two different filter operations on a rectangular strip of the foveated image frame that corresponds to an overlap region including a portion of the first image data and a portion of the up-sampled second image data to render the foveated image frame on the display, the third metadata to specify a width in pixels of the rectangular strip.

2. The head-mounted display device of claim 1 , wherein the first metadata and the second metadata are included in a metadata packet from the host device, the device controller to obtain the first and second metadata before the device controller is to obtain the first image data and the second image data.

3. The head-mounted display device of claim 1 , wherein the first metadata, the second metadata and the second image data are included in a data packet from the host device.

4. The head-mounted display device of claim 1 , wherein the first metadata is to specify at least one of the second resolution of the second image data or a down-sizing ratio corresponding to the second resolution, and the second metadata is to specify a position of the first image data in the foveated image frame.

5. The head-mounted display device of claim 4 , wherein the device controller is further to:
up-sample third image data with a third resolution from the host device to generate up-sampled third image data with the first resolution, the third resolution lower than the first resolution and the second resolution, the device controller to up-sample the third image data based on fourth metadata from the host device, the fourth metadata to specify at least one of the third resolution of the third image data or a down-sizing ratio corresponding to the third resolution; and
combine the first image data, the up-sampled second image data and the up-sampled third image data based on the second metadata and fifth metadata from the host device to render the foveated image frame on the display, the fifth metadata to specify a position of the second image data in the foveated image frame.

6. The head-mounted display device of claim 5 , wherein the first image data is to correspond to a first region of the foveated image frame, the second image data is to correspond to a second region of the foveated image frame, the third image data is to correspond to a third region of the foveated image frame, the second region is included in the third region, and the first region is included in the second region.

7. The head-mounted display device of claim 1 , wherein the first resolution of the first image data is to correspond to a resolution of the display.

8. The head-mounted display device of claim 1 , wherein the device controller is to obtain the first image data and the second image data via respective data transmission pipelines of a plurality of data transmission pipelines implemented by the device controller and a corresponding host controller of the host device, a first one of the data transmission pipelines associated with a first region of a first screen of the display, a second one of the data transmission pipelines associated with a first region of a second screen of the display, a third one of the data transmission pipelines associated with a second region of the first screen of the display, and a fourth one of the data transmission pipelines associated with a second region of the second screen of the display.

9. One or more non-transitory computer readable storage media comprising computer readable instructions which, when executed, cause one or more processors of a head-mounted display device to at least:
store first image data with a first resolution from a host device at the head-mounted display device;
obtain second image data from the host device via a data interface after the first image data is stored at the head-mounted display device, the second image data to have a second resolution lower than the first resolution;
as the second image data is obtained from the host device via the data interface, up-sample the second image data based on first metadata to generate up-sampled second image data to combine with the first image data, the up-sampled second image data to have the first resolution;
combine the stored first image data and the up-sampled second image data based on second metadata from the host device to form a foveated image frame; and
apply, based on third metadata from the host device, a combination of at least two different filter operations on a rectangular strip of the foveated image frame that corresponds to an overlap region including a portion of the first image data and a portion of the up-sampled second image data to render the foveated image frame on the display of the head-mounted display device, the third metadata to specify a width in pixels of the rectangular strip.

10. The one or more storage media of claim 9 , wherein the first metadata and the second metadata are included in a metadata packet from the host device, and the instructions, when executed, cause the one or more processors to obtain the metadata packet before the first image data and the second image data.

11. The one or more storage media of claim 9 , wherein the first metadata, the second metadata and the second image data are included in a data packet from the host device.

12. The one or more storage media of claim 9 , wherein the first metadata is to specify at least one of the second resolution of the second image data or a down-sizing ratio corresponding to the second resolution, and the second metadata is to specify a position of the first image data in the foveated image frame.

13. The one or more storage media of claim 9 , wherein the instructions, when executed, cause the one or more processors to obtain the first image data and the second image data via respective data transmission pipelines of a plurality of data transmission pipelines implemented between the head-mounted display device and the host device, a first one of the data transmission pipelines associated with a first region of a first screen of a display of the head-mounted display device, a second one of the data transmission pipelines associated with a first region of a second screen of the display, a third one of the data transmission pipelines associated with a second region of the first screen of the display, and a fourth one of the data transmission pipelines associated with a second region of the second screen of the display.

14. An apparatus to implement foveated image rendering, the apparatus comprising:
means for storing first image data for a foveated image frame from a host device, the first image data to have a first resolution, the means for storing to store the first

Assignees

• Intel Corp

Inventors

• Steyskal Aaron
• Ansari Nausheen
• Suzuki Nobuyuki
• Zhuang Zhiming
• Kuwahara Atsuo
• Smith Gary K

Classifications

• H04L67/131 Primary

  Protocols for games, networked simulations or virtual reality · CPC title

• G06T1/20

  Processor architectures; Processor configuration, e.g. pipelining · CPC title

• G02B2027/0147

  comprising a device modifying the resolution of the displayed image · CPC title

• G06T1/60

  Memory management · CPC title

• G06T3/4038 Primary

  Image mosaicing, e.g. composing plane images from plane sub-images · CPC title