Object pre-encoding for 360-degree view for optimal quality and latency

Patent metadata

Field	Value
Publication number	US-11800232-B2
Application number	US-202117158627-A
Country	US
Kind code	B2
Filing date	Jan 26, 2021
Priority date	Apr 24, 2017
Publication date	Oct 24, 2023
Grant date	Oct 24, 2023

Abstract

Systems, apparatuses and methods may provide for technology to improve an appearance of objects that enter a viewable area of a 360-degree video. The technology may include a head mounted display (HMD), a viewport comprising a viewable area within the HMD, and a memory to store objects of interest not currently in the viewable area.

First claim

What is claimed is:

1. A system, comprising:
a display having a viewable area defining a viewport;
a client communicatively coupled to the display to deliver a portion of a 360-degree video of a scene to substantially fit within the viewport; and
a sideband memory to store in a pre-rendered form objects of interest separate from the scene, wherein one of the objects of interest is a first object that is in motion and is not currently in the viewport,
wherein the objects of interest are pre-rendered separate from any video containing the scene and prior to a determination that the one of the objects of interest has entered the scene,
wherein the display comprises a head mounted display (HMD) communicatively coupled to a motion engine to translate a user's head movement or eye movement to pan around the 360-degree video,
wherein the client is to display in a current frame the first object upon a determination, based on motion information about the first object, that the first object is suddenly entering the viewport, and
wherein the client is to display in a subsequent frame a second object of interest upon a determination, based on the user's head movement or eye movement, that the second object is suddenly entering the viewport.

2. The system as recited in claim 1 , wherein the client is further to display in a current frame a third object of interest upon a determination that the third object of interest is entering the current frame.

3. The system as recited in claim 1 , wherein the objects of interest are identified in metadata of the 360-degree video, and wherein the metadata relates to one or more of an area of likely focus, crowdsource data, or audio data.

4. The system as recited in claim 3 , wherein at least one of the objects of interest is identified in the metadata of the 360-degree video as likely to enter the viewport at a subsequent time.

5. A graphics apparatus comprising logic implemented in one or more of configurable logic or fixed-functionality hardware logic, the logic to:
deliver a portion of a 360-degree video of a scene to a display of a client, the portion comprising a viewable area;
identify objects of interest in the 360-degree video currently outside of the viewable area wherein one of the objects of interest is a first object that is in motion;
pre-render the objects of interest separate from any video containing the scene and prior to a determination that one of the objects of interest has entered the scene;
monitor bandwidth being used to deliver the portion of the 360-degree video; and
if bandwidth is available, deliver at least a partial one of the pre-rendered objects of interest to a memory at the display,
wherein the display comprises a head mounted display (HMD) communicatively coupled to a motion engine to translate a user's head movement or eye movement to pan around the 360-degree video,
wherein the first object is to be displayed in a current frame upon a determination, based on motion information about the first object, that the first object is suddenly entering the viewable area, and
wherein a second object of interest is to be displayed in a subsequent frame upon a determination, based on the user's head movement or eye movement, that the second object is suddenly entering the viewable area.

6. The apparatus of claim 5 , wherein the logic is further to:
determine if the at least partial one of the pre-rendered objects of interest is entering the viewable area; and
reference the memory to display the object of interest in the scene.

7. The apparatus of claim 5 , wherein the objects of interest are identified in metadata of the 360-degree video, and wherein the metadata relates to one or more of an area of likely focus, crowdsource data, or audio data.

8. The apparatus of claim 7 , wherein at least one of the objects of interest is identified in the metadata of the 360-degree video as likely to enter the viewport at a subsequent time.

9. The apparatus of claim 7 , wherein the crowdsource data relates to where other viewers have panned at a point in the 360-degree video.

10. At least one non-transitory computer readable storage medium comprising a set of instructions which, when executed by a processor, cause a graphics system to:
deliver a portion of a 360-degree video of a scene to a display, the portion comprising a viewable area;
identify objects of interest in the 360-degree video currently outside of the viewable area wherein one of the objects of interest is a first object that is in motion;
pre-render the objects of interest separate from any video containing the scene and prior to a determination that one of the objects of interest has entered the scene;
monitor bandwidth being used to deliver the portion of the 360-degree video; and
if bandwidth is available, deliver at least a partial one of the pre-rendered objects of interest to a memory at the display,
wherein the display comprises a head mounted display (HMD) communicatively coupled to a motion engine to translate a user's head movement or eye movement to pan around the 360-degree video,
wherein the first object is to be displayed in a current frame upon a determination, based on motion information about the first object, that the first object is suddenly entering the viewable area, and
wherein a second object of interest is to be displayed in a subsequent frame upon a determination, based on the user's head movement or eye movement, that the second object is suddenly entering the viewable area.

11. The at least one non-transitory computer readable storage medium of claim 10 , wherein the instructions, when executed, cause the graphics system to:
determine if the at least partial one of the pre-rendered objects of interest is entering the viewable area; and
reference the memory to display the object of interest in the scene.

12. The at least one non-transitory computer readable storage medium of claim 10 , wherein the objects of interest are identified in metadata of the 360-degree video, and wherein the metadata relates to one or more of an area of likely focus, crowdsource data, or audio data.

13. The at least one non-transitory computer readable storage medium of claim 12 , wherein at least one of the objects of interest is identified in the metadata of the 360-degree video as likely to enter the viewport at a subsequent time.

14. The at least one non-transitory computer readable storage medium of claim 12 , wherein the crowdsource data relates to where other viewers have panned at a point in the 360-degree video.

15. A method, comprising:
delivering a portion of a 360-degree video of a scene to a display, the portion comprising a viewable area;
identifying objects of interest in the 360-degree video currently outside of the viewable area, wherein one of the objects of interest is a first object that is in motion;
pre-rendering the objects of interest separate from any video containing the scene and prior to a determination that one of the objects of interest has entered the scene;
monitoring bandwidth being used to deliver the portion of the 360-degree video; and
if bandwidth is available, delivering on a sideband at least a partial one of the pre-rendered objects of interest to a memory at the display,
wherein the display comprises a head mounted display (HMD) communicatively coupled to a motion engine to translate a user's head movement or eye movement to pan around the 360-degree video,
wherein the first object is to be displayed in a current frame upon a determination, based on motion information about the first object, that the first object is suddenly entering the

Assignees

• Intel Corp

Inventors

• Tanner Jason
• Kao Wen-Fu
• Liu Ping
• Chiu Yi-Jen
• Peng Ya-Ti

Classifications

• H04N23/698 Primary

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• G02B27/017

  Head mounted · CPC title

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06F3/147

  using display panels · CPC title

• G06T11/00

  Two-dimensional [2D] image generation · CPC title