MV/mode prediction, ROI-based transmit, metadata capture, and format detection for 360 video

Patent metadata

Field	Value
Publication number	US-10506255-B2
Application number	US-201715476989-A
Country	US
Kind code	B2
Filing date	Apr 1, 2017
Priority date	Apr 1, 2017
Publication date	Dec 10, 2019
Grant date	Dec 10, 2019

Abstract

An embodiment of an electronic processing system may include a 2D frame which corresponds to a projection of a 360 video space, and a component predictor to predict an encode component for a first block of a 2D frame based on encode information from a neighboring block which is neighboring to the first block of the 2D frame only in the 360 video space, a prioritizer to prioritize transmission for a second block of the 2D frame based on an identified region of interest, and/or a format detector to detect a 360 video format of the 2D frame based on image content. A 360 video capture device may include a contextual tagger to tag 360 video content with contextual information which is contemporaneous with the captured 360 video content. Other embodiments are disclosed and claimed.

First claim

What is claimed is:

1. An electronic processing system, comprising:
a display processor to generate image data for a display;
a memory communicatively coupled to the display processor to store a two-dimensional (2D) frame which corresponds to a projection from a 360 video space;
a component predictor communicatively coupled to the display processor to predict an encode component for a first block of the 2D frame based on encode information from one or more neighboring blocks of the 2D frame, wherein the one or more neighboring blocks of the 2D frame are contiguous to the first block of the 2D frame only when displayed in the 360 video space;
a prioritizer communicatively coupled to the display processor to prioritize transmission for a packet of the 2D frame based on an identified region of interest; and
a format detector communicatively coupled to the display processor to detect a 360 video format of the 2D frame based on an image content of the 2D frame,
wherein the first block of the 2D frame and the one or more neighboring blocks of the 2D frame are approximately equidistant from a same point in the 2D frame, the same point being a corner of two faces of the 2D frame.

2. The system of claim 1 , further comprising:
a 360 video capture device communicatively coupled to the display processor to capture 360 video content; and
a contextual tagger communicatively coupled to the 360 video capture device to tag the 360 video content with contextual information which is contemporaneous with the captured 360 video content.

3. The system of claim 2 , wherein the contextual information includes one or more of motion information, location information, velocity information, acceleration information, orientation information, and direction information.

4. The system of claim 1 , wherein the encode information includes one or more of motion vector information and mode information.

5. The system of claim 1 , wherein the prioritizer is further to prioritize encode for a second block of the 2D frame based on the identified region of interest.

6. The system of claim 1 , wherein the detected 360 video format includes one or more of a fish-eye format, an equirectangular projection format, a cube-map format, and a packed cube-map format.

7. A graphics apparatus, comprising:
an encoder to encode a first block of a two-dimensional (2D) frame, where the 2D frame corresponds to a projection of a 360 video space; and
a component predictor communicatively coupled to the encoder to determine if the first block is a neighbor of a second block of the 2D frame in the 360 video space, and to predict an encode component for the second block based on encode information for the first block if the first block is determined to be contiguous to the second block, the first block being contiguous to the second block when displayed in the 360 video space,
wherein the first block of the 2D frame and the second block of the 2D frame are approximately equidistant from a same point in the 2D frame, the same point being a corner of two faces of the 2D frame.

8. The apparatus of claim 7 , wherein the component predictor is further to predict a motion vector for the second block based on encoded motion vector information for the first block if the first block is determined to be the neighbor of the second block in the 360 video space.

9. The apparatus of claim 7 , wherein the component predictor is further to predict a mode for the second block based on encoded mode information for the first block if the first block is determined to be the neighbor of the second block in the 360 video space.

10. The apparatus of claim 7 , wherein the encoder is further to encode the second block of the 2D frame based on the predicted encode component.

11. The apparatus of claim 7 , further comprising:
a prioritizer communicatively coupled to the encoder to prioritize transmission for a packet of the 2D frame based on an identified region of interest.

12. The apparatus of claim 7 , further comprising:
a format detector communicatively coupled to the encoder to detect a 360 video format of the 2D frame based on an image content of the 2D frame.

13. The apparatus of claim 7 , further comprising:
a 360 video capture device communicatively coupled to the encoder to capture 360 video content; and
a contextual tagger communicatively coupled to the 360 video capture device to tag the 360 video content with contextual information which is contemporaneous with the captured 360 video content.

14. A method of processing a 360 video, comprising:
encoding a first block of a two-dimensional (2D) frame, where the 2D frame corresponds to a projection of a 360 video space;
determining if the first block is a neighbor of a second block of the 2D frame in the 360 video space; and
predicting an encode component for the second block based on encode information for the first block if the first block is determined to be contiguous to the second block, the first block being contiguous to the second block when displayed in the 360 video space,
wherein the first block of the 2D frame and the second block of the 2D frame are approximately equidistant from a same point in the 2D frame, the same point being a corner of two faces of the 2D frame.

15. The method of claim 14 , further comprising:
predicting a motion vector for the second block based on encoded motion vector information for the first block if the first block is determined to be the neighbor of the second block in the 360 video space.

16. The method of claim 14 , further comprising:
predicting a mode for the second block based on encoded mode information for the first block if the first block is determined to be the neighbor of the second block in the 360 video space.

17. The method of claim 14 , further comprising:
encoding the second block of the 2D frame based on the predicted encode component.

18. At least one computer readable medium, comprising a set of instructions, which when executed by a computing device cause the computing device to:
encode a first block of a two-dimensional (2D) frame, where the 2D frame corresponds to a projection of a 360 video space;
determine if the first block is a neighbor of a second block of the 2D frame in the 360 video space; and
predict an encode component for the second block based on encode information for the first block if the first block is determined to be contiguous to the second block, the first block being contiguous to the second block when displayed in the 360 video space,
wherein the first block of the 2D frame and the second block of the 2D frame are approximately equidistant from a same point in the 2D frame, the same point being a corner of two faces of the 2D frame.

19. The at least one computer readable medium of claim 18 , comprising a further set of instructions, which when executed by a computing device cause the computing device to:
predict a motion vector for the second block based on encoded motion vector information for the first block if the first block is determined to be the neighbor of the second block in the 360 video space.

20. The at least one computer readable medium of claim 18 , comprising a further set of instructions, which when executed by a computing device cause the computing device to:
predict a mode for the second block based on encoded mode information for the first block if the first block is determined to be the neighbor of the second block in the 360 video space.

21. The at least one computer readable medi

Assignees

• Intel Corp

Inventors

• Boyce Jill M
• Lee Sang-Hee
• Appu Abhishek R
• Kao Wen-Fu
• Ray Joydeep
• Peng Ya-Ti
• Rowe Keith W
• Fu Fangwen
• Yedidi Satya N
• Mohan Sumit
• Holland James M

Classifications

• G06T12/00

  Tomographic reconstruction from projections · CPC title

• H04N19/597

  specially adapted for multi-view video sequence encoding · CPC title

• H04N19/176

  the region being a block, e.g. a macroblock · CPC title

• H04N19/52

  by predictive encoding · CPC title

• H04N21/234

  Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs · CPC title