Selective packing of patches for immersive video

Patent metadata

Field	Value
Publication number	US-12219115-B2
Application number	US-202318505356-A
Country	US
Kind code	B2
Filing date	Nov 9, 2023
Priority date	Jul 31, 2018
Publication date	Feb 4, 2025
Grant date	Feb 4, 2025

Abstract

Embodiments are generally directed to selective packing of patches for immersive video. An embodiment of a processing system includes one or more processor cores; and a memory to store data for immersive video, the data including a plurality of patches for multiple projection directions. The system is select the patches for packing, the selection of the patches based at least in part on which of the multiple projection directions is associated with each of the patches. The system is to encode the patches into one or more coded pictures according to the selection of the patches.

First claim

What is claimed is:

1. A apparatus comprising:
one or more processor cores; and
a memory to store data for immersive video;
wherein the one or more processor cores are to:
generate a plurality of patches, each patch representing a portion of video content that is visible from an associated projection direction of a plurality of projection directions for the immersive video;
encode and pack the plurality of patches into a plurality of tiles, each of the plurality of tiles including one or more patches that are visible from an associated projection direction;
identify an immersive video image to be generated, the immersive video image having an associated projection direction and location;
select a set of tiles of the plurality of tiles from the memory for the immersive video image, the set of tiles including tiles for multiple projection directions to be used in generation of the immersive video image;
decode and unpack the patches in the selected set of tiles; and
generate the identified immersive video image, generation of the immersive video image including use of one or more patches from the selected set of tiles.

2. The apparatus of claim 1 , wherein the plurality of patches are generated during a same frame time.

3. The apparatus of claim 1 , wherein the one or more patches encoded and packed in a tile associated with a first projection direction are limited to portions of video content that are occluded from one or more other projection directions of the plurality of projection directions.

4. The apparatus of claim 1 , wherein the one or more processor cores are to decode and unpack the patches from the selected set of tiles without decoding and unpacking one or more other tiles of the plurality of tiles.

5. The apparatus of claim 1 , wherein the one or more processor cores are to select the set of tiles based at least in part on a current or anticipated viewpoint for the immersive video image.

6. The apparatus of claim 1 , wherein the plurality of tiles are included within a coded picture.

7. The apparatus of claim 1 , wherein the plurality of patches includes at least one of texture data and depth data.

8. The apparatus of claim 1 , wherein the plurality of tiles includes motion-constrained tiles in an HEVC (High Efficiency Video Coding) motion-constrained tile set (MCTS).

9. The apparatus of claim 1 , wherein the immersive video includes 6DoF (6 Degrees of Freedom) video.

10. The apparatus of claim 1 , wherein the one or more processor cores includes one or more graphics processor cores of a graphical processing unit (GPU).

11. A non-transitory computer-readable storage medium having stored thereon data representing sequences of instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
generating a plurality of patches, each patch representing a portion of video content that is visible from an associated projection direction of a plurality of projection directions for immersive video;
encoding and packing the plurality of patches into a plurality of tiles, each of the plurality of tiles including one or more patches that are visible from an associated projection direction;
identifying an immersive video image to be generated, the immersive video image having an associated projection direction and location;
selecting a set of tiles of the plurality of tiles for the immersive video image, the set of tiles including tiles for multiple projection directions to be used in generation of the immersive video image;
decoding and unpacking the patches in the selected set of tiles; and
generating the identified immersive video image, generation of the immersive video image including use of one or more patches from the selected set of tiles.

12. The storage medium of claim 11 , wherein the plurality of patches are generated during a same frame time.

13. The storage medium of claim 11 , wherein the one or more patches encoded and packed in a tile associated with a first projection direction are limited to portions of video content that are occluded from one or more other projection directions of the plurality of projection directions.

14. The storage medium of claim 11 , wherein decoding and unpacking the patches from the selected set of tiles includes decoding and unpacking the patches from the selected set of tiles without decoding and unpacking one or more other tiles of the plurality of tiles.

15. The storage medium of claim 11 , wherein selecting the set of tiles includes selecting the set of tiles based at least in part on a current or anticipated viewpoint for the immersive video image.

16. A system comprising:
one or more processors including a graphics processor; and
a memory to store data for immersive video;
wherein the graphics processor is to:
generate a plurality of patches, each patch representing a portion of video content that is visible from an associated projection direction of a plurality of projection directions for the immersive video;
encode and pack the plurality of patches into a plurality of tiles, each of the plurality of tiles including one or more patches that are visible from an associated projection direction;
identify an immersive video image to be generated, the immersive video image having an associated projection direction and location;
select a set of tiles of the plurality of tiles from the memory for the immersive video image, the set of tiles including tiles for multiple projection directions to be used in generation of the immersive video image;
decode and unpack the patches in the selected set of tiles; and
generate the identified immersive video image, generation of the immersive video image including use of one or more patches from the selected set of tiles.

17. The system of claim 16 , wherein the plurality of patches are generated during a same frame time.

18. The system of claim 16 , wherein the one or more patches encoded and packed in a tile associated with a first projection direction are limited to portions of video content that are occluded from one or more other projection directions of the plurality of projection directions.

19. The system of claim 16 , wherein the graphics processor is to decode and unpack the patches from the selected set of tiles without decoding and unpacking one or more other tiles of the plurality of tiles.

20. The system of claim 16 , wherein the graphics processor is to select the set of tiles based at least in part on a current or anticipated viewpoint for the immersive video image.

Assignees

• Intel Corp

Inventors

• Ruhm Eyal
• Boyce Jill
• Shenberg Asaf J

Classifications

• H04N13/167

  Synchronising or controlling image signals · CPC title

• H04N19/176

  the region being a block, e.g. a macroblock · CPC title

• H04N19/42

  characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation (H04N19/635 takes precedence) · CPC title

• H04N13/161 Primary

  Encoding, multiplexing or demultiplexing different image signal components (for multi-view video sequence encoding H04N19/597) · CPC title

• H04N13/275 Primary

  from three-dimensional [3D] object models, e.g. computer-generated stereoscopic image signals · CPC title