Methods and apparatus for signaling viewports and regions of interest for point cloud multimedia data

Patent metadata

Field	Value
Publication number	US-11200700-B2
Application number	US-202016738387-A
Country	US
Kind code	B2
Filing date	Jan 9, 2020
Priority date	Jan 10, 2019
Publication date	Dec 14, 2021
Grant date	Dec 14, 2021

Abstract

The techniques described herein relate to methods, apparatus, and computer readable media configured to encode and/or decode video data. Point cloud video data is received that includes metadata specifying one or more regions of interest of the point cloud video data. A first region of interest is determined from the one or more regions of interest. A portion of the point cloud video data associated with the first region of interest is determined. Point cloud media is generated for viewing by a user based on the determined portion of the point cloud video data associated with the first region of interest.

First claim

What is claimed is:

1. A decoding method for decoding video data, the method comprising:
receiving point cloud video data comprising:
a plurality of media samples of the point cloud video data; and
a sample entry structure comprising metadata specifying one or more regions of interest of the point cloud video data, wherein the sample entry structure includes one or more of:
dynamic location information indicative of whether a location of the one or more regions change in at least one of the plurality of media samples of the point cloud video data that refer to the sample entry structure; and
dynamic size information indicative of whether a size of the one or more regions changes in at least one of the plurality of media samples that refer to the sample entry structure;

determining a first region of interest from the one or more regions of interest;
determining a portion of the point cloud video data associated with the first region of interest, comprising determining at least one of the plurality of media samples of the point cloud video data references the sample entry structure; and
generating, based on the determined portion of the point cloud video data associated with the first region of interest, point cloud media for viewing by a user.

2. The method of claim 1 , wherein:
receiving the point cloud video data comprises receiving a set of two-dimensional (2D) planar video bit streams and second metadata specifying a conversion of the set of 2D planar video bit streams to 3D volumetric media; and
determining the portion of the point cloud video data associated with the first region of interest comprises determining a subset of data of the set of 2D planar video bit streams associated with the first region of interest.

3. The method of claim 1 , further comprising:
receiving user interaction data associated with the point cloud video data, wherein the user interaction data comprises data indicative of the user's location, the user's viewpoint, or some combination thereof;
determining a second region of interest from the one or more regions of interest, wherein the second region of interest is different than the first region of interest;
determining a second portion of the point cloud video data associated with the second region of interest, wherein the second portion is different than the portion associated with the first region of interest; and
generating, based on the determined second portion of the point cloud video data associated with the second region of interest, second point cloud media for viewing by the user.

4. The method of claim 1 , wherein:
determining the first region of interest comprises determining a first identifier for a geometry track of the point cloud video data, a second identifier for a texture track of the point cloud video data, or both, wherein the geometry track specifies one or more geometric aspects of the point cloud video data, and the texture track specifies one or more texture aspects of the point cloud video data; and
determining the portion of the point cloud video data associated with the first region of interest comprises selecting the geometry track identified by the first identifier, selecting the texture track identified by the second identifier, or both.

5. The method of claim 1 ,
wherein the sample entry structure is a region structure associated with the point cloud video data that specifies one or more aspects of the first region of interest based on a sphere, wherein the one or more aspects comprise a location of the first region of interest, an orientation of the first region of interest, a shape of the first region of interest, a size range of the first region of interest, or some combination thereof, the region structure comprising:
data indicative of whether the one or more aspects of the region structure comprises data indicative of one or more global aspects of the first region of interest that applies to each of a set of samples associated with the first region structure, including the dynamic location information for the location of the first region of interest, the dynamic size information for the size range of the first region of interest, or both; and

determining the first region of interest in the point cloud video data based on the data indicative of whether the region structure comprises data indicative of the one or more global aspects of the first region of interest.

6. The method of claim 5 , wherein determining the first region of interest in the point cloud video data based on the data indicative of whether the region structure comprises data indicative of one or more global aspects of the first region of interest comprises:
determining that the region structure does not specify at least one global aspect of the one or more aspects; and
determining, for each sample associated with the region structure, a sample-specific aspect for the sample for the first region of interest.

7. The method of claim 5 , wherein determining the first region of interest in the point cloud video data comprises determining the location of the first region of interest by determining a center location of the first region of interest.

8. The method of claim 5 , wherein determining the first region of interest in the point cloud video data comprises determining the orientation of the first region of interest by determining a center azimuth, a center elevation, and a center tilt of the first region of interest.

9. The method of claim 5 , wherein determining the first region of interest in the point cloud video data comprises determining the size range of the first region of interest by determining an azimuth range and an elevation range of the first region of interest.

10. A method for encoding video data, the method comprising:
encoding point cloud video data comprising:
a plurality of media samples of the point cloud video data; and
a sample entry structure comprising metadata specifying one or more regions of interest of the point cloud video data, wherein the sample entry structure includes one or more of:
dynamic location information indicative of whether a location of the one or more regions change in samples of the plurality of media samples of the point cloud video data that refer to the sample entry structure; and
dynamic size information indicative of whether a size of the one or more regions changes in the samples that refer to the sample entry structure,

the encoding comprising:
determining a first region of interest of the one or more regions of interest;
determining a portion of the point cloud video data associated with the first region of interest, comprising determining at least one of the plurality of media samples of the point cloud video data references the sample entry structure; and
encoding, based on the determined portion of the point cloud video data associated with the first region of interest, the point cloud video data comprising encoding the sample entry structure comprising metadata specifying the one or more regions of interest of the point cloud video data.

11. An apparatus configured to decode video data, the apparatus comprising a processor in communication with memory, the processor being configured to execute instructions stored in the memory that cause the processor to perform:
receiving point cloud video data comprising:
a plurality of media samples of the point cloud video data; and
a sample entry structure comprising metadata specifying one or more regions of interest of the point cloud video data, wherein the sample entry structure includes one or more of:
dynamic location information indicative of whether a location of the one or more regions change in samples of the plura

Assignees

• Mediatek Singapore Pte Ltd

Inventors

• Wang Xin
• Chen Lulin

Classifications

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T3/40

  Scaling of whole images or parts thereof, e.g. expanding or contracting · CPC title

• H04N19/597

  specially adapted for multi-view video sequence encoding · CPC title

• G06T7/0002

  Inspection of images, e.g. flaw detection · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title