Hybrid projection-based point cloud texture coding

Patent metadata

Field	Value
Publication number	US-10853975-B2
Application number	US-201916246975-A
Country	US
Kind code	B2
Filing date	Jan 14, 2019
Priority date	Jan 26, 2018
Publication date	Dec 1, 2020
Grant date	Dec 1, 2020

Abstract

An apparatus receives a video which includes at least one three-dimensional (3D) object in a 3D physical space. A 3D geometrical representation of a point cloud is generated based on the video. The 3D geometrical representation of the point cloud includes a first set of points associated with geometrical information and texture information corresponding to the at least one 3D object. A plurality of two-dimensional (2D) projections are generated from the 3D geometrical representation of the point cloud. A second set of points that are occluded in the first set of points is detected, corresponding to the plurality of 2D projections. The plurality of 2D projections and the second set of points are distinctly encoded, and the remaining points, other than the detected second set of points, in the first set of points are discarded for efficient compression of the 3D geometrical representation of the point cloud.

First claim

What is claimed is:

1. An apparatus, comprising:
circuitry configured to:
receive a video that includes at least one three-dimensional (3D) object in a 3D physical space;
generate a 3D geometrical representation of a point cloud corresponding to the at least one 3D object based on the received video, wherein the 3D geometrical representation of the point cloud comprises a first set of points associated with geometrical information and texture information corresponding to the at least one 3D object;
generate a first plurality of two-dimensional (2D) projections from the 3D geometrical representation of the point cloud, wherein each 2D projection of the first plurality of 2D projections represents a projection of texture from a first viewing angle of a plurality of different viewing angles of the 3D geometrical representation of the point cloud of the at least one 3D object;
detect a second set of points that is occluded in the first set of points of the 3D geometrical representation of the point cloud, wherein the detected second set of points corresponds to the first plurality of 2D projections;
distinctly encode the first plurality of 2D projections and the detected second set of points; and
remove a third set of points, from the first set of points of the 3D geometrical representation of the point cloud, wherein the third set of points is different from the detected second set of points.

2. The apparatus according to claim 1 , wherein
the circuitry is further configured to transmit a first bit stream to at least one of a plurality of devices, and
the first bit stream includes at least the encoded first plurality of 2D projections and the encoded second set of points.

3. The apparatus according to claim 1 , wherein the circuitry is further configured to:
encode the first plurality of 2D projections based on one of a plurality of 2D image encoding techniques; and
encode the second set of points based on one of a plurality of 3D object encoding techniques.

4. The apparatus according to claim 1 , wherein the received video is captured from a plurality of viewpoints by a plurality of sensing devices.

5. The apparatus according to claim 1 , wherein
the encoded first plurality of 2D projections corresponds to a first bitstream,
the encoded second set of points corresponds to a second bitstream, and
the circuitry is further configured to:
merge the encoded first plurality of 2D projections and the encoded second set of points into a third bitstream; and
transmit the third bitstream to at least one of a plurality of devices.

6. The apparatus according to claim 1 , wherein
the at least one 3D object is a moving object that moves in the 3D physical space, and
the first plurality of 2D projections and the detected second set of points are associated with a first pose of the at least one 3D object that moves in the 3D physical space.

7. The apparatus according to claim 6 , wherein the circuitry is further configured to:
detect a change in the first pose to a second pose of the at least one 3D object;
generate a second plurality of 2D projections for the second pose of the at least one 3D object;
update the second set of points that is occluded in the first set of points of the 3D geometrical representation of the point cloud, wherein the updated second set of points corresponds to the second plurality of 2D projections; and
distinctly encode the second plurality of 2D projections and the updated second set of points, wherein the second plurality of 2D projections and the updated second set of points are associated with the second pose of the at least one 3D object.

8. The apparatus according to claim 1 , wherein
the apparatus is communicatively coupled to a plurality of devices, and
the circuitry is further configured to:
receive, from the plurality of devices, device information related to rendering capability of the plurality of devices, wherein the device information indicates an ability of the plurality of devices to reconstruct encoded information for display; and
determine a transmission mode from a plurality of transmission modes based on the received device information.

9. The apparatus according to claim 8 , wherein
the circuitry is further configured to separately transmit, based on the received device information, a first bitstream and a second bitstream to a first device from the plurality of devices in a first transmission mode of the plurality of transmission modes,
the first bitstream includes the encoded first plurality of 2D projections, and
the second bitstream includes the encoded second set of points.

10. The apparatus according to claim 8 , wherein
the circuitry is further configured to transmit, based on the received device information, a bitstream to a second device from the plurality of devices in a second transmission mode of the plurality of transmission modes, and
the bitstream includes a combination of the encoded first plurality of 2D projections and the encoded second set of points.

11. The apparatus according to claim 1 , wherein the circuitry is further configured to:
determine texture information associated with the first plurality of 2D projections based on the first set of points;
encode the texture information associated with the first plurality of 2D projections based on one of a plurality of 2D image encoding techniques;
determine the texture information associated with the second set of points based on the first set of points; and
encode the texture information associated with only the second set of points based on one of a plurality of 3D object encoding techniques.

12. The apparatus according to claim 1 , wherein
the circuitry is further configured to project the texture of the 3D geometrical representation of the point cloud on a specified 2D plane for the generation of the first plurality of 2D projections, and
the first plurality of 2D projections corresponds to 2D images that are generated based on the projection of the texture of the 3D geometrical representation of the point cloud on the specified 2D plane.

13. The apparatus according to claim 12 , wherein the projection of the texture of the 3D geometrical representation of the point cloud is a cube map projection of the 3D geometrical representation of the point cloud.

14. The apparatus according to claim 12 , wherein
the projection of the texture of the 3D geometrical representation of the point cloud corresponds to orthogonal projections of the 3D geometrical representation of the point cloud,
the circuitry is further configured to detect the second set of points that is occluded in the first set of points of the 3D geometrical representation of the point cloud, and
the detected second set of points corresponds to the orthogonal projections.

15. The apparatus according to claim 12 , wherein
the projection of the texture of the 3D geometrical representation of the point cloud corresponds to cylindrical projections of the 3D geometrical representation of the point cloud,
the circuitry is further configured to detect the second set of points that is occluded in the first set of points of the 3D geometrical representation of the point cloud, and
the detected second set of points corresponds to the cylindrical projections.

16. A method, comprising:
receiving, by circuitry in an apparatus, a video that includes at least one three-dimensional (3D) object in a 3D physical space;
generating, by the circuitry, a 3D geometrical representation of a point cloud corresponding to the at least one 3D object based on the received video, wherein the 3D

Assignees

• Sony Corp

Inventors

• Vosoughi Arash
• Graziosi Danillo

Classifications

• H04L67/10 Primary

  in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46) · CPC title

• G06T9/40 Primary

  Tree coding, e.g. quadtree, octree · CPC title

• H04L67/12

  specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks · CPC title

• H04L67/04

  specially adapted for terminals or networks with limited capabilities; specially adapted for terminal portability · CPC title

• G06T15/04

  Texture mapping · CPC title