Image processing apparatus and method

Patent metadata

Field	Value
Publication number	US-11405644-B2
Application number	US-201917257865-A
Country	US
Kind code	B2
Filing date	Jul 19, 2019
Priority date	Aug 2, 2018
Publication date	Aug 2, 2022
Grant date	Aug 2, 2022

Abstract

An image processing apparatus and a method comprises generating a video frame that includes a patch obtained by projecting, onto a two dimensional plane, a point cloud that represents an object having a three-dimensional shape as a group of points; generating a thumbnail two-dimensional image, the thumbnail two-dimensional image being generated independently from the patch; embedding the thumbnail two-dimensional image into the video frame; and encoding the video frame to generate a bitstream.

First claim

The invention claimed is:

1. An image processing apparatus comprising:
circuitry configured to
generate a video frame that includes a patch obtained by projecting, onto a two dimensional plane, a point cloud that represents an object having a three-dimensional shape as a group of points;
generate a thumbnail two-dimensional image, the thumbnail two-dimensional image being generated independently from the patch;
embed the thumbnail two-dimensional image into the video frame, the thumbnail two-dimensional image being arranged along the patch with an offset; and
encode the video frame, as coded data of the video frame including the thumbnail two-dimensional image, to generate a bitstream, in which the coded data of the video frame is decoded to reconstruct the point cloud at a decoding device and the thumbnail two-dimensional image is extracted from the bitstream and decoded to display at the decoding device independently from reconstruction of the point cloud.

2. The image processing apparatus according to claim 1 , wherein
the thumbnail two-dimensional image is a rendered image obtained by rendering the object.

3. The image processing apparatus according to claim 2 , wherein
the rendered image is an image obtained by rendering just like imaging the object from a recommended camera position and direction.

4. The image processing apparatus according to claim 3 , wherein
the circuitry is configured to generate a moving image constituted by the video frame including a plurality of the rendered images, which are moving images, and
the circuitry is configured to encode the moving image to generate the bitstream.

5. The image processing apparatus according to claim 4 , wherein
the plurality of the rendered images, which are moving images, are rendered images obtained by rendering the object with the same camera work as each other.

6. The image processing apparatus according to claim 1 , wherein
the circuitry is configured to generate a color video

frame that includes the patch obtained by projecting attribute information of the point cloud onto the two-dimensional plane and the thumbnail two-dimensional image different from the patch.

7. The image processing apparatus according to claim 1 , wherein
the circuitry is configured to encode the video frame in a multi layered structure, and
the circuitry is configured to generate a moving image that includes the thumbnail two-dimensional image in the video frame of some of layers in the multi-layered structure.

8. The image processing apparatus according to claim 1 , wherein
the circuitry is configured to encode the video frame in a multi layered structure, and
the circuitry is configured to generate a moving image that includes the thumbnail two-dimensional image in the video frame of all layers in the multi-layered structure.

9. The image processing apparatus according to claim 1 , wherein
the circuitry is configured to generate the bitstream that further includes information regarding the thumbnail two-dimensional image.

10. The image processing apparatus according to claim 9 , wherein
the information regarding the thumbnail two-dimensional image includes two-dimensional image presence/absence identification information that indicates whether or not the bitstream includes data of the thumbnail two-dimensional image.

11. The image processing apparatus according to claim 9 , wherein
the information regarding the thumbnail two-dimensional image includes two-dimensional image spatial position management information for managing a position in a spatial direction of the thumbnail two-dimensional image.

12. The image processing apparatus according to claim 9 , wherein
the information regarding the thumbnail two-dimensional image includes two-dimensional image temporal position management information for managing a position in a time direction of the thumbnail two-dimensional image.

13. The image processing apparatus according to claim 9 , wherein
the information regarding the thumbnail two-dimensional image includes two-dimensional image reproduction assisting information for assisting reproduction of the thumbnail two dimensional image.

14. The image processing apparatus according to claim 1 , wherein
the circuitry is configured to encode the thumbnail two-dimensional image independently of the patch.

15. The image processing apparatus according to claim 14 , wherein
the circuitry is configured to encode the thumbnail two-dimensional image by using a coding parameter for the thumbnail two-dimensional image.

16. The image processing apparatus according to claim 1 , wherein
the thumbnail two-dimensional image is displayed as a content of the object without being rendered at a decoding process.

17. An image processing method comprising:
generating a video frame that includes a patch obtained by projecting, onto a two-dimensional plane, a point cloud that represents an object having a three dimensional shape as a group of points;
generating a thumbnail two-dimensional image, the thumbnail two-dimensional image being generated independently from the patch;
embedding the thumbnail two-dimensional image into the video frame, the thumbnail two-dimensional image being arranged along the patch with an offset; and
encoding the video frame, as coded data of the video frame including the thumbnail two-dimensional image, to generate a bitstream, in which the coded data of the video frame is decoded to reconstruct the point cloud at a decoding device and the thumbnail two-dimensional image is extracted from the bitstream and decoded to display at the decoding device independently from reconstruction of the point cloud.

18. An image processing apparatus comprising:
circuitry configured to
extract, from a bitstream that includes coded data of a video frame that includes a patch obtained by projecting, onto a two-dimensional plane, a point cloud that represents an object having a three-dimensional shape as a group of points, and a thumbnail two dimensional image which is generated independently from the patch and embedded into the video frame arranged along the patch with an offset;
decode the coded data of the video frame to reconstruct the point cloud; and
decode the coded data extracted from the bitstream to restore the thumbnail two-dimensional image, independently from reconstruction of the point cloud by decoding the coded data of the video frame.

19. An image processing method comprising:
extracting, from a bitstream that includes coded data of a video frame that includes a patch obtained by projecting, onto a two-dimensional plane, a point cloud that represents an object having a three-dimensional shape as a group of points, and a thumbnail two-dimensional image which is generated independently from the patch and embedded into the video frame arranged along the patch with an offset;
decoding the coded data of the video frame to reconstruct the point cloud; and
decoding the coded data extracted from the bitstream to restore the thumbnail two-dimensional image, independently from reconstruction of the point cloud by decoding the coded data of the video frame.

Assignees

• Sony Corp

Inventors

• Nakagami Ohji
• Yano Koji
• Kato Tsuyoshi
• Kuma Satoru

Classifications

• H04N19/46 Primary

  Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence) · CPC title

• H04N19/167

  Position within a video image, e.g. region of interest [ROI] · CPC title

• H04N19/597 Primary

  specially adapted for multi-view video sequence encoding · CPC title

• H04N19/184

  the unit being bits, e.g. of the compressed video stream · CPC title

• H04N19/70

  characterised by syntax aspects related to video coding, e.g. related to compression standards · CPC title