Image processing apparatus and image processing method

Patent metadata

Field	Value
Publication number	US-11290698-B2
Application number	US-201615766625-A
Country	US
Kind code	B2
Filing date	Oct 28, 2016
Priority date	Nov 11, 2015
Publication date	Mar 29, 2022
Grant date	Mar 29, 2022

Abstract

There is provided an image processing apparatus and an image processing method by which three-dimensional data can be generated with high accuracy on the basis of two-dimensional image data and depth image data. A coordinate transformation data generation unit generates, on the basis of two-dimensional image data of a first viewpoint group and two-dimensional image data of a second viewpoint group, coordinate transformation data for converting a three-dimensional position in a first three-dimensional coordinate system of the first viewpoint group into a three-dimensional position in a second three-dimensional coordinate system of the second viewpoint group. A metadata addition unit transmits coordinate transformation information including encoded data of the two-dimensional image data of the first viewpoint group and depth image data, encoded data of the two-dimensional image data and depth image data of the second viewpoint group and coordinate transformation data generated by the coordinate transformation data generation unit.

First claim

The invention claimed is:

1. An image processing apparatus, comprising:
a coordinate transformation data generation unit configured to generate, based on two-dimensional image data of a first viewpoint and two-dimensional image data of a second viewpoint, coordinate transformation data for converting a three-dimensional position in a first three-dimensional coordinate system of the first viewpoint into a three-dimensional position in a second three-dimensional coordinate system of the second viewpoint; and
a transmission unit configured to transmit coordinate transformation information including
first encoded data that includes encoded data of the two-dimensional image data of the first viewpoint and depth image data of the first viewpoint indicative of a position of each pixel of a plurality of pixels of the two-dimensional image data of the first viewpoint in a depthwise direction of an image pickup object,
second encoded data that includes encoded data of the two-dimensional image data of the second viewpoint and depth image data of the second viewpoint indicative of a position of each pixel of a plurality of pixels of the two-dimensional image data of the second viewpoint in the depthwise direction of the image pickup object, and
the coordinate transformation data generated by the coordinate transformation data generation unit,

wherein the first encoded data and the second encoded data are synthesized based on synchronism deviation information in order to display image data of the synthesized encoded data from a selected free viewpoint other than the first viewpoint or the second viewpoint,
wherein the synchronism deviation information indicates a difference in image pickup time between the first encoded data and the second encoded data, is encoded as metadata in one of the first encoded data or the second encoded data, and is included in the synthesized encoded data, and
wherein the coordinate transformation data generation unit and the transmission unit are each implemented via at least one processor.

2. The image processing apparatus according to claim 1 , wherein
the coordinate transformation data is represented by a representation method same as that of an external parameter of a camera.

3. The image processing apparatus according to claim 1 , wherein
the first viewpoint comprises a plurality of first viewpoints, and
the coordinate transformation data generation unit generates the coordinate transformation data for each first viewpoint of the plurality of first viewpoints.

4. The image processing apparatus according to claim 3 , wherein
the coordinate transformation information includes coordinate transformation common information indicative of whether or not the coordinate transformation data of all of the first viewpoints are same.

5. The image processing apparatus according to claim 3 , wherein,
in a case in which the coordinate transformation data of all of the first viewpoints are same, the transmission unit transmits the coordinate transformation information including coordinate transformation common information indicating that the coordinate transformation data of all of the first viewpoints are same and the coordinate transformation data common to all of the first viewpoints.

6. The image processing apparatus according to claim 1 , wherein
the first encoded data and the second encoded data are synthesized in a synchronized relation with each other based on the synchronism deviation information.

7. The image processing apparatus according to claim 1 , wherein
the synchronism deviation information includes a synchronism deviation common flag indicating whether synchronism deviations are equal.

8. An image processing method by an image processing apparatus, comprising:
generating, based on two-dimensional image data of a first viewpoint and two-dimensional image data of a second viewpoint, coordinate transformation data for converting a three-dimensional position in a first three-dimensional coordinate system of the first viewpoint into a three-dimensional position in a second three-dimensional coordinate system of the second viewpoint; and
transmitting the generated coordinate transformation information including
first encoded data that includes encoded data of the two-dimensional image data of the first viewpoint and depth image data of the first viewpoint indicative of a position of each pixel of a plurality of pixels of the two-dimensional image data of the first viewpoint in a depthwise direction of an image pickup object,
second encoded data that includes encoded data of the two-dimensional image data of the second viewpoint and depth image data of the second viewpoint indicative of a position of each pixel of a plurality of pixels of the two-dimensional image data of the second viewpoint in the depthwise direction of the image pickup object, and
the generated coordinate transformation data,

wherein the first encoded data and the second encoded data are synthesized based on synchronism deviation information in order to display image data of the synthesized encoded data from a selected free viewpoint other than the first viewpoint or the second viewpoint, and
wherein the synchronism deviation information indicates a difference in image pickup time between the first encoded data and the second encoded data, is encoded as metadata in one of the first encoded data or the second encoded data, and is included in the synthesized encoded data.

9. An image processing apparatus, comprising:
a decoding unit configured to decode
first encoded data that includes encoded data of two-dimensional image data of a first viewpoint and depth image data of the first viewpoint indicative of a position of each pixel of a plurality of pixels of the two-dimensional image data of the first viewpoint in a depthwise direction of an image pickup object, and
second encoded data that includes encoded data of two-dimensional image data of a second viewpoint and depth image data of the second viewpoint indicative of a position of each pixel of a plurality of pixels of the two-dimensional image data of the second viewpoint in the depthwise direction of the image pickup object;

a first three-dimensional position conversion unit configured to convert, based on a first camera parameter in a first three-dimensional coordinate system of the first viewpoint and the two-dimensional image data and the depth image data of the first viewpoint obtained as a result of the decoding by the decoding unit, a two-dimensional position of each pixel of a plurality of pixels of the two-dimensional image data of the first viewpoint into a three-dimensional position in the first three-dimensional coordinate system; and
a coordinate transformation unit configured to convert, based on coordinate transformation information including coordinate transformation data for converting a three-dimensional position in the first three-dimensional coordinate system into a three-dimensional position in a second three-dimensional coordinate system of the second viewpoint, a three-dimensional position in the first three-dimensional coordinate system after the conversion by the first three-dimensional position conversion unit into a three-dimensional position in the second three-dimensional coordinate system,
wherein the first encoded data and the second encoded data are synthesized based on synchronism deviation information in order to display image data of the synthesized encoded data from a selected free viewpoint other than the first viewpoint or the second viewpoint,
wherein the synchronism deviation information indicates a difference in image pickup time between the first encoded data and the second encoded data, is encoded as metadata in one of the fi

Assignees

• Sony Corp

Inventors

• Tanaka Junichi

Classifications

• H04N19/597 Primary

  specially adapted for multi-view video sequence encoding · CPC title

• H04N13/194

  Transmission of image signals · CPC title

• G06T7/593

  from stereo images · CPC title

• H04N13/161 Primary

  Encoding, multiplexing or demultiplexing different image signal components (for multi-view video sequence encoding H04N19/597) · CPC title

• H04N19/46

  Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence) · CPC title