Image processing device and image processing method

Patent metadata

Field	Value
Publication number	US-9979961-B2
Application number	US-201214003986-A
Country	US
Kind code	B2
Filing date	Mar 9, 2012
Priority date	Mar 18, 2011
Publication date	May 22, 2018
Grant date	May 22, 2018

Abstract

The invention relates to an image processing device and an image processing method capable of collectively encoding a color image and a depth image of different resolutions. The image processing device comprising an image frame converting unit that converts the resolution of the depth image to the same resolution as that of the color image. An additional information generating unit that generates additional information including information to specify the color image, or information to specify the depth image, image frame conversion information indicating an area of a black image included in the depth image, the resolution of which is converted, and resolution information to distinguish whether the resolutions of the color image and the depth image are different from each other. This technology may be applied to the image processing device of images of multiple viewpoints.

First claim

The invention claimed is:

1. An image processing device, comprising:
one or more processors configured to:
generate additional information that includes information to specify:
a color image of a subject and a first depth image of the subject;
conversion information that indicates a difference between a first number of pixels of a second depth image of the subject and a second number of pixels of the first depth image,
wherein a first resolution of the second depth image is based on a second resolution of the first depth image,
wherein the first resolution of the second depth image is same as a third resolution of the color image; and

resolution information that indicates the third resolution of the color image and the second resolution of the first depth image;

encode, based on the additional information, the color image and the second depth image to generate an encoded bit stream,
wherein the encoded bit stream comprises the additional information as SEI (supplemental enhancement information) in the encoded bit stream; and

transmit the encoded bit stream.

2. The image processing device according to claim 1 , wherein the one or more processors are further configured to encode the color image and the second depth image, based on an MVC (multiview video coding) scheme.

3. The image processing device according to claim 1 , wherein the one or more processors are further configured to:
encode the color image and the second depth image, based on an MVC (multiview video coding) scheme,
wherein the additional information specifies:
the color image and the first depth image as SEI in the encoded bit stream; and
the conversion information and the resolution information as a Subset SPS (sequence parameter set) in the encoded bit stream.

4. The image processing device according to claim 1 , wherein the one or more processors are further configured to convert the second resolution of the first depth image to the first resolution.

5. The image processing device according to claim 4 , wherein the one or more processors are further configured to:
convert the second resolution of the first depth image to the first resolution by addition of an image to the first depth image,
wherein the second resolution of the first depth image is less than the third resolution of the color image.

6. An image processing method, comprising:
in an image processing device:
generating additional information including information to specify:
a color image of a subject and a depth image of the subject;
conversion information that indicates a difference between a first number of pixels of a second depth image and second number of pixels of a first depth image,
wherein a first resolution of the second depth image is based on a second resolution of the first depth image,
wherein the first resolution of the second depth image is same as a third resolution of the color image; and

resolution information that indicates the third resolution of the color image and the second resolution of the first depth image;

encoding, based on the additional information, the color image and the second depth image to generate an encoded bit stream,
wherein the encoded bit stream comprises the additional information as SEI (supplemental enhancement information) in the encoded bit stream; and

transmitting the encoded bit stream.

7. An image processing device, comprising:
one or more processors configured to:
receive an encoded bit stream comprising additional information as SEI (supplemental enhancement information) in the encoded bit stream,
wherein the encoded bit stream is based on a color image of a subject and a first depth image of the subject,
wherein a first resolution of the first depth image is same as a second resolution of the color image, and

wherein the additional information includes information to specify:
the color image and a second depth image of the subject,
conversion information that indicates a difference between a first number of pixels of the first depth image and a second number of pixels of the second depth image,
wherein the first resolution of the first depth image is based on a third resolution of the second depth image; and
resolution information that indicates the second resolution of the color image and the third resolution of the second depth image;

decode the encoded bit stream to generate the color image and the first depth image;
extract the second depth image with the third resolution from the generated first depth image, based on the received additional information;
generate a new color image based on the color image and the extracted second depth image.

8. The image processing device according to claim 7 , wherein the encoded bit stream comprises the color image and the first depth image that are encoded based on an MVC (multiview video coding) scheme.

9. The image processing device according to claim 7 ,
wherein the encoded bit stream comprises the color image and the first depth image that are encoded based on an MVC (multiview video coding) scheme,
wherein the additional information specifies:
the color image and the second depth image as SEI in the encoded bit stream; and
the conversion information and the resolution information as a Subset SPS (sequence parameter set) in the encoded bit stream.

10. The image processing device according to claim 7 ,
wherein the first resolution of the first depth image is based on conversion of the third resolution of the second depth image by addition of an image to the second depth image,
wherein the third resolution of the second depth image is lower than the second resolution of the color image.

11. An image processing method, comprising:
in an image processing device:
receiving an encoded bit stream comprising additional information as SEI (supplemental enhancement information) in the encoded bit stream,
wherein the encoded bit stream is based on a color image of a subject and a first depth image of the subject,
wherein a first resolution of the first depth image is same as a second resolution of the color image, and

wherein the additional information includes information to specify:
the color image and a second depth image of the subject,
conversion information that indicates a difference between a first number of pixels of the first depth image and a second number of pixels of the second depth image,
wherein the first resolution of the first depth image is based on a third resolution of the second depth image; and
resolution information that indicates the second resolution of the color image and the third resolution of the second depth image;

decoding the encoded bit stream to generate the color image and the first depth image;
extracting the second depth image with the third resolution from the generated first depth image, based on the received additional information; and
generating a new color image based on the color image and the extracted second depth image.

12. An image processing device, comprising:
one or more processors configured to:
receive an encoded bit stream comprising additional information as SEI (supplemental enhancement information) in the encoded bit stream,
wherein the encoded bit stream is based on a color image of a subject and a first depth image of the subject,
wherein a first resolution of the first depth image is same as a second resolution of the color image, and
wherein the additional information includes information to specify:
the color image and a second depth image of the subject,
conversion information that indicates a difference between a first number of pixels of the first depth image and a second number of pixels o

Assignees

• Takahashi Yoshitomo
• Hattori Shinobu
• Sony Corp

Inventors

• Takahashi Yoshitomo
• Hattori Shinobu

Classifications

• H04N13/282 Primary

  for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems · CPC title

• H04N19/597

  specially adapted for multi-view video sequence encoding · CPC title

• H04N19/59

  involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution · CPC title

• H04N2213/005

  Aspects relating to the "3D+depth" image format · CPC title

• H04N13/161

  Encoding, multiplexing or demultiplexing different image signal components (for multi-view video sequence encoding H04N19/597) · CPC title