Image processing device, image processing method, and program for correcting depth image based on positional information

Patent metadata

Field	Value
Publication number	US-9710955-B2
Application number	US-201214110957-A
Country	US
Kind code	B2
Filing date	May 11, 2012
Priority date	May 19, 2011
Publication date	Jul 18, 2017
Grant date	Jul 18, 2017

Abstract

This technology relates to an image processing device, an image processing method, and a program capable of realizing motion parallax closely related to that of the real world. An image processing unit corrects a depth image formed of a depth value indicating a position in a depth direction of each pixel of a color image of a predetermined viewpoint externally input as an input image based on positional information indicating a position of a viewer and the like. This technology is applicable to an image processing device, which performs a process to correct the depth image based on the positional information as an image process, for example.

First claim

The invention claimed is:

1. An image processing device, comprising:
an image processing unit, which corrects a depth image formed of a depth value indicating a position in a depth direction of each pixel of a color image of a predetermined viewpoint based on a change in viewer positional information indicating that a position of a viewer is moved in a depth direction with regard to a position at which the depth image is displayed,
wherein the image processing unit determines a central point of scaling of the depth image and the color image of the predetermined viewpoint based on the change in viewer positional information, determines a scaling rate of each pixel of the depth image and the color image of the predetermined viewpoint so as to be a larger depth value scaling rate for pixels having a larger horizontal distance from the central point and a smaller initial depth value prior to the change in viewer positional information, and performs the scaling of the depth value and a pixel value forming the color image of the predetermined viewpoint centering around the central point at the scaling rate of the pixel for each pixel,
wherein the scaling rate of each pixel of the depth image and the color image of the predetermined viewpoint changes based on the change in viewer positional information, and
wherein the image processing unit is implemented via at least one processor.

2. The image processing device according to claim 1 , wherein
the image processing unit corrects the depth value of each pixel of the depth image of the predetermined viewpoint in a direction toward a near side by the correction amount when the position of the viewer is on a side closer to a display device, which displays the color image of the predetermined viewpoint based on the depth image of the predetermined viewpoint corrected by the image processing unit, than a predetermined initial position and corrects the depth value of each pixel of the depth image of the predetermined viewpoint in a direction toward a far side by the correction amount when the position of the viewer is on a side opposite to the display device across the predetermined initial position.

3. The image processing device according to claim 1 , wherein
the image processing unit enlarges the depth value of each pixel of the depth image of the predetermined viewpoint at the scaling rate when the position of the viewer is on a side closer to a display device, which displays the color image of the predetermined viewpoint based on the depth image of the predetermined viewpoint corrected by the image processing unit, than a predetermined initial position and reduces the depth value of each pixel of the depth image of the predetermined viewpoint at the scaling rate when the position of the viewer is on a side opposite to the display device across the predetermined initial position.

4. The image processing device according to claim 3 , further comprising:
an enlarging processing unit, which enlarges the depth image of the predetermined viewpoint, wherein
the image processing unit performs the scaling of the depth image of the predetermined viewpoint enlarged by the enlarging processing unit based on the change in viewer positional information,
wherein the enlarging processing unit and the image processing unit are each implemented via at least one processor.

5. The image processing device according to claim 3 , wherein the image processing unit further performs the scaling of the enlarged depth image centering around the central point at a larger scaling rate for the pixel farther from central point with a smaller depth value.

6. The image processing device according to claim 1 , wherein
the image processing unit also performs a process to improve enhancement or a contrast to an area in the vicinity of a point of gaze of the color image of the predetermined viewpoint based on point-of-gaze positional information indicating a position of the point of gaze of the viewer, and performs a process to reduce the enhancement or the contrast to an area other than the area in the vicinity of the point of gaze.

7. The image processing device according to claim 1 , further comprising:
a display control unit, which performs a projecting process to project the color image of the predetermined viewpoint on the position of the viewer based on the positional information and the depth image of the predetermined viewpoint corrected by the image processing unit and allows a display device to display the color image of the predetermined viewpoint obtained as a result of the projecting process,
wherein the display control unit is implemented via at least one processor.

8. The image processing device according to claim 1 , further comprising:
a selecting unit, which selects the depth image of the predetermined viewpoint from depth images of a plurality of viewpoints based on the positional information,
wherein the selecting unit is implemented via at least one processor.

9. The image processing device according to claim 1 , further comprising:
a selecting unit, which selects depth images of a plurality of predetermined viewpoints from the depth images of a plurality of viewpoints based on the positional information; and
a combining unit, which combines the depth images of the plurality of predetermined viewpoints selected by the selecting unit to generate the depth image of the predetermined viewpoint,
wherein the selecting unit and the combining unit are each implemented via at least one processor.

10. The image processing device according to claim 1 , wherein
the predetermined viewpoint is a plurality of viewpoints, and
the image processing unit corrects the depth image of the viewpoint based on the positional information for each viewpoint of the predetermined viewpoints.

11. The image processing device according to claim 10 , further comprising:
a display control unit, which performs a projecting process to project the color image of the predetermined viewpoint on the position of the viewer based on the positional information and the depth image of the predetermined viewpoint corrected by the image processing unit and allows a display device, which displays an image of viewpoints more than the predetermined viewpoints, to display the color image of the predetermined viewpoint obtained as a result of the projecting process for each viewpoint of the predetermined viewpoints; and
a selecting unit, which selects the viewpoint assigned to the color image of the predetermined viewpoint obtained as the result of the projecting process from the viewpoints corresponding to the image displayed on the display device based on the positional information and selects the depth image of the predetermined viewpoint from depth images of a plurality of viewpoints based on the viewpoint,
wherein the display control unit, the selecting unit are each implemented via at least one processor.

12. The image processing device according to claim 10 , further comprising:
a display control unit, which performs a projecting process to project the color image of the predetermined viewpoint on the position of the viewer based on the positional information and the depth image of the predetermined viewpoint corrected by the image processing unit and allows a display device, which displays an image of viewpoints more than the predetermined viewpoints, to display the color image of the predetermined viewpoint obtained as a result of the projecting process for each viewpoint of the predetermined viewpoints;
a selecting unit, which selects the viewpoint assigned to the color image of the predetermined viewpoint obtained as the result of the projecting process from the view

Assignees

• Kobayashi Goh
• Kawai Takuro
• Ihara Toshinori
• Yamaoka Keisuke
• Sony Corp

Inventors

• Kobayashi Goh
• Kawai Takuro
• Ihara Toshinori
• Yamaoka Keisuke

Classifications

• G06T2219/2016

  Rotation, translation, scaling · CPC title

• H04N13/371

  for tracking viewers with different interocular distances; for tracking rotational head movements around the vertical axis · CPC title

• H04N13/373

  for tracking forward-backward translational head movements, i.e. longitudinal movements · CPC title

• H04N13/128

  Adjusting depth or disparity · CPC title

• G06T19/20 Primary

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title