Motion compensation method and apparatus for depth images

Patent metadata

Field	Value
Publication number	US-2016232684-A1
Application number	US-201314353171-A
Country	US
Kind code	A1
Filing date	Oct 18, 2013
Priority date	Oct 18, 2013
Publication date	Aug 11, 2016
Grant date	—

Abstract

In one embodiment, an image processor is configured to obtain a plurality of phase images for each of first and second depth frames. For each of a plurality of pixels of a given one of the phase images of the first depth frame, the image processor determines an amount of movement of a point of an imaged scene between the pixel of the given phase image and a pixel of a corresponding phase image of the second depth frame, and adjusts pixel values of respective other phase images of the first depth frame based on the determined amount of movement. A motion compensated first depth image is generated utilizing the given phase image and the adjusted other phase images of the first depth frame. Movement of a point of the imaged scene is determined, for example, between pixels of respective n-th phase images of the first and second depth frames.

First claim

1 . A method comprising:
obtaining a plurality of phase images for each of first and second depth frames; for each of a plurality of pixels of a given one of the phase images of the first depth frame: determining an amount of movement of a point of an imaged scene between the pixel of the given phase image of the first depth frame and a pixel of a corresponding phase image of the second depth frame; and adjusting pixel values of respective other phase images of the first depth frame based on the determined amount of movement; wherein a motion compensated first depth image is generated utilizing the given phase image and the adjusted other phase images of the first depth frame; and wherein said obtaining, determining and adjusting are implemented in at least one processing device comprising a processor coupled to a memory.

2 . The method of claim 1 wherein the pluralities of phase images for respective ones of the first and second depth frames comprise respective sequences of at least four phase images each having a different capture time.

3 . The method of claim 1 wherein the determining and adjusting are repeated for substantially all of the pixels of the given phase image that are associated with a particular object of the imaged scene.

4 . The method of claim 1 wherein determining an amount of movement comprises determining an amount of movement of a point of an imaged scene between a pixel of an n-th one of the phase images of the first depth frame and a pixel of an n-th one of the phase images of the second depth frame.

5 . The method of claim 4 wherein determining an amount of movement comprises determining an amount of movement of a point of the imaged scene between a pixel of an initial one of the phase images of the first depth frame and a pixel of an initial one of the phase images of the second depth frame.

6 . The method of claim 4 wherein determining an amount of movement comprises solving an equation of the following form:
I n ( x+nV x Δt,y+nV y Δt,t+nΔt )−= I n ( x|V x (Δ T|nΔt ), y|V y (Δ T|nΔt ), t |ΔT|nΔT )

to determine a velocity (V x , V y ) of the point of the imaged scene, where I n (x,y,t) denotes the value of pixel (x,y) of the n-th phase image at time t, Δt denotes the time between two consecutive phase images of a given one of the first and second depth frames, and ΔT denotes the time between the first and second depth frames.

7 . The method of claim 6 wherein solving the equation comprises solving a system of multiple equations of the form:

∂

I
n

∂
x



V
x

+

∂

I
n

∂
y



V
y

+

∂

I
n

∂
t

=
0

for respective ones of the phase images of the first and second depth frames.

8 . The method of claim 1 wherein adjusting pixel values of respective other phase images of the first depth frame comprises transforming the other phase images such that the point of the imaged scene has substantially the same pixel coordinates in each of the phase images of the first depth frame.

9 . The method of claim 1 wherein adjusting pixel values of respective other phase images of the first depth frame comprises:
moving values of the pixels of respective other phase images to positions within those images corresponding to a position of the pixel in the given phase image; and
assigning replacement values to the pixels for which values were moved.

10 . The method of claim 9 wherein assigning replacement values comprises at least one of:
assigning the replacement values as predetermined values;
assigning the replacement values based on values of corresponding pixels in a phase image of at least one previous or subsequent depth frame; and
assigning the replacement values as a function of a plurality of neighboring pixel values within the same phase image.

11 . The method of claim 1 further comprising:
generating a motion compensated first amplitude image corresponding to the first depth image;
wherein the motion compensated first amplitude image is generated utilizing the given phase image and the adjusted other phase images of the first depth frame.

12 . A computer-readable storage medium having computer program code embodied therein, wherein the computer program code when executed in the processing device causes the processing device to perform the method of claim 1 .

13 . An apparatus comprising:
at least one processing device comprising a processor coupled to a memory; wherein said at least one processing device is configured: to obtain a plurality of phase images for each of first and second depth frames; for each of a plurality of pixels of a given one of the phase images of the first depth frame: to determine an amount of movement of a point of an imaged scene between the pixel of the given phase image of the first depth frame and a pixel of a corresponding phase image of the second depth frame; and to adjust pixel values of respective other phase images of the first depth frame based on the determined amount of movement; wherein a motion compensated first depth image is generated utilizing the given phase image and the adjusted other phase images of the first depth frame.

14 . The apparatus of claim 13 wherein said at least one processing device is implemented within a depth imager.

15 . The apparatus of claim 14 wherein the depth imager comprises a ToF camera.

16 . An integrated circuit comprising the apparatus of claim 13 .

17 . The integrated circuit of claim 16 wherein the integrated circuit is adapted for coupling to an image sensor of a depth imager.

18 . A depth imager comprising:
an image sensor; and an image processor coupled to the image sensor; wherein the image processor is configured: to obtain from the image sensor a plurality of phase images for each of first and second depth frames; for each of a plurality of pixels of a given one of the phase images of the first depth frame: to determine an amount of movement of a point of an imaged scene between the pixel of the given phase image of the first depth frame and a pixel of a corresponding phase image of the second depth frame; and to adjust pixel values of respective other phase images of the first depth frame based on the determined amount of movement; wherein a motion compensated

Assignees

• Kholodenko Alexander Borisovich
• Parkhomenko Denis Vladimirovich
• Petyushko Alexander Alexandrovich
• Parfenov Denis Vasilievich
• Zaicevs Deniss
• Lsi Corp

Inventors

• Kholodenko Alexander B
• Parkhomenko Denis V
• Petyushko Alexander A
• Parfenov Denis V
• Zaytsev Denis V

Classifications

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G01S7/497

  Means for monitoring or calibrating · CPC title

• G06T7/285

  using a sequence of stereo image pairs · CPC title

• G06T7/269

  using gradient-based methods · CPC title

• H04N13/204 Primary

  using stereoscopic image cameras (stereoscopic photography G03B35/00) · CPC title