Scanning and tracking dynamic objects with depth cameras

Patent metadata

Field	Value
Publication number	US-9299195-B2
Application number	US-201414224575-A
Country	US
Kind code	B2
Filing date	Mar 25, 2014
Priority date	Mar 25, 2014
Publication date	Mar 29, 2016
Grant date	Mar 29, 2016

Abstract

A video conference server receives a plurality of video frames including a current frame and at least one previous frame. Each of the video frames includes a corresponding image and a corresponding depth map. The server produces a directional distance function (DDF) field that represents an area surrounding a target surface of the object captured in the current frame. A forward transformation is generated that modifies the reference surface to align with the target surface. Using at least a portion of the forward transformation, a backward transformation is calculated that modifies the target surface of the current frame to align with the reference surface. The backward transformation is then applied to the DDF to generate a transformed DDF. The server updates the reference model with the transformed DDF and transmits data for the updated reference model to enable a representation of the object to be produced at a remote location.

First claim

What is claimed is:

1. A method comprising:
receiving a plurality of video frames comprising a current frame and at least one previous frame, wherein each of the plurality of video frames includes data for a corresponding image and data for a corresponding depth map;
receiving a reference model comprising data from the at least one previous frame, the reference model including data representing a reference surface corresponding to at least one dynamic object captured in the plurality of video frames;
processing the data representing the depth map of the current frame to produce data for a directional distance function (DDF) field that represents an area surrounding a target surface of the at least one dynamic object captured in the current frame;
generating a forward transformation that modifies the data representing the reference surface to align with data representing the target surface and generate data for a modified reference surface;
calculating a backward transformation that modifies data representing the target surface of the current frame to align with the reference surface, wherein the backward transformation uses at least a portion of the forward transformation;
generating a transformed DDF by applying the backward transformation to the DDF;
fusing the transformed DDF into a multi-mode distance function (MDDF) comprising multiple distance values and direction vectors at each of a plurality of voxels; and
updating the reference model based on the MDDF to produce data for an updated reference model.

2. The method of claim 1 , further comprising transmitting data for the updated reference model to enable a representation of the dynamic object to be produced at a remote location.

3. The method of claim 1 , further comprising:
updating the data for the reference surface using the updated reference model to produce an updated reference surface; and
applying a texture to the data for the updated reference surface based on at least one image from the plurality of video frames.

4. The method of claim 1 , further comprising receiving the plurality of video frames from a plurality of video cameras, each of the plurality of video cameras configured to measure a distance to one or more objects in a physical scene.

5. The method of claim 4 , wherein each of the plurality of video frames comprises a composite of video frames from the plurality of video cameras.

6. The method of claim 1 , wherein the portion of the forward transformation used in calculating the backward transformation comprises a set of matched key points.

7. The method of claim 1 , wherein generating data for the modified reference surface comprises deforming each particular point with a three dimensional position vector ν from a plurality of points on the reference surface to align with the target surface by applying a linearly blended affine transformation from a set of points neighboring the particular point according to

v
~

=

∑

j
∈
N

⁢

⁢

w
j

⁡

[

A
j

⁡

(

v
-

g
j

)

+

g
j

+

t
j

]

,

where {tilde over (ν)} is a position vector of the particular point after the deformation, N is the set of neighboring nodes of ν on the deformation graph, w j is a blending weight for a j th node in the set N, g j is a position vector of the j th node, A j is a 3×3 matrix, and t j is a translation vector of the j th node, wherein A j and t j represent the affine transformation associated with the j th node on the deformation graph, and wherein collectively {<A j , t j >} j=0, . . . , J represent the deformation parameters of the deformation graph, where J is the number of the graph nodes in the deformation graph.

8. The method of claim 7 , wherein the deformation parameters {<A j , t j >} j=0, . . . , J in the deformation graph are estimated by solving

min

{

〈

A
j

,

t
j

〉

}

⁢

w
rot

⁢

E
rot

+

w
reg

⁢

E
reg

+

w
con

⁢

E
con

+

w
dns_pts

⁢

E
dns_pts

+

w
clr

⁢

E
clr

,

where w rot , w reg , w con , w dns _ pts , and w clr , are weighting coefficients, E rot is an energy term that constrains column vectors of A j to be orthogonal and unitary, E reg is an energy term that ensures smoothness of the deformed reference surface, E

Assignees

• Cisco Tech Inc

Inventors

• Dou Mingsong
• Fuchs Henry
• Marathe Madhav

Classifications

• G06T17/00 Primary

  Three-dimensional [3D] modelling for computer graphics · CPC title

• G06T2215/16

  Using real world measurements to influence rendering · CPC title

• G06T19/20 Primary

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• G06T7/0065

  Physics · mapped topic

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title