Estimation of object properties in 3D world

Patent metadata

Field	Value
Publication number	US-9158972-B2
Application number	US-201414478242-A
Country	US
Kind code	B2
Filing date	Sep 5, 2014
Priority date	Jun 7, 2011
Publication date	Oct 13, 2015
Grant date	Oct 13, 2015

Abstract

Objects within two-dimensional video data are modeled by three-dimensional models as a function of object type and motion through manually calibrating a two-dimensional image to the three spatial dimensions of a three-dimensional modeling cube. Calibrated three-dimensional locations of an object in motion in the two-dimensional image field of view of a video data input are determined and used to determine a heading direction of the object as a function of the camera calibration and determined movement between the determined three-dimensional locations. The two-dimensional object image is replaced in the video data input with an object-type three-dimensional polygonal model having a projected bounding box that best matches a bounding box of an image blob, the model oriented in the determined heading direction. The bounding box of the replacing model is then scaled to fit the object image blob bounding box, and rendered with extracted image features.

First claim

What is claimed is:

1. A method for modeling objects within two-dimensional video data by three-dimensional models as a function of object type and motion, the method comprising:
in response to observing an image of an object in motion in a two-dimensional image field of view of a video data input from a camera, determining an initial three-dimensional location of the observed two-dimensional object image as an intersection between a ground plane of the camera field of view and a backward projected line passing through a center of the camera and a point on the object two-dimensional image within a focal plane in the two-dimensional image field of view of the video data input at an initial time;
determining a second three-dimensional location of the observed two-dimensional object image via the processor as an intersection between the ground plane and another backward projected line passing through the center of the camera and the point on the object two-dimensional image within the video data input two-dimensional image field of view focal plane at a second time that is subsequent to the initial time;
determining a heading direction of the object as a function of a movement of the two-dimensional object image from the determined initial three-dimensional location to the determined second subsequent three-dimensional location;
replacing the two-dimensional object image within the video data input with a selected three-dimensional polygonal model;
orienting the selected three-dimensional polygonal model replacing the two-dimensional object image within the video data input in the determined heading direction; and
scaling a bounding box of the selected three-dimensional polygonal model replacing the two-dimensional object image within the video data input from the camera to fit an image blob bounding box determined for the observed two-dimensional object image.

2. The method of claim 1 , further comprising:
integrating computer-readable program code into a computer system comprising a processor, a computer readable memory and a computer readable storage medium; and
wherein the processor executes program code instructions stored on the computer-readable storage medium via the computer readable memory and thereby performs the steps of determining the initial three-dimensional location of the observed two-dimensional object image in response to observing the image of the object in motion in the two-dimensional image field of view of the video data input from the camera, determining the second three-dimensional location of the observed two-dimensional object image, determining the heading direction of the object, replacing the two-dimensional object image within the video data input with the selected three-dimensional polygonal model, orienting the selected three-dimensional polygonal model in the determined heading direction, and scaling the bounding box of the selected three-dimensional polygonal model within the video data input from the camera to fit the image blob bounding box determined for the observed two-dimensional object image.

3. The method of claim 1 , further comprising:
selecting the selected three-dimensional polygonal model from a plurality of object-type three-dimensional polygonal models that each have projected sizes and different bounding box aspect ratios, in response to the selected three-dimensional polygonal model having:
a ratio of projected size to a size of image blob bounding box determined for the observed two-dimensional object image that meets a threshold value; and
an aspect ratio that is more similar to an aspect ratio of the image blob bounding box determined for the observed two-dimensional object image, relative to the aspect ratios of the bounding boxes of others of the plurality of object-type three-dimensional polygonal models.

4. The method of claim 3 , wherein the step of selecting the selected three-dimensional polygonal model from the plurality of object-type three-dimensional polygonal models further comprises:
determining for each of the models weighted sums of their respective aspect ratios and ratios of projected size to the size of the image blob bounding box determined for the observed two-dimensional object image, wherein the aspect ratios are weighted more heavily than the ratios of projected size; and
selecting a model having a best weighted sum value as the selected model.

5. The method of claim 4 , further comprising:
selecting the model having a best weighted sum value as the selected model from a subset of the models that each have an aspect ratio similarity to the aspect ratio of the image blob bounding box that satisfies a threshold condition.

6. The method of claim 4 , further comprising:
selecting one of a single-dimension scaling process and a multi-dimension scaling process as appropriate to the object type of the replacing polygonal three-dimensional model, wherein the single-dimension and multi-dimension scaling processes are appropriate to different ones of the object types of the polygonal three-dimensional model; and
wherein the scaling the bounding box ratio of the replacing polygonal three-dimensional model comprises using the selected one of the single-dimension scaling process and the multi-dimension scaling process.

7. The method of claim 6 , wherein the selected scaling process is the single-dimension scaling process, and wherein the scaling the bounding box ratio of the replacing polygonal three-dimensional model to correspond to the object image blob bounding box ratio comprises:
determining a first spatial dimension of three-dimensional spatial dimensions of the projected bounding box of the selected model, the first spatial dimension of the object image blob bounding box, and a first spatial dimension ratio between the determined selected model first spatial dimension and the determined object image blob bounding box first spatial dimension;
scaling the bounding box of the selected model in the first dimension by the determined first spatial dimension ratio to match the object image blob bounding box; and
shifting the selected model so that a location point of the selected model on a boundary box line of the projected bounding box of the selected model that is normal to an axis of the first dimension is co-located with a corresponding point of the object image blob on a corresponding boundary box line of the object image blob bounding box, wherein the selected model location point is on a back projection line comprising the corresponding point of the object image blob and the center of the camera.

8. The method of claim 6 , wherein the selected scaling process is the multi-dimension scaling process, and wherein the scaling the bounding box ratio of the replacing polygonal three-dimensional model to correspond to the object image blob bounding box ratio comprises:
determining a first dimensional vector of three-dimensional spatial dimensions of the two-dimensional object image blob bounding box in image space on the image ground plane aligned with a first dimension axis of the object image blob bounding box;
determining a second dimensional vector of the three-dimensional spatial dimensions of the two-dimensional object image blob bounding box in image space on the image ground plane through alignment with a second dimension axis of the object image blob bounding box;
determining a heading direction vector as a function of calibrating of the camera and the movement of the two-dimensional object image from the determined initial three-dimensional location to the determined second subsequent three-dimensional location;
determining a third dimension vector perpendicular to the heading vector and aligned with a third dimension axis of the three-dimensional spatial dimensions

Assignees

• IBM

Inventors

• Datta Ankur
• Feris Rogerio S
• Zhai Yun

Classifications

• G06T7/251 Primary

  involving models · CPC title

• G06T17/10

  Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes · CPC title

• G06K9/00711 Primary

  Physics · mapped topic

• G06T7/2046

  Physics · mapped topic

• G06K9/00771

  Physics · mapped topic