Estimation of object properties in 3D world

Patent metadata

Field	Value
Publication number	US-9805505-B2
Application number	US-201615223018-A
Country	US
Kind code	B2
Filing date	Jul 29, 2016
Priority date	Jun 7, 2011
Publication date	Oct 31, 2017
Grant date	Oct 31, 2017

Abstract

Objects within two-dimensional video data are modeled by three-dimensional models as a function of object type and motion through manually calibrating a two-dimensional image to the three spatial dimensions of a three-dimensional modeling cube. Calibrated three-dimensional locations of an object in motion in the two-dimensional image field of view of a video data input are determined and used to determine a heading direction of the object as a function of the camera calibration and determined movement between the determined three-dimensional locations. The two-dimensional object image is replaced in the video data input with an object-type three-dimensional polygonal model having a projected bounding box that best matches a bounding box of an image blob, the model oriented in the determined heading direction. The bounding box of the replacing model is then scaled to fit the object image blob bounding box, and rendered with extracted image features.

First claim

What is claimed is:

1. A computer-implemented method for modeling objects within two-dimensional video data by three-dimensional models, the method comprising executing on a processor the steps of:
replacing an image of an object observed within a two-dimensional image field of view of video data from a camera with a three-dimensional polygonal model that is selected from a plurality of object-type three-dimensional polygonal models that each have projected sizes and different bounding box aspect ratios, in response to the selected three-dimensional polygonal model having:
a ratio of a projected size to a size of an image blob bounding box determined for the observed two-dimensional object image that meets a threshold value; and
an aspect ratio that is more similar to an aspect ratio of the image blob bounding box determined for the observed two-dimensional object image, relative to aspect ratios of bounding boxes of respective ones of others of the plurality of object-type three-dimensional polygonal models.

2. The method of claim 1 , further comprising:
integrating computer-readable program code into a computer system comprising a processor, a computer readable memory and a computer readable storage medium; and
wherein the processor executes program code instructions stored on the computer-readable storage medium via the computer readable memory and thereby performs the steps of replacing the image of the object observed within the two-dimensional image field of view of video data from the camera with the three-dimensional polygonal model selected in response to having the ratio of projected size to the size of an image blob bounding box determined for the observed two-dimensional object image that meets the threshold value, and the aspect ratio that is more similar to the aspect ratio of the image blob bounding box determined for the observed two-dimensional object image relative to the aspect ratios of the bounding boxes of others of the plurality of object-type three-dimensional polygonal models.

3. The method of claim 1 , wherein the step of selecting the selected three-dimensional polygonal model from the plurality of object-type three-dimensional polygonal models further comprises:
determining for each of the models weighted sums of their respective aspect ratios and ratios of projected size to the size of the image blob bounding box determined for the observed two-dimensional object image, wherein the aspect ratios are weighted more heavily than the ratios of projected size; and
selecting a model having a best weighted sum value as the selected model.

4. The method of claim 1 , further comprising:
scaling a bounding box of the selected three-dimensional polygonal model within the video data input from the camera to fit the image blob bounding box determined for the observed two-dimensional object image, by scaling in no more than two of the dimensions of the three-dimensional polygonal model.

5. The method of claim 4 , wherein the step of scaling the bounding box of the selected three-dimensional polygonal model comprises:
selecting one of a single-dimension scaling process and a multi-dimension scaling process as most appropriate to an object type of the polygonal three-dimensional model, wherein the single-dimension and multi-dimension scaling processes are appropriate to different ones of object types of different ones of the plurality of polygonal three-dimensional models; and
wherein the scaling the bounding box ratio of the replacing polygonal three-dimensional model comprises using the selected one of the single-dimension scaling process and the multi-dimension scaling process.

6. The method of claim 5 , wherein the selected scaling process is the single-dimension scaling process, and wherein the scaling the bounding box ratio of the replacing polygonal three-dimensional model to correspond to the object image blob bounding box ratio comprises:
determining a first spatial dimension of three-dimensional spatial dimensions of the projected bounding box of the selected model, the first spatial dimension of the object image blob bounding box, and a first spatial dimension ratio between the determined selected model first spatial dimension and the determined object image blob bounding box first spatial dimension;
scaling the bounding box of the selected model in the first dimension by the determined first spatial dimension ratio to match the object image blob bounding box; and
shifting the selected model so that a location point of the selected model on a boundary box line of the projected bounding box of the selected model that is normal to an axis of the first dimension is co-located with a corresponding point of the object image blob on a corresponding boundary box line of the object image blob bounding box, wherein the selected model location point is on a back projection line comprising the corresponding point of the object image blob and the center of the camera.

7. The method of claim 5 , wherein the selected scaling process is the multi-dimension scaling process, and wherein the scaling the bounding box ratio of the replacing polygonal three-dimensional model to correspond to the object image blob bounding box ratio comprises:
determining a first dimensional vector of three-dimensional spatial dimensions of the two-dimensional object image blob bounding box in image space on an image ground plane aligned with a first dimension axis of the object image blob bounding box;
determining a second dimensional vector of the three-dimensional spatial dimensions of the two-dimensional object image blob bounding box in image space on the image ground plane through alignment with a second dimension axis of the object image blob bounding box;
determining a heading direction vector as a function of calibrating of the camera and the movement of the two-dimensional object image from the determined initial three-dimensional location to the determined second subsequent three-dimensional location;
determining a third dimension vector perpendicular to the heading vector and aligned with a third dimension axis of the three-dimensional spatial dimensions;
projecting the first dimensional vector to the heading direction vector and the third dimension vector to obtain first projected dimension scaling factors for each of the first dimension and the second dimension;
projecting the second dimensional vector to the heading direction vector and the third dimension vector to obtain second projected dimension scaling factors for each of the first dimension and the second dimension;
determining a final scaling factor for the first dimension by summing the first and second projected dimension scaling factors obtained for the first dimension;
determining a final scaling factor for the second dimension by summing the first and second projected dimension scaling factors obtained for the second dimension;
selecting one of the determined final first and second dimension scaling factors as a final third dimension scaling factor; and
scaling each of the three dimensions of the selected model by their respective final first, second and third dimension scaling factors.

8. The method of claim 5 , further comprising:
determining the initial three-dimensional location of the observed two-dimensional object image as an intersection between a ground plane of the camera field of view and a backward projected line passing through a center of the camera and a point on the object two-dimensional image within a focal plane in the two-dimensional image field of view of the video data input at the initial time; and
determining the second three-dimensional location of the observed two-dimensional object image as an intersection between the ground plane and another back

Assignees

• IBM
• Int Business Machines Corproation

Inventors

• Datta Ankur
• Feris Rogerio S
• Zhai Yun

Classifications

• G06T7/251 Primary

  involving models · CPC title

• G06T17/10 Primary

  Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes · CPC title

• G06K9/52

  Physics · mapped topic

• G06T7/20

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title

• G06K9/46

  Physics · mapped topic