Systems and methods for tracking interacting objects

Patent metadata

Field	Value
Publication number	US-9794525-B2
Application number	US-201514666906-A
Country	US
Kind code	B2
Filing date	Mar 24, 2015
Priority date	Mar 25, 2014
Publication date	Oct 17, 2017
Grant date	Oct 17, 2017

Abstract

Systems and methods for tracking interacting objects may acquire, with a sensor, and two or more images associated with two or more time instances. A processor may generate image data from the two or more images. The processor may apply an extended Probability Occupancy Map (POM) algorithm to the image data to obtain probability of occupancy for a container class of potentially interacting objects, probability of occupancy for a containee class of the potentially interacting objects, and a size relationship of the potentially interacting objects, over a set of discrete locations on a ground plane for each time instance. The processor may estimate trajectories of an object belonging to each of the two classes by determining a solution of a tracking model on the basis of the occupancy probabilities and a set of rules describing the interaction between objects of different or the same classes.

First claim

What is claimed is:

1. A method for tracking interacting objects comprising:
acquiring, with at least one sensor, at least two images associated with at least two time instances;
generating, with at least one processor, image data from the at least two images;
applying, with the at least one processor, an extended Probability Occupancy Map (POM) algorithm to the image data to obtain probability of occupancy for a container class of potentially interacting objects, probability of occupancy for a containee class of the potentially interacting objects, and a size relationship of the potentially interacting objects, over a set of discrete locations on a ground plane for each time instance; and
estimating, with the at least one processor, trajectories of at least one object belonging to each of the two classes by determining a solution of a tracking model on the basis of the occupancy probabilities and a set of rules describing the interaction between objects of different or the same classes,
wherein estimating the trajectories comprises:
modeling, with the at least one processor, at least one transition, each transition being between a first one of the at least two images defining a first location with a first orientation of the at least one object, and a second one of the at least two images defining a second location with a second orientation of the at least one object, to produce a flow variable for each transition;
modeling, with the at least one processor, a set of rules that describes the interaction between objects of the same or different classes to allow physically-plausible interactions and disallow implausible interactions;
defining, with the at least one processor, an objective function in terms of a linear combination of each of the flow variables; and
solving, with the at least one processor, the objective function using at least one linear solver.

2. The method of claim 1 , wherein an object classified within the container class or the containee class is a vehicle, a person, or an object that can be carried by a person.

3. The method of claim 1 , wherein acquiring the at least two images comprises:
setting up at least one sensor around an area of interest;
generating, with the at least one processor, a camera calibration of the at least one sensor;
generating, with the at least one processor, a discretization of a ground surface of the area of interest into square grids each having a location, wherein each of the grids can be potentially occupied by the objects; and
keeping the cameras steady during each period of acquiring each of the at least two images.

4. The method of claim 3 , wherein applying the POM algorithm comprises, for each time instance:
generating, with the at least one processor, a background subtraction from the image data and obtaining a binary image; and
generating, with the at least one processor, a probability for the object occupying a specific square grid location with a specific orientation.

5. The method of claim 4 , wherein applying the POM algorithm further comprises:
for each class, initializing, with the at least one processor, a template of an object comprising a probability of a specific location with a specific orientation into the square grid;
for each template in the area of interest, generating, with the at least one processor, a probability of a presence of the object.

6. The method of claim 1 , wherein modeling the at least one transition comprises:
for each object of one of the classes at the first location with the first orientation, determining, with the at least one processor, a possible transition neighborhood of the object in the second image; and
modeling, with the at least one processor, the transition of each object to each determined neighborhood with a flow variable.

7. The method of claim 1 , wherein the set of rules comprises:
for objects of the same class, a same spatial location cannot be occupied by more than one object;
the container object can only appear or disappear at the edge of the area of interest;
the containee object can appear or disappear at the locations of a container object or at the edge of the area of interest; and
a maximum number of instances of the object among the area of interest.

8. The method of claim 1 , wherein the at least one sensor comprises at least one camera.

9. A system for tracking interacting objects comprising:
at least one sensor configured to acquire at least two images associated with at least two time instances; and
at least one processor in communication with the at least one sensor and configured to:
generate image data from the at least two images;
apply an extended Probability Occupancy Map (POM) algorithm to the image data to obtain probability of occupancy for a container class of potentially interacting objects, probability of occupancy for a containee class of the potentially interacting objects, and a size relationship of the potentially interacting objects, over a set of discrete locations on a ground plane for each time instance; and
estimate trajectories of at least one object belonging to each of the two classes by determining a solution of a tracking model on the basis of the occupancy probabilities and a set of rules describing the interaction between objects of different or the same classes,

wherein the at least one processor is configured to estimate the trajectories by:
modeling at least one transition, each transition being between a first one of the at least two images defining a first location with a first orientation of the at least one object, and a second one of the at least two images defining a second location with a second orientation of the at least one object, to produce a flow variable for each transition;
modeling a set of rules that describes the interaction between objects of the same or different classes to allow physically-plausible interactions and disallow implausible interactions;
defining an objective function in terms of a linear combination of each of the flow variables; and
solving, with the at least one processor, the objective function using at least one linear solver.

10. The system of claim 9 , wherein an object classified within the container class or the containee class is a vehicle, a person, or an object that can be carried by a person.

11. The system of claim 9 , wherein acquiring the at least two images comprises:
setting up at least one sensor around an area of interest;
generating a camera calibration of the at least one sensor;
generating a discretization of a ground surface of the area of interest into square grids each having a location, wherein each of the grids can be potentially occupied by the objects; and
keeping the cameras steady during each period of acquiring each of the at least two images.

12. The system of claim 11 , wherein the at least one processor is configured to apply the POM algorithm, for each time instance, by:
generating a background subtraction from the image data and obtaining a binary image; and
generating a probability for the object occupying a specific square grid location with a specific orientation.

13. The system of claim 12 , wherein the at least one processor is further configured to apply the POM algorithm by:
for each class, initializing a template of an object comprising a probability of a specific location with a specific orientation into the square grid;
for each template in the area of interest, generating a probability of a presence of the object.

14. The system of claim 9 , wherein the at least one processor is configured to model the at least one transition comprises:
for ea

Assignees

• Ecole Polytechnique Fed De Lausanne (Epfl)

Inventors

• Turetken Engin
• Fua Pascal
• Fleuret Francois
• Wang Xinchao

Classifications

• G06V10/84

  using probabilistic graphical models from image or video features, e.g. Markov models or Bayesian networks · CPC title

• G06F18/29

  Graphical models, e.g. Bayesian networks · CPC title

• G06K2009/3291

  Physics · mapped topic

• G06T2207/30232

  Surveillance · CPC title

• G06T2207/30236

  Traffic on road, railway or crossing · CPC title