Apparatus and method for object tracking

Patent metadata

Field	Value
Publication number	US-9613273-B2
Application number	US-201514716661-A
Country	US
Kind code	B2
Filing date	May 19, 2015
Priority date	May 19, 2015
Publication date	Apr 4, 2017
Grant date	Apr 4, 2017

Abstract

An object tracking device includes a short-term processing portion and a long short-term processing portion that are implemented by circuitry and work in a collaborative manner to track an object. The short-term processing portion includes a filter that tracks the object based on short-term memory and spatiotemporal consistency. The long short-term processing portion performs key-point matching-tracking and estimation based on a key-point database in order to track the object. A controller determines an output of the object tracking device based on the processing conducted by the short-term and long short-term processing portions of the tracking device, respectively.

First claim

What is claimed is:

1. An object tracking device comprising:
a first tracker implemented by circuitry, the circuitry configured to:
determine and extract features, of a candidate image patch from an input image frame,
evaluate the candidate image patch and cyclically shifted image patches of the candidate image patch, to determine a location of a tracked object;
resize and extract features from a predetermined number of image patches surrounding the determined location to obtain a first instance of the tracked object;

a second tracker implemented by circuitry, the circuitry configured to:
detect a plurality of key-points from the input image frame,
classify, based on a key-point database, each detected key-point as one of a matched target key-point, a matched background key-point, and an unmatched key-point,
compute, based on consecutive input image frames, an active set of key-points,
estimate, based on the computed active set of key-points and the matched target key-points, a set of inlier key-points included in a target bounding box that corresponds to a second instance of the tracked object; and

a controller implemented by circuitry, the circuitry being configured to generate a tracked object based on a comparison of the first instance of the tracked object and the second instance of the tracked object.

2. The object tracking device of claim 1 , wherein the determined location of the tracked object has a highest filtering score, and the first instance of the tracked object is included in an image patch having a highest filter response.

3. The object tracking device of claim 2 , wherein the circuitry is further configured to:
update the filter coefficients and an object template in an interpolating fashion at a predetermined update rate.

4. The object tracking device of claim 1 , wherein the extracted features are based on a histogram of oriented gradient descriptors and color attributes, and the detected key-points are characterized by scale-invariant-feature-transform descriptors.

5. The object tracking device of claim 1 , wherein the circuitry of the controller is configured to classify the detected key-point as a matched target key-point based on a matching confidence of the detected key-point and a first neighbor of the detected keypoint that belongs to the key-point database being above a first threshold.

6. The object tracking device of claim 5 , wherein the circuitry of the controller is further configured to classify the detected key-point as a matched target key-point based on a ratio of distance parameter being below a second threshold, the ratio of distance parameter being a ratio of the Euclidean distance between the detected key-point and the first nearest neighbor to the Euclidean distance between the detected key-point and the second nearest neighbor.

7. The object tracking device of claim 5 , wherein the matching confidence is a cosine similarity between the detected key-point and the first nearest neighbor of the detected key-point.

8. The object tracking device of claim 1 , wherein the active set of key-points includes key-points whose displacement between the consecutive input image frames is less than a predetermined threshold.

9. The object tracking device of claim 1 , wherein the circuitry of the controller is further configured to
compute an intersection-over-union (IOU) of the first instance of the tracked object and the second instance of the tracked object; and
determine whether the first instance of the tracked object and the second instance of the tracked object are consistent based on the computed IOU and a predetermined consistency threshold.

10. The object tracking device of claim 1 , wherein the circuitry of the controller is further configured to:
update the active set of key-points for each input image frame by determining a redundant set of key-points within the active set of key-points, the redundant set of key-points being computed based on quantization ID's of the key-points; and
update the key-point database based on a number of inlier key-points being greater than a predetermined threshold and a number of matched background key-points lying within the target bounding box being zero.

11. The object tracking device of claim 10 , wherein a timespan corresponding to an amount of time the key-points are maintained in the key-point database is exponentially distributed based on a relative strength parameter of the key-point database, a period of a forgetting parameter, and a scale of a timespan parameter.

12. A method of object tracking comprising:
determining, and extracting features of a candidate image patch, from an input image frame;
evaluating by circuitry, the candidate image patch and cyclically shifted image patches of the candidate image patch, to determine a location of a tracked object;
resizing, and extracting features, from a predetermined number of image patches surrounding the determined location to obtain a first instance of the tracked object;
detecting a plurality of key-points from the input image frame;
classifying, based on a key-point database, each detected key-point as one of a matched target key-point, a matched background key-point, and an unmatched key-point,
computing by circuitry, based on consecutive input image frames, an active set of key-points;
estimating, based on the computed active set of key-points and the matched target key-points, a set of inlier key-points included in a target bounding box that corresponds to a second instance of the tracked object; and
generating by circuitry, a tracked object based on a comparison of the first instance of the tracked object and the second instance of the tracked object.

13. The method of object tracking of claim 12 , wherein the determined location of the tracked object has a highest filtering score, and the first instance of the tracked object is included in an image patch having a highest filter response.

14. The method of object tracking of claim 12 , further comprising:
updating the filter coefficients and an object template in an interpolating fashion at a predetermined update rate.

15. The method of object tracking of claim 12 , wherein the circuitry is configured to classify the detected key-point as a matched target key-point based on a matching confidence of the detected key-point and a first neighbor of the detected keypoint that belongs to the key-point database being above a first threshold.

16. The method of object tracking of claim 15 , wherein the circuitry is further configured to classify the detected key-point as a matched target key-point based on a ratio of distance parameter being below a second threshold, the ratio of distance parameter being a ratio of the Euclidean distance between the detected key-point and the first nearest neighbor to the Euclidean distance between the detected key-point and the second nearest neighbor.

17. The method of object tracking of claim 12 , further comprising:
computing by circuitry, an intersection-over-union (IOU) of the first instance of the tracked object and the second instance of the tracked object; and
determining whether the first instance of the tracked object and the second instance of the tracked object are consistent based on the computed IOU and a predetermined consistency threshold.

18. A non-transitory computer readable medium having stored thereon a program that when executed by a computer causes the computer to execute a method to track an object, the method comprising:
determining, and extracting features of a candidate image patch, from an

Assignees

• Toyota Motor Eng & Mfg North America Inc
• Univ Of Tech Sydney

Inventors

• Hong Zhibin
• Mei Xue
• Chen Zhe
• Wang Chaohui
• Prokhorov Danil
• Tao Dacheng

Classifications

• G06V10/40 Primary

  Extraction of image or video features · CPC title

• G06F18/24

  Classification techniques · CPC title

• G06K9/00

  Physics · mapped topic

• G06K9/52

  Physics · mapped topic

• G06T3/40

  Scaling of whole images or parts thereof, e.g. expanding or contracting · CPC title