Object detection and tracking

Patent metadata

Field	Value
Publication number	US-9729865-B1
Application number	US-201414307492-A
Country	US
Kind code	B1
Filing date	Jun 18, 2014
Priority date	Jun 18, 2014
Publication date	Aug 8, 2017
Grant date	Aug 8, 2017

Abstract

Various embodiments enable a primary user to be identified and tracked using stereo association and multiple tracking algorithms. For example, a face detection algorithm can be run on each image captured by a respective camera independently. Stereo association can be performed to match faces between cameras. If the faces are matched and a primary user is determined, a face pair is created and used as the first data point in memory for initializing object tracking. Further, features of a user's face can be extracted and the change in position of these features between images can determine what tracking method will be used for that particular frame.

First claim

What is claimed is:

1. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause a computing device to:
generate first right image data during a first period of time with a right camera;
generate first left image data during the first period of time with a left camera, the right camera and the left camera having at least partially overlapping fields of view;
detect, using a face detection algorithm, a first right representation of a face in the first right image data, the face detection algorithm returning a first right bounding box for the face in the first right image data;
detect, using the face detection algorithm, a first left representation of the face in the first left image data, the face detection algorithm returning a first left bounding box for the face in the first left image data;
detect, using a feature extraction algorithm, a set of features of the face in the first right image data by analyzing the first right image data within the first right bounding box;
detect, using the feature extraction algorithm, the set of features of the face in the first left image data by analyzing the first left image data within the first left bounding box;
determine a first right position for a point relative to the set of features in the first right image data;
determine a first left position for the point relative to the set of features in the first left image data;
generate second right image data during a second period of time with the right camera;
generate second left image data during the second period of time with the left camera;
detect, using the face detection algorithm, a second right representation of the face in the second right image data, the face detection algorithm returning a second right bounding box for the face in the second right image data;
detect, using the face detection algorithm, a second left representation of the face in the second left image data, the face detection algorithm returning a second left bounding box for the face in the second left image data;
detect, using the feature extraction algorithm, the set of features of the face in the second right image data by analyzing the second right image data within the second right bounding box;
detect, using the feature extraction algorithm, the set of features of the face in the first left image data by analyzing the second left image data within the second left bounding box;
determine a second right position for the point relative to the set of features in the second right image data;
determine a second left position for the point relative to the set of features in the second left image data;
determine a right two-dimensional (2D) change in position of the point relative to the set of features between the first right image data and the second right image data;
determine a left 2D change in position of the point relative to the set of features between the first left image data and the second left image data;
determine, using a feature tracking algorithm and based at least in part on the right 2D change in position, a right 2D output for the point relative to the set of features;
determine, using the feature tracking algorithm and based at least in part on the left 2D change in position, a left 2D output for the point relative to the set of features;
determine stereo disparity of the point relative to the set of features between the right 2D output and the left 2D output in the at least partially overlapping fields of view of the right camera and the left camera;
determine a z-depth for the point relative to the set of features of the face using the stereo disparity and calibration information for the right camera and the left camera to determine a three-dimensional (3D) position for the point relative to the set of features;
generate third right image data during a third period of time with the right camera;
generate third left image data during the third period of time with the left camera;
determine that a third right representation of the face is detected in the third right image data;
determine that a third left representation of the face is not detected in the third left image data;
generate a template of the face using information for the face from the third right image data; and
use the template to detect the face in the third left image data.

2. The non-transitory computer-readable storage medium of claim 1 , wherein the instructions that, when executed by the processor, further cause the computing device to:
determine the right 2D change in position being less than a first threshold;
determine the left 2D change in position being less than a first threshold;
determine, using the feature tracking algorithm, the first right position for the point relative to the set of features from the first right image data as the right 2D output; and
determine, using the feature tracking algorithm, the first left position for the point relative to the set of features from the first left image data as the left 2D output.

3. The non-transitory computer-readable storage medium of claim 1 , wherein the instructions that, when executed by the processor, further cause the computing device to:
determine the right 2D change in position being between a first threshold and a second threshold;
track, using a single point tracking algorithm, the point relative to the set of features of the face between the first right position and the second right position to determine the right 2D output;
determine the left 2D change in position being between a first threshold and a second threshold; and
track, using the single point tracking algorithm, the point relative to the set of features of the face between the first left position and the second left position to determine the left 2D output.

4. The non-transitory computer-readable storage medium of claim 1 , wherein the instructions that, when executed by the processor, further cause the computing device to:
determine the right 2D change in position being greater than a second threshold;
determine the left 2D change in position being greater than the second threshold;
determine, using the feature tracking algorithm, the second right position for the point relative to the set of features from the second right image data as the right 2D output; and
determine, using the feature tracking algorithm, the second left position for the point relative to the set of features from the second left image data as the left 2D output.

5. A computer-implemented method, comprising:
detecting an object in first stereo image data generated during a first period of time by two or more image capturing elements with overlapping fields of view;
associating a bounding box with the object;
analyzing the first stereo image data within the bounding box to determine one or more points relative to features of the object;
tracking the one or more points in the first stereo image data to determine a two-dimensional (2D) position of the one or more points;
determining stereo disparity for the 2D position of at least one of the one or more points in the first stereo image data;
determining a three-dimensional (3D) position of the one or more points relative to the features of the object based at least in part on the stereo disparity and information associated with the two or more image capturing elements;
generating third image data during a second period of time;
generating fourth image data during the second period of time, the third image data and the fourth image data being parts of a stereo image pair;
determining that the object was detected in the third image data;
determining that the object was not detected in the fourth image data;
generating a template of the object using image information for the object from the third im

Assignees

• Amazon Tech Inc

Inventors

• Kuo Cheng-Hao
• Thomas Jim Oommen
• Ma Tianyang
• Mangiat Stephen Vincent
• Mehta Sisil Sanjeev
• Tyagi Ambrish
• Agrawal Amit Kumar
• Fu Kah Kuen
• Ramaswamy Sharadh

Classifications

• H04N5/76

  Television signal recording · CPC title

• H04N13/0271

  Electricity · mapped topic

• H04N2013/0081

  Depth or disparity estimation from stereoscopic image signals · CPC title

• G06K9/00268

  Physics · mapped topic

• H04N13/0468 Primary

  Electricity · mapped topic