Tracking and following people with a mobile robotic device

Patent metadata

Field	Value
Publication number	US-9321173-B2
Application number	US-201213531401-A
Country	US
Kind code	B2
Filing date	Jun 22, 2012
Priority date	Jun 22, 2012
Publication date	Apr 26, 2016
Grant date	Apr 26, 2016

Abstract

Tracking and following technique embodiments are presented that are generally employed to track and follow a person using a mobile robotic device having a color video camera and a depth video camera. A computer associated with the mobile robotic device is used to perform various actions. Namely, in a tracking mode, a face detection method and the output from the color video camera is used to detect potential persons in an environment. In addition, a motion detection method and the output from the depth video camera is also used to detect potential persons in the environment. Detection results obtained using the face and motion detection methods are then fused and used to determine the location of one or more persons in the environment. Then, in a following mode, a mobile robotic device following method is used to follow a person whose location was determined in the tracking mode.

First claim

Wherefore, what is claimed is:

1. A computer-implemented process for tracking and following a person using a mobile robotic device having a color video camera and a depth video camera, comprising:
using a computer associated with the mobile robotic device to perform the following process actions:
in a tracking mode,
using a face detection method and the output from the color video camera to detect the locations of one or more persons potentially in an environment,
using a motion detection method and the output from the depth video camera to detect the locations of one or more persons potentially in the environment,
fusing detection results obtained using the face and motion detection methods, said fusing comprising, for each person potentially in the environment whose location was detected using the face detection method, ascertaining if the location of the person corresponds to the location of a person potentially in the environment detected using the motion detection method, and deeming that location to be the location of one person in the environment, and
using the fused detection results to establish the locations of one or more persons in the environment; and

in a following mode, employing a mobile robotic device following method to follow a person whose location was established in the tracking mode.

2. The process of claim 1 , further comprising an action in the tracking mode of tracking the location of each person in the environment over time.

3. The process of claim 2 , wherein the action of tracking the location of each person in the environment over time, comprises periodically repeating the actions of,
using the face detection method and the output from the color video camera to detect potential persons in an environment,
using the motion detection method and the output from the depth video camera to detect potential persons in the environment,
fusing detection results obtained using the face and motion detection methods, and
using the fused detection results to determine the location of one or more persons in the environment.

4. The process of claim 1 , further comprising an action in the tracking mode of attempting to ascertain the identity of one or more persons in the environment.

5. The process of claim 1 , wherein the following mode action of employing the mobile robotic device following method to follow said person, comprises an action of initiating the following mode in response to a request from one of the persons detected in the environment to follow that person.

6. The process of claim 1 , wherein the following mode action of employing the mobile robotic device following method to follow said person, comprises an actions of:
repositioning said color and depth video cameras, from a view comprising the face of the person, to a view comprising the torso of the person and the path ahead of the mobile robotic device such that obstacles lying in the path of the mobile robotic device encountered while following the person are in the view of the color and depth video cameras;
employing depth frames captured with the depth video camera during said repositioning to establish torso depth values for each pixel of the frames representing the torso of the person;
once the repositioning of the color and depth video cameras is complete,
identifying pixels in the last-captured image frame of the color video camera that correspond in location within the scene depicted to pixels in the last-captured depth frame of the depth video camera which represent the torso of the person, and
computing a first color histogram of the person's torso using the identified pixels in the last-captured image frame of the color video camera,
associating the color histogram with the person, and
storing the color histogram in a memory associated with the mobile robotic device; and

whenever the person changes location, using said mobile robotic device following method to follow the person, while at the same time avoiding obstacles lying in the path taken to follow the person and tracking the location of the person as the person moves.

7. The process of claim 6 , wherein the process action of using said mobile robotic device following method to follow the person, comprises an action of taking a path to follow the person that maintains the ability to track the location of the person despite obstacles that have the potential to block a line of sight between the mobile robotic device and the person.

8. The process of claim 6 , wherein the process action of tracking the location of the person as the person moves, comprises the actions of:
(a) establishing a bounding box around the pixels in the depth frame from the depth video camera representing the torso of the person that was employed in identifying pixels in the frame from the color video camera used to compute the first color histogram of the person's torso;
(b) establishing an initial confidence level representing the likelihood that the bounding box contains pixels representing the torso of the person;
(c) inputting a newly-captured depth frame as the current depth frame;
(d) identifying pixels in the current depth frame exhibiting the same depth values within a prescribed tolerance range and which form a human torso-shaped blob;
(e) determining if the identified pixels meet conditions comprising,
being contained with a region that is a prescribed amount larger than the last-established bounding box and which is centered in a location in the current depth frame that corresponds to the location where the last-established bounding box was centered in the depth frame it was established in, and
exhibiting depth values that are within a prescribed tolerance of the depth values exhibited by pixels in the depth frame captured immediately preceding the current depth frame that represented the torso of the person;

(f) whenever the identified pixels meet the conditions,
computing an additional color histogram of the person's torso using pixels in the last-captured image frame of the color video camera that correspond in location within the scene to the identified pixels of the current depth camera frame which met said conditions,
determining if the last-computed additional color histogram differs from each previously-stored color histogram to a prescribed degree, and if so associating the last-computed additional color histogram with the person and storing the last-computed additional color histogram in said memory associated with the mobile robotic device,
increasing the current confidence level by a prescribed amount up to a maximum value, and
establishing a new bounding box around the pixels in the current depth frame representing the torso of the person, and
repeating actions (c) through (f), as appropriate;

(g) whenever the identified pixels do not meet the conditions,
decreasing the current confidence level by a prescribed amount,
determining if the decreased confidence level falls below a prescribed minimum acceptable confidence level,
whenever the decreased confidence level does not fall below the prescribed minimum acceptable confidence level, repeating actions (c) through (g), as appropriate.

9. The process of claim 8 , wherein whenever it is determined the decreased confidence level does fall below the prescribed minimum acceptable confidence level, performing the action of:
suspending the following mode; and
attempting to reacquire the person.

10. The process of claim 9 , wherein the process action of attempting to reacquire the person, comprises the actions of:
attempting to find an area in the last-captured image frame of the color video camera that a color histogram computed for the area matches to a prescribed degree one of t

Assignees

• Kikkeri Harshavardhana Narayana
• Koenig Michael F
• Microsoft Technology Licensing Llc

Inventors

• Kikkeri Harshavardhana Narayana
• Koenig Michael F

Classifications

• G05D1/0242

  using non-visible light signals, e.g. IR or UV signals · CPC title

• B25J9/104 Primary

  with cables, chains or ribbons · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T2207/30201

  Face · CPC title

• G06T7/20

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title