Systems and Methods for Capturing Images and Annotating the Captured Images with Information

Patent metadata

Field	Value
Publication number	US-2016167226-A1
Application number	US-201414572712-A
Country	US
Kind code	A1
Filing date	Dec 16, 2014
Priority date	Dec 16, 2014
Publication date	Jun 16, 2016
Grant date	—

Abstract

The present teachings provide an autonomous mobile robot that includes a drive configured to maneuver the robot over a ground surface within an operating environment; a camera mounted on the robot having a field of view including the floor adjacent the mobile robot in the drive direction of the mobile robot; a frame buffer that stores image frames obtained by the camera while the mobile robot is driving; and a memory device configured to store a learned data set of a plurality of descriptors corresponding to pixel patches in image frames corresponding to portions of the operating environment and determined by mobile robot sensor events.

First claim

What is claimed is:

1 . A method for training a classifier of a mobile robot, the method comprising:
obtaining a plurality of image frames along a drive direction of the mobile robot,
the plurality of image frames comprising a base image frame corresponding to an initial pose of the mobile robot and subsequent image frames obtained at intervals during forward travel of the mobile robot,
the mobile robot having a forward facing camera mounted thereon for obtaining the image frames,
the camera having a field of view including the floor in front of the robot, and
the robot having a memory device configured to store a learned data set of a plurality of descriptors determined by mobile robot events;
assuming that a location is traversable floor, wherein the mobile robot is configured to detect traversable floor and non-traversable non-floor with one or more sensors mounted on the mobile robot; determining that the location is non-floor based on a robot sensor event at the location; retrieving from a frame buffer an image frame obtained immediately prior to the sensor event; generating a floor descriptor corresponding to the characteristics of the floor at the bottom of the image frame captured by the camera immediately prior to the sensor event; generating a non-floor descriptor corresponding to characteristics of the non-floor at the top of the image frame captured by the camera immediately prior to the sensor event; and storing the floor descriptor and the non-floor descriptor in the learned data set.

2 . The method of claim 1 , wherein the sensor event is detection of a collision and determining that the mobile robot has collided with the obstacle includes:
receiving a bumper signal indicating a collision from a bumper sensor of the mobile robot; and verifying that the mobile robot was traveling straight prior to the collision for at least one second.

3 . The method of claim 1 , wherein the location corresponds to a patch of pixels and the one or more characteristics of the patch of pixels includes color and/or texture.

4 . The method of claim 1 , wherein the learned dataset is continuously updated and the oldest descriptors are replaced by new descriptors.

5 . The method of claim 1 , wherein the learned dataset is stored in memory and remains accessible by the robot between runs.

6 . The method of claim 1 , wherein the learned dataset is unpopulated at the start of each new run and the classifier trains the dataset with descriptors over the run of the mobile robot.

7 . An autonomous mobile robot comprising:
a drive configured to maneuver the robot over a ground surface within an operating environment; a camera mounted on the robot having a field of view including the floor adjacent the mobile robot in the drive direction of the mobile robot; a frame buffer that stores image frames obtained by the camera while the mobile robot is driving; and a memory device configured to store a learned data set of a plurality of descriptors corresponding to pixel patches in image frames corresponding to portions of the operating environment and determined by mobile robot sensor events.

8 . The mobile robot of claim 7 , further comprising one or more processors executing a training process for a classifier of the learned data set, the process comprising:
determining based on one or more sensor events that the mobile robot collided with an obstacle; retrieving a pre-collision frame from the frame buffer; identifying a lower portion of the pre-collision frame and an upper portion of the pre-collision frame; generating a first descriptor corresponding to the ground surface that the mobile robot is traveling on based on the lower portion; generate a second descriptor corresponding to at least part of the obstacle observed in the pre-collision frame based on the upper portion; and
storing the first descriptor and the second descriptor in the learned data set.

9 . An autonomous mobile robot comprising:
a drive configured to maneuver to mobile robot over a ground surface within an operating environment; a camera mounted on the mobile robot having a field of view including the floor adjacent the mobile robot in the drive direction of the mobile robot; a frame buffer that stores image frames obtained by the camera while the mobile robot is driving; and a memory device configured to store a learned data set of a plurality of descriptors corresponding to pixel patches in image frames corresponding to portions of the operating environment and determined by mobile robot sensor events; one or more processors executing a training process for a classifier of the learned data set, the process comprising: assuming that a portion of the base image frame is traversable floor, wherein the mobile robot is configured to detect traversable floor and non-traversable non-floor with one or more sensors mounted on the mobile robot; determining whether the mobile robot has traversed a threshold distance in a same direction since obtaining the base image frame; identifying an upper portion and a lower portion of the base image frame; identifying a section of the lower portion of the base image frame corresponding to a current pose of the mobile robot, the section being an area of a ground surface depicted in the lower portion of the base image frame at a depth corresponding to a drive distance traversed by the mobile robot from the initial pose to the current pose; generating a floor descriptor of the section; and storing the floor descriptor in the learned data set.

10 . The mobile robot of claim 9 , wherein the process further comprises not detecting a sensor event of a collision with an obstacle while traversing the threshold distance.

11 . The mobile robot of claim 9 , wherein the process further comprises:
detecting a new heading of the mobile robot; and obtaining a plurality of image frames along the new heading of the mobile robot, the plurality of image frames comprising a new base image frame corresponding to a new initial pose of the mobile robot at the new heading.

12 . The mobile robot of claim 9 , wherein determining whether the mobile robot has traversed a threshold distance in the same direction since obtaining the base image without detecting a sensor event comprises determining a distance between the initial pose of the mobile robot and a current pose of the mobile robot.

13 . The mobile robot of claim 9 , wherein the process further comprises generating a non-floor descriptor corresponding to characteristics of the non-floor within the upper portion of the base image frame.

14 . The mobile robot of claim 9 , wherein identifying the upper portion and the lower portion of the base image frame comprises identifying a horizon within the base image.

15 . A method for training a classifier of a mobile robot, the method comprising:
obtaining a plurality of image frames along a drive direction of the mobile robot,
the plurality of image frames comprising a base image frame corresponding to an initial pose of the mobile robot and subsequent image frames obtained at intervals during forward travel of the mobile robot,
the mobile robot having a forward facing camera mounted thereon for obtaining the image frames,
the camera having a field of view including the floor in front of the robot, and
the robot having a memory device configured to store a learned data set of a plurality of descriptors determined by mobile robot events;
tracking a location as non-traversable non-floor based on the plurality of descriptors, wherein the mobile robot is configured to detect tra

Assignees

• Irobot Corp

Inventors

• Schnittman Mark S

Classifications

• G06V30/1916 Primary

  Validation; Performance evaluation · CPC title

• G06V20/20

  in augmented reality scenes · CPC title

• G06T7/55

  from multiple images · CPC title

• B25J5/00

  Manipulators mounted on wheels or on carriages (B25J1/00 takes precedence; programme-controlled manipulators B25J9/00 {; vehicle aspects B60, B62, e.g. remote-controlled steering for motor vehicles B62D1/24; control of position of vehicles G05D1/00}) · CPC title

• Y10S901/01

  Mobile robot · CPC title