Mobile cleaning robot artificial intelligence for situational awareness

Patent metadata

Field	Value
Publication number	US-10878294-B2
Application number	US-201815863591-A
Country	US
Kind code	B2
Filing date	Jan 5, 2018
Priority date	Jan 5, 2018
Publication date	Dec 29, 2020
Grant date	Dec 29, 2020

Abstract

A mobile cleaning robot includes a cleaning head configured to clean a floor surface in an environment, and at least one camera having a field of view that extends above the floor surface. The at least one camera is configured to capture images that include portions of the environment above the floor surface. The robot includes a recognition module is configured to recognize objects in the environment based on the images captured by the at least one camera, in which the recognition module is trained at least in part using the images captured by the at least one camera. The robot includes a storage device is configured to store a map of the environment. The robot includes a control module configured to control the mobile cleaning robot to navigate in the environment using the map and operate the cleaning head to perform cleaning tasks taking into account of the objects recognized by the recognition module.

First claim

What is claimed is:

1. A mobile cleaning robot comprising:
a cleaning head configured to clean a floor surface in an environment;
at least one camera having a field of view that extends above the floor surface, in which the at least one camera is configured to capture images that include portions of the environment above the floor surface;
a recognition module configured to recognize a type of each of one or more objects in the environment based on the images captured by the at least one camera, in which the recognition module is trained at least in part using the images captured by the at least one camera;
a storage device configured to store a map of the environment; and
a control module configured to control the mobile cleaning robot to navigate in the environment using the map and to select a cleaning mode for operation of the cleaning head to perform cleaning tasks in the vicinity of a particular object, the selecting including (i) recognizing, with the recognition module, the particular object as a particular type of object, and (ii) selecting a cleaning mode from a number of cleaning modes, the selected cleaning mode corresponding to the recognized particular type of object.

2. The mobile cleaning robot of claim 1 in which the recognition module comprises a neural network that is trained at least in part using the images captured by the at least one camera.

3. The mobile cleaning robot of claim 2 in which the neural network is pre-trained to recognize objects that belong to a predetermined group of objects before the mobile cleaning robot starts to capture images of the environment at an initial time, the recognition module is configured to update the neural network using the images of the environment captured by the at least one camera, and the updated neural network is configured to recognize the objects in the environment more accurately than the pre-trained neural network.

4. The mobile cleaning robot of claim 1 in which the control module is configured to select between a deep cleaning mode or a no-touch cleaning mode.

5. The mobile cleaning robot of claim 1 in which the recognition module is configured to, for each of a plurality of objects that are barriers to movements of the robot in the environment, classify whether the object is a semi-permanent barrier that is moved infrequently or an impermanent barrier that is moved frequently based on images of the object captured over a period of time, and
the control module is configured to operate the cleaning head to perform the cleaning tasks taking into account of whether each of the objects recognized by the recognition module is a semi-permanent barrier or an impermanent barrier.

6. The mobile cleaning robot of claim 5 in which the control module updates the map to store information about a plurality of locations, wherein for each location, the map stores information about at least one of (i) whether there is a semi-permanent barrier at the location, (ii) whether there is an impermanent barrier at the location, or (iii) how frequently an impermanent barrier appears at the location.

7. The mobile cleaning robot of claim 5 in which the storage device stores information about, for each of at least some of the impermanent barriers, a user preference regarding how the mobile cleaning robot should respond when encountering the impermanent barrier.

8. The mobile cleaning robot of claim 7 in which the control module is configured to, upon encountering an impermanent barrier at a location and determining that the storage device does not have information about a user preference regarding how the mobile cleaning robot should respond when encountering the impermanent barrier, send a notice to a user to request an instruction regarding how to respond to the presence of the impermanent barrier.

9. The mobile cleaning robot of claim 8 in which the notice includes a plurality of user-selectable options including at least one of (i) an option for the robot to go to other locations and return to the location after a preset amount of time, (ii) an option to perform an extended cleaning task at the location on a next cleaning session, (iii) an option to move the impermanent barrier, or (iv) an option to select a no-touch mode for the location or a region encompassing the location, in which the no-touch mode indicates that the robot is controlled to not bump into any object above ground level at the location or the region encompassing the location.

10. The mobile cleaning robot of claim 1 in which the recognition module is configured to recognize presence of one or more individuals in the environment, and the control module is configured to control the mobile cleaning robot to navigate in the environment and perform the cleaning tasks also taking into account the presence of the one or more individuals.

11. The mobile cleaning robot of claim 10 in which the control module is configured to perform at least one of:
(i) upon detecting that a door of a room is closed, and that no presence of individuals has been detected in the environment since a previous cleaning session, skip cleaning the room until a next cleaning session, or
(ii) upon detecting that a door of a room is closed at a first time point, and that presence of one or more individuals has been detected in the environment since the first time point, return to check whether the door of the room is open at a second time point that is a period of time after the first time point, and upon determining that the door is open, performing a cleaning task in the room.

12. The mobile cleaning robot of claim 1 in which the recognition module is configured to recognize doors in the environment, and the control module is configured to schedule the cleaning tasks based on an open or closed state of each of the doors encountered by the mobile cleaning robot.

13. The mobile cleaning robot of claim 12 in which the control module is configured to:
upon recognizing that a door is closed, control the mobile cleaning robot to navigate to another area to perform a cleaning task at the other area,
after a period of time, return to check whether the door is open, and upon determining that the door is open, enter an area behind a doorway associated with the door and perform a cleaning task in the area behind the doorway.

14. The mobile cleaning robot of claim 12 in which the control module is configured to, upon recognizing that a door is closed, perform at least one of (i) send a message to request assistance in opening the door, or (ii) send a message indicating that an area behind the door was not cleaned because the door was closed.

15. The mobile cleaning robot of claim 12 in which the control module is configured to, upon recognizing that a door is open, maintain a distance from the door to avoid bumping into the door when navigating in a vicinity of the door.

16. The mobile cleaning robot of claim 1 , comprising a learning module configured to determine a foot traffic pattern in the environment based on the images captured by the at least one camera over a period of time.

17. The mobile cleaning robot of claim 16 in which the control module is configured to schedule a cleaning task to be performed at an area taking into account of the foot traffic pattern at the area.

18. The mobile cleaning robot of claim 17 in which the control module is configured to schedule a cleaning task to be performed at the area during a time period when there is less foot traffic at the area as compared to other time periods.

19. The mobile cleaning robot of claim 1

Assignees

• Irobot Corp

Inventors

• Jones Christopher V
• Hall Gary Ellis
• Baron Steven J
• Hild Brent
• Zickler Stefan
• Sinnigen John

Classifications

• G06V30/19173

  Classification techniques · CPC title

• G06V10/82

  using neural networks · CPC title

• G06F18/2431

  Multiple classes · CPC title

• G06F18/24143

  Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN] · CPC title

• G05D1/243

  Means capturing signals occurring naturally from the environment, e.g. ambient optical, acoustic, gravitational or magnetic signals (using passive navigation aids external to the vehicle G05D1/244; using signals from positioning sensors located off-board the vehicle G05D1/249) · CPC title