Obstacle recognition method for autonomous robots

Patent metadata

Field	Value
Publication number	US-11467587-B2
Application number	US-202016995480-A
Country	US
Kind code	B2
Filing date	Aug 17, 2020
Priority date	Feb 29, 2016
Publication date	Oct 11, 2022
Grant date	Oct 11, 2022

Abstract

Provided is a method for operating a robot, including capturing images of a workspace, comparing at least one object from the captured images to objects in an object dictionary, identifying a class to which the at least one object belongs using an object classification unit, instructing the robot to execute at least one action based on the object class identified, capturing movement data of the robot, and generating a planar representation of the workspace based on the captured images and the movement data, wherein the captured images indicate a position of the robot relative to objects within the workspace and the movement data indicates movement of the robot.

First claim

The invention claimed is:

1. A method for operating a robot, comprising:
capturing, by at least one image sensor disposed on a robot, images of a workspace;
obtaining, by a processor of the robot or via the cloud, the captured images;
comparing, by the processor of the robot or via the cloud, at least one object from the captured images to objects in an object dictionary;
identifying, by the processor of the robot or via the cloud, a class to which the at least one object belongs using an object classification unit;
instructing, by the processor of the robot, the robot to execute at least one action based on the object class identified;
capturing, by at least one sensor of the robot, movement data of the robot; and
generating, by the processor of the robot or via the cloud, a spatial representation of the workspace based on the captured images and the movement data, wherein the captured images indicate a position of the robot relative to objects within the workspace and the movement data indicates movement of the robot.

2. The method of claim 1 , wherein a planar representation or a combination of a spatial and planar representation is generated instead of the spatial representation.

3. The method of claim 1 , wherein the robot executes the at least one action in at least one of: a current work session and future work sessions.

4. The method of claim 1 , wherein comparing the at least one object from the captured images to objects in an object dictionary comprises generating a feature vector and characteristics data of the at least one object from the captured images.

5. The method of claim 4 , wherein feature vector and characteristics data comprises any of edge characteristic combinations, basic shape characteristic combinations, size characteristic combinations, and color characteristic combinations.

6. The method of claim 1 , wherein comparing the at least one object with objects in the object dictionary is performed using a neural network.

7. The method of claim 1 , wherein the at least one action comprises at least one of executing an altered navigation path to avoid driving over the object identified and maneuvering around the object identified and continuing along the planned navigation path.

8. The method of claim 1 , the at least one action is based at least on real time observations.

9. The method of claim 1 , wherein the object dictionary is based on a training set in which images of a plurality of examples of the objects in the object dictionary are processed by the processor under varied lighting conditions and camera poses to extract and compile feature vector and characteristics data and associate that feature vector and characteristics data with a corresponding object.

10. The method of claim 1 , wherein the object dictionary comprises any of: cables, cords, wires, toys, jewelry, garments, socks, shoes, shoelaces, feces, liquids, keys, food items, remote controls, plastic bags, purses, backpacks, earphones, cell phones, tablets, laptops, chargers, animals, fridges, televisions, chairs, tables, light fixtures, lamps, fan fixtures, cutlery, dishware, dishwashers, microwaves, coffee makers, smoke alarms, plants, books, washing machines, dryers, watches, blood pressure monitors, blood glucose monitors, first aid items, power sources, Wi-Fi repeaters, entertainment devices, appliances, and Wi-Fi routers.

11. The method of claim 1 , further comprising:
determining, by the processor of the robot or via the cloud, distances to objects in the captured images;
identifying, by the processor of the robot or via the cloud, an opening in the workspace based on the distances to objects; and
segmenting, by the processor of the robot or via the cloud, the workspace into subareas based on at least a position of one opening in the workspace.

12. The method of claim 1 , further comprising:
determining, by the processor of the robot or via the cloud, distances to objects in the captured images; and
instructing, by the processor of the robot, the robot to avoid the objects in at least one of: a current work session and future work sessions.

13. The method of claim 1 , wherein the at least one sensor comprises at least one of: an optical tracking sensor, an imaging sensor, an inertial measurement unit, an odometry encoder, a LIDAR sensor, a depth camera, and a gyroscope.

14. The method of claim 1 , wherein capturing movement data comprises:
capturing, by an optical tracking sensor, a plurality of images of surfaces within a field of view of the optical tracking sensor while the robot moves within the workspace;
obtaining, by the processor of the robot or via the cloud, the plurality of images;
determining, by the processor of the robot or via the cloud, linear movement of the optical tracking sensor based on the plurality of images captured, wherein linear movement of the optical tracking sensor is equivalent to linear movement of the robot; and
determining, with the processor of the robot or via the cloud, rotational movement of the robot based on the linear movement of the optical tracking sensor.

15. The method of claim 1 , wherein capturing movement data comprises:
capturing, by an odometry encoder, odometry data;
determining, with the processor of the robot or via the cloud, a number of wheel rotations based on the odometry data; and
determining, with the processor of the robot or via the cloud, movement of the robot based on the number of wheel rotations.

16. The method of claim 1 , wherein capturing movement data comprises:
capturing, by an inertial measurement unit, inertial measurement unit data; and
determining, with the processor of the robot or via the cloud, movement of the robot based on the inertial data.

17. The method of claim 1 , wherein capturing movement data comprises:
capturing, by a 360 degrees rotating LIDAR, distance data; and
determining, with the processor of the robot or via the cloud, movement of the robot based on the changes in the distance data.

18. The method of claim 1 , wherein capturing movement data comprises:
capturing, by a depth camera, distance data; and
determining, with the processor of the robot or via the cloud, movement of the robot based on the changes in the distance data.

19. The method of claim 1 , wherein capturing movement data comprises:
capturing, by at least one sensor, second movement data of the robot from a previous position to a current position; and
correcting, by the processor of the robot or via the cloud, the movement data based on a translation vector of the second movement data describing movement of the robot from the previous position to the current position to account for error in the movement data caused by slippage of the robot.

20. The method of claim 1 , wherein generating the spatial representation of the workspace further comprises:
determining, by the processor of the robot or via the cloud, an overlapping area of a first image and a second image by comparing sensor readings of the first image to sensor readings of the second image, wherein:
the first image and the second image are taken from different positions, and
the sensor readings of the first image and the sensor readings of the second image comprise raw pixel intensity values;

spatially aligning, by the processor of the robot or via the cloud, sensor readings of the first image and sensor readings of the second image based on the overlapping area; and
inferring, by the processor of the robot or via the cloud, features of the workspace based on th

Assignees

• Ebrahimi Afrouzi Ali
• Mehrnia Soroush
• Fath Lukas
• Ai Inc

Inventors

• Ebrahimi Afrouzi Ali
• Mehrnia Soroush
• Fath Lukas

Classifications

• G06V20/10

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• A47L9/0686

  Nozzles with cleaning cloths, e.g. using disposal fabrics for covering the nozzle · CPC title

• A47L9/1409

  Rigid filtering receptacles (dust collecting receptacles for cyclones A47L9/1683) · CPC title