Obstacle recognition method for autonomous robots

Patent metadata

Field	Value
Publication number	US-10788836-B2
Application number	US-202016832180-A
Country	US
Kind code	B2
Filing date	Mar 27, 2020
Priority date	Feb 29, 2016
Publication date	Sep 29, 2020
Grant date	Sep 29, 2020

Abstract

Provided is a method including capturing, by an image sensor disposed on a robot, images of a workspace; obtaining, by a processor of the robot or via the cloud, the captured images; comparing, by the processor of the robot or via the cloud, at least one object from the captured images to objects in an object dictionary; identifying, by the processor of the robot or via the cloud, a class to which the at least one object belongs using an object classification unit; and instructing, by the processor of the robot, the robot to execute at least one action based on the object class identified.

First claim

The invention claimed is:

1. A method for operating a robot, comprising:
capturing, by an image sensor disposed on a robot, images of a workspace;
obtaining, by a processor of the robot or via the cloud, the captured images;
comparing, by the processor of the robot or via the cloud, at least one object from the captured images to objects in an object dictionary;
identifying, by the processor of the robot or via the cloud, a class to which the at least one object belongs using an object classification unit;
instructing, by the processor of the robot, the robot to execute at least one action based on the object class identified;
capturing, by at least one sensor of the robot, movement data of the robot; and
generating, by the processor of the robot or via the cloud, a spatial representation of the workspace based on the captured images and the movement data, wherein the captured images are indicative of the position of the robot relative to objects within the workspace and the movement data is indicative of movement of the robot.

2. The method of claim 1 , wherein comparing the at least one object from the captured images to objects in an object dictionary comprises generating a feature vector and characteristics data of the at least one object from the captured images.

3. The method of claim 2 , wherein feature vector and characteristics data comprises any of edge characteristic combinations, basic shape characteristic combinations, size characteristic combinations, and color characteristic combinations.

4. The method of claim 1 , wherein comparing the at least one object with objects in the object dictionary is performed using a neural network.

5. The method of claim 1 , wherein the at least one action comprises at least one of executing an altered navigation path to avoid driving over the object identified and maneuvering around the object identified and continuing along the planned navigation path.

6. The method of claim 1 , the at least one action is based at least on real time observations.

7. The method of claim 1 , wherein the object dictionary is based on a training set in which images of a plurality of examples of the objects in the object dictionary are processed by the processor under varied lighting conditions and camera poses to extract and compile feature vector and characteristics data and associate that feature vector and characteristics data with a corresponding object.

8. The method of claim 1 , wherein the object dictionary comprises any of: cables, cords, wires, toys, jewelry, garments, socks, shoes, shoelaces, feces, liquids, keys, food items, remote controls, plastic bags, purses, backpacks, earphones, cell phones, tablets, laptops, chargers, animals, fridges, televisions, chairs, tables, light fixtures, lamps, fan fixtures, cutlery, dishware, dishwashers, microwaves, coffee makers, smoke alarms, plants, books, washing machines, dryers, watches, blood pressure monitors, blood glucose monitors, first aid items, power sources, Wi-Fi repeaters, entertainment devices, appliances, and Wi-Fi routers.

9. The method of claim 1 , further comprising:
determining, by the processor of the robot or via the cloud, distances to objects in the captured images;
identifying, by the processor of the robot or via the cloud, an opening in the workspace based on the distances to objects; and
segmenting, by the processor of the robot or via the cloud, the workspace into subareas based on at least a position of one opening in the workspace.

10. The method of claim 1 , further comprising:
identifying, by the processor of the robot or via the cloud, a particular person or pet using facial recognition techniques.

11. The method of claim 1 , wherein the at least one sensor comprises at least one of: an optical tracking sensor, an imaging sensor, an inertial measurement unit, an odometry encoder, and a gyroscope.

12. The method of claim 1 , wherein capturing movement data comprises:
capturing, by an optical tracking sensor, a plurality of images of surfaces within a field of view of the optical tracking sensor while the robot moves within the workspace;
obtaining, by the processor of the robot or via the cloud, the plurality of images;
determining, by the processor of the robot or via the cloud, linear movement of the optical tracking sensor based on the plurality of images captured, wherein linear movement of the optical tracking sensor is equivalent to linear movement of the robot; and
determining, with the processor of the robot or via the cloud, rotational movement of the robot based on the linear movement of the optical tracking sensor.

13. The method of claim 1 , wherein capturing movement data comprises:
capturing, by at least one sensor, second movement data of the robot from a previous position to a current position; and
correcting, by the processor of the robot or via the cloud, the movement data based on a translation vector of the second movement data describing movement of the robot from the previous position to the current position to account for error in the movement data caused by slippage of the robot.

14. The method of claim 1 , wherein generating the spatial representation of the workspace further comprises:
determining, by the processor of the robot or via the cloud, an overlapping area of a first image and a second image by comparing sensor readings of the first image to sensor readings of the second image, wherein:
the first image and the second image are taken from different positions, and
the sensor readings of the first image and the sensor readings of the second image comprise raw pixel intensity values;

spatially aligning, by the processor of the robot or via the cloud, sensor readings of the first image and sensor readings of the second image based on the overlapping area; and
inferring, by the processor of the robot or via the cloud, features of the workspace based on the spatially aligned sensor readings of the first image and the second image.

15. The method of claim 14 , wherein determining the overlapping area comprises:
detecting a first edge at a first position in the first image based on a derivative of pixel values in the first image;
detecting a second edge at a second position in the first image based on the derivative of pixel values in first image;
detecting a third edge in a third position in the second image based on a derivative of pixel values in the second image;
determining that the third edge is not the same edge as the second edge based on shapes of the third edge and the second edge not matching;
determining that the third edge is the same edge as the first edge based on shapes of the first edge and the third edge at least partially matching; and
determining a first translation vector that associates the first image with the second image.

16. The method of claim 1 , further comprising:
determining, by the processor of the robot or via the cloud, depths to objects in the captured images; and
associating, by the processor of the robot or via the cloud, consecutive images captured in intervals with each other based on respective values indicating respective angular displacements of corresponding depths in respective frames of reference corresponding to respective fields of view.

17. The method of claim 1 , further comprising:
creating, by the processor of the robot or via the cloud, a first iteration of the spatial representation of the workspace, wherein:
the first iteration of the spatial representation is based at least on sensor data sensed by at least one sensor in a first position and orient

Assignees

• Ebrahimi Afrouzi Ali
• Mehrnia Soroush
• Fath Lukas
• Ai Inc

Inventors

• Ebrahimi Afrouzi Ali
• Mehrnia Soroush
• Fath Lukas

Classifications

• G06V20/10

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• B25J9/1697

  Vision controlled systems · CPC title

• B25J9/1676

  Avoiding collision or forbidden zones · CPC title