Method and apparatus for combining data to construct a floor plan

Patent metadata

Field	Value
Publication number	US-11348269-B1
Application number	US-202016920328-A
Country	US
Kind code	B1
Filing date	Jul 2, 2020
Priority date	Jul 27, 2017
Publication date	May 31, 2022
Grant date	May 31, 2022

Abstract

A robot for perceiving a spatial representation of an environment, including: an actuator, at least one sensor, a processor, and memory storing instructions that when executed by the processor effectuates operations including: capturing a plurality of data by the at least one sensor of the robot, wherein: the plurality of data comprises first data comprising pixel characteristics indicative of features of the environment and second data indicative of depth to objects in the environment; the plurality of data is captured from different positions within the environment through which the robot moves, the plurality of data corresponding with respective positions from which the plurality of data was captured; and the plurality of data captured from different respective positions within the environment corresponds to respective fields of view; and aligning the plurality of data as it is captured to more accurately perceive the spatial representation of the environment.

First claim

The invention claimed is:

1. A robot for perceiving a spatial model of an environment, comprising:
an actuator configured to move the robot through the environment;
at least one sensor mechanically coupled to the robot;
a processor configured to receive sensed data from the at least one sensor and control the actuator; and
memory storing instructions that when executed by the processor effectuates operations comprising:
capturing a plurality of data by the at least one sensor of the robot while the robot moves within the environment, wherein:

the plurality of data comprises first data comprising pixel characteristics indicative of features of the environment and second data indicative of depth to objects in the environment;
the first data and the second data are captured by at least one of:
a same sensor;
a same sensor in combination with structured light emissions; and
at least one depth sensor;
the plurality of data is captured from different positions within the environment through which the robot moves, the plurality of data corresponding with respective positions from which the plurality of data was captured; and
the plurality of data captured from the respective positions within the environment corresponds to respective fields of view from which the plurality of data was captured; and aligning, with a processor of the robot, the plurality of data as it is captured to more accurately perceive the spatial model of the environment.

2. The robot of claim 1 , wherein the operations further comprise:
detecting a feature in a first image of the first data;
detecting the same feature in a second image of the first data;
determining a value indicative of a difference in a position of the feature in the second image relative to the first image in a common frame of reference;
aligning the plurality of data captured based on the value; and
correcting third data indicative of angular and linear translation of the robot between when the first image of the first data is captured and when the second image of the first data is captured based on the value, wherein the third data comprises at least one of: optical encoder data, gyroscope data, inertial measurement unit data, and optical tracking sensor data.

3. The robot of claim 1 , wherein the at least one depth sensor comprises at least one of:
a structured light sensor or a time-of-flight sensor; and
a limited field of view.

4. The robot of claim 1 , wherein the plurality of data is captured by at least one of:
sensors with overlapping fields of view, with non-overlapping fields of view, or with overlapping and non-overlapping fields of view;
a sensor with a long range of sight and a sensor with short range of sight; and
a sensor with a long range of sight with low accuracy and a sensor with short range of sight with high accuracy.

5. The robot of claim 1 , wherein the operations further comprise:
detecting a first edge at a first position in a first image of the first data based on a derivative of the first data with respect to one or more spatial coordinates of the first data in the first image;
detecting a second edge at a second position in the first image based on the derivative of the first data with respect to one or more spatial coordinates of first data in the first image;
detecting a third edge in a third position in a second image of the first data based on a derivative of the first data with respect to one or more spatial coordinates of the first data in the second image;
determining that the third edge is not the same edge as the second edge based on shapes of the third edge and the second edge not matching;
determining that the third edge is the same edge as the first edge based on shapes of the first edge and the third edge at least partially matching; and
determining a value indicative of a difference between the first position and the third position;
aligning the plurality of data captured based on the value; and
correcting third data indicative of angular and linear translation of the robot between when the first image of the first data is captured and when the second image of the first data is captured based on the value.

6. The robot of claim 1 , wherein at least some data processing of the spatial model is offloaded from the robot to the cloud, and wherein the spatial model is stored in memory accessible to the robot during a subsequent operational session for use in autonomously navigating the environment.

7. The robot of claim 1 , further comprising at least one cleaning tool comprising any of a vacuum, a mop, and a sweeper, wherein the at least one cleaning tool operate simultaneously or individually or in alternation based on the surface the robot is operating on or settings configured by the user.

8. The method of claim 7 , wherein the robot cleans areas of the environment that require only vacuuming before cleaning areas of the environment that require mopping.

9. The robot of claim 1 , wherein the robot is wirelessly connected with an application of a communication device configured to:
display at least one of: the spatial representation of the environment as its being built and after completion; a movement path of the robot; a current position of the robot; a current position of a charging station of the robot; robot status; a current quantity of total area cleaned; a total area cleaned after completion of a task; a battery level; a current cleaning duration; an estimated total cleaning duration required to complete a task; an estimated total battery power required to complete a task, a time of completion of a task; obstacles within the spatial representation including object type of the obstacle and percent confidence of the object type; obstacles within the spatial representation including obstacles with unidentified object type; issues requiring user attention within the spatial representation; a fluid flow rate for different areas within the spatial representation; a notification that the robot has reached a particular location; cleaning history; user manual; maintenance information; and firmware information;
receive an input designating at least one of: an object type of an obstacle with unidentified object type; a schedule for cleaning different areas within the spatial representation; vacuuming or mopping or vacuuming and mopping for cleaning different areas within the spatial representation; a suction level for cleaning different areas within the spatial representation; a no-entry zone; a no-mopping zone; a virtual wall; a modification to the spatial representation; a fluid flow rate level for mopping different areas within the spatial representation; an order of cleaning different areas of the environment; deletion or addition of a robot paired with the application; an instruction to find the robot; an instruction to contact customer service; an instruction to update firmware; a driving speed of the robot; a volume of the robot; a voice type of the robot; pet details; deletion of an obstacle within the spatial representation; an instruction for a charging station of the robot; an instruction for the charging station of the robot to empty a bin of the robot into a bin of the charging station; and an instruction for the charging station of the robot to fill a fluid reservoir of the robot; and
receive an input enacting an instruction for the robot to at least one of: pause a current task; start mopping or vacuuming; dock at the charging station; start cleaning; spot clean; navigate to a particular location; and move or rotate in a particular direction.

10. The robot of claim 1 , wherein the operations further comprise:
determining, with the processor of the robot, a location, a height, a width, and a

Assignees

• Ebrahimi Afrouzi Ali
• Zhang Chen
• Schweigert Sebastian
• Fath Lukas
• Ai Inc

Inventors

• Ebrahimi Afrouzi Ali
• Zhang Chen
• Schweigert Sebastian
• Fath Lukas

Classifications

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• A46B13/001

  Cylindrical or annular brush bodies · CPC title

• G06N3/088

  Non-supervised learning, e.g. competitive learning · CPC title

• G06N3/008

  based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour · CPC title

• G06F2203/011

  Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns · CPC title