Pixelwise filterable depth maps for robots

Patent metadata

Field	Value
Publication number	US-11618167-B2
Application number	US-201916726769-A
Country	US
Kind code	B2
Filing date	Dec 24, 2019
Priority date	Dec 24, 2019
Publication date	Apr 4, 2023
Grant date	Apr 4, 2023

Abstract

A method includes receiving sensor data from a plurality of robot sensors on a robot. The method includes generating a depth map that includes a plurality of pixel depths. The method includes determining, for each respective pixel depth, based on the at least one robot sensor associated with the respective pixel depth, a pixelwise confidence level indicative of a likelihood that the respective pixel depth accurately represents a distance between the robot and a feature of the environment. The method includes generating a pixelwise filterable depth map for a control system of the robot. The pixelwise filterable depth map is filterable to produce a robot operation specific depth map. The robot operation specific depth map is determined based on a comparison of each respective pixelwise confidence level with a confidence threshold corresponding to at least one operation of the robot controlled by the control system of the robot.

First claim

What is claimed is:

1. A method comprising:
receiving sensor data from a plurality of robot sensors on a robot, including at least a first type of sensor and a second type of sensor, wherein the sensor data comprises a plurality of pixels that represent an environment of the robot;
generating a depth map comprising a plurality of pixel depths, wherein each pixel depth is determined based on the sensor data, and wherein each pixel depth is associated with at least one robot sensor of the plurality of robot sensors;
determining, for each respective pixel depth, based on the type of sensor of the at least one robot sensor associated with the respective pixel depth, a pixelwise confidence level indicative of a likelihood that the respective pixel depth accurately represents a distance between the robot and a feature of the environment;
generating, based on the pixelwise confidence level determined for each respective pixel depth, a pixelwise filterable depth map for a control system of the robot;
determining, for the robot, a confidence threshold for a particular robot operation;
filtering the pixelwise filterable depth map to produce a robot operation specific depth map, wherein the robot operation specific depth map is determined based on a comparison of each respective pixelwise confidence level with the confidence threshold corresponding to the particular robot operation of the robot controlled by the control system of the robot; and
controlling, by the control system, the robot to perform the particular robot operation based on the robot operation specific depth map.

2. The method of claim 1 , further comprising determining a set of confidence levels for the pixelwise confidence levels based on the plurality of robot sensors, wherein a first confidence level in the set of confidence levels corresponds to the first type of sensor and wherein a second confidence level in the set of confidence levels corresponds to the second type of sensor.

3. The method of claim 2 , wherein the first confidence level is above the confidence threshold and the second confidence level is below the confidence threshold, the method further comprising filtering out pixels from the pixelwise filterable depth map that correspond to the second type of sensor.

4. The method of claim 3 , wherein the first type of sensor comprises a stereoscopic image capture device and wherein the second type of sensor comprises a monoscopic image capture device.

5. The method of claim 3 , wherein the first type of sensor comprises a light ranging and detection (LIDAR) device and wherein the second type of sensor comprises an image capture device.

6. The method of claim 1 , wherein the depth map comprises a first region associated with a first sensor and a second region associated with a second sensor, wherein the second region is adjacent to the first region such that the first region and the second region adjoin at a plurality of edge pixel depths, and wherein determining the pixelwise confidence level for each pixel depth comprises:
determining a confidence level for each pixel depth in the second region based on a distance between the pixel and the plurality of edge pixel depths.

7. The method of claim 1 , further comprising:
receiving, via a user interface, a user-selected confidence threshold associated with the particular robot operation, wherein the user-selected confidence threshold corresponds to the confidence threshold; and
filtering the pixelwise filterable depth map to produce a robot operation specific depth map associated with the user-selected confidence threshold.

8. The method of claim 1 ,
wherein the robot operation specific depth map comprises a used portion comprising unfiltered pixel depths and an unused portion comprising filtered pixel depths; and the method further comprises:
performing the operation of the robot using the unfiltered pixel depths in the used portion of the operation specific depth map.

9. The method of claim 8 , further comprising:
determining a proportion of unfiltered pixel depths relative to a total number of pixel depths in the pixelwise filterable depth map; and
determining that the proportion of unfiltered pixel depths exceeds a threshold proportion of unfiltered pixel depths associated with the operation of the robot,
wherein performing the operation of the robot comprises, performing the operation of the robot responsive to determining that the proportion of unfiltered pixel depths exceeds the threshold proportion of unfiltered pixel depths.

10. The method of claim 8 , further comprising:
determining that the used portion of the robot operation specific depth map comprises an uninterrupted region of unfiltered pixel depths that exceeds a threshold region size,
wherein performing the operation of the robot comprises, performing the operation of the robot responsive to determining that the used portion of the robot operation specific depth map comprises the uninterrupted region of unfiltered pixel depths that exceeds the threshold region size.

11. The method of claim 8 , wherein the operation of the robot comprises navigating the robot within the environment, the method further comprising:
selecting a path for navigating the robot within the environment that corresponds to the used portion of the robot operation specific depth map.

12. The method of claim 11 , further comprising:
navigating the robot within the environment;
changing the robot operation specific depth map while navigating the robot within the environment based on additional sensor data from the at least one sensor; and
updating the path for navigating the robot within the environment based on a change in the used portion of robot operation specific depth map.

13. The method of claim 1 , wherein a plurality of robot operations each correspond to a different confidence threshold, the method further comprising:
receiving an instruction to perform the particular robot operation; and
selecting the confidence threshold based on the particular robot operation from the plurality of robot operations.

14. The method of claim 1 , wherein determining, for each respective pixel depth, the pixelwise confidence level comprises:
comparing pixel depths from two or more sensors;
determining a degree of similarity between the compared pixel depths for each respective pixel depth in the depth map; and
determining, for each respective pixel depth, the pixelwise confidence level based at least in part on determining the degree of similarity between the compared pixel depths for each respective pixel depth in the depth map.

15. A robot comprising:
a first sensor of a first type;
a second sensor of a second type;
a control system having one or more processors;
a non-transitory computer readable medium; and
program instructions stored on the non-transitory computer readable medium and executable by the one or more processors to:
receive sensor data from the first sensor and the second sensor, wherein the sensor data comprises a plurality of pixels that represent an environment of the robot;
generate a depth map comprising a plurality of pixel depths, wherein each pixel depth is determined based on the sensor data, and wherein each pixel depth is associated with at least one robot sensor of the first sensor or the second sensor;
determine, for each respective pixel depth, based on the type of sensor of the at least one robot sensor associated with the respective pixel depth, a pixelwise confidence level indicative of a likelihood that the respective pixel depth accurately represents a distance between the robot and a feature of

Assignees

• X Dev Llc

Inventors

• Satat Guy
• Quinlan Michael

Classifications

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T2207/10021

  Stereoscopic video; Stereoscopic image sequence · CPC title

• G06T2207/10012

  Stereo images · CPC title

• B25J19/023

  including video camera means · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title