On-floor obstacle detection method and mobile machine using the same

Patent metadata

Field	Value
Publication number	US-11734850-B2
Application number	US-202117239697-A
Country	US
Kind code	B2
Filing date	Apr 26, 2021
Priority date	Apr 26, 2021
Publication date	Aug 22, 2023
Grant date	Aug 22, 2023

Abstract

On-floor obstacle detection using an RGB-D camera is disclosed. An obstacle on a floor is detected by receiving an image including depth channel data and RGB channel data through the RGB-D camera, estimating a ground plane corresponding to the floor based on the depth channel data, obtaining a foreground of the image corresponding to the ground plane based on the depth channel data, performing a distribution modeling on the foreground of the image based on the RGB channel data to obtain a 2D location of the obstacle, and transforming the 2D location of the obstacle into a 3D location of the obstacle based on the depth channel data.

First claim

What is claimed is:

1. A method for detecting an obstacle on a floor using an RGB-D camera, comprising:
receiving an image through the RGB-D camera, wherein the image comprises depth channel data and RGB channel data;
estimating a ground plane corresponding to the floor based on the depth channel data;
obtaining a foreground of the image corresponding to the ground plane based on the depth channel data;
performing a distribution modeling on the foreground of the image based on the RGB channel data to obtain a 2D location of the obstacle; and
transforming the 2D location of the obstacle into a 3D location of the obstacle based on the depth channel data;
wherein the obtaining the foreground of the image corresponding to the ground plane based on the depth channel data comprises:
taking a position of each point in the depth channel data distant to the ground plane with at least a minimum distance as a candidate location of the obstacle; and
taking an area in the image corresponding to all the candidate locations of the obstacle as the foreground of the image.

2. The method of claim 1 , wherein the estimating the ground plane corresponding to the floor based on the depth channel data comprises:
dividing a plurality of points in the depth channel data into one or more patches each representing a plane in the depth channel data;
obtaining an included angle of each of the one or more planes with respect to the RGB-D camera based on the depth channel data; and
forming the ground plane corresponding to at least one of the one or more planes according to the included angle of the one or more planes.

3. The method of claim 2 , wherein the obtaining the included angle of each of the one or more planes with respect to the RGB-D camera based on the depth channel data comprises:
obtaining the included angle between each of the one or more planes and the RGB-D camera through a normal vector of the plane, wherein the normal vector is obtained by fitting to pixels in the depth channel data.

4. The method of claim 2 , wherein the forming the ground plane corresponding to at least one of the one or more planes according to the included angle of the one or more planes comprises:
connecting the neighboring planes as a single plane, wherein a difference between the included angles of the neighboring planes is not larger than a predetermined difference; and
taking the largest single plane as the ground plane.

5. The method of claim 1 , wherein the performing the distribution modeling on the foreground of the image based on the RGB channel data to obtain the 2D location of the obstacle comprises:
generating a mask based on the foreground of the obstacle;
obtaining a background of the image using the mask;
performing the distribution modeling on the foreground and the background of the image based on the RGB channel data; and
obtaining the 2D location of the obstacle based on the foreground of the image.

6. The method of claim 1 , wherein the 3D location comprises a plurality of coordinates corresponding to the obstacle.

7. A mobile machine, comprising:
an RGB-D camera;
one or more processors; and
one or more memories storing one or more programs configured to be executed by the one or more processors, wherein the one or more programs comprise instructions to:
receive an image through the RGB-D camera, wherein the image comprises depth channel data and RGB channel data;
estimate a ground plane corresponding to the floor based on the depth channel data;
obtain a foreground of the image corresponding to the ground plane based on the depth channel data;
perform a distribution modeling on the foreground of the image based on the RGB channel data to obtain a 2D location of the obstacle; and
transform the 2D location of the obstacle into a 3D location of the obstacle based on the depth channel data;
wherein the performing the distribution modeling on the foreground of the image based on the RGB channel data to obtain the 2D location of the obstacle comprises:
generating a mask based on the foreground of the obstacle;
obtaining a background of the image using the mask;
performing the distribution modeling on the foreground and the background of the image based on the RGB channel data; and
obtaining the 2D location of the obstacle based on the foreground of the image.

8. The mobile machine of claim 7 , wherein the estimating the ground plane corresponding to the floor based on the depth channel data comprises:
dividing a plurality of points in the depth channel data into one or more patches each representing a plane in the depth channel data;
obtaining an included angle of each of the one or more planes with respect to the RGB-D camera based on the depth channel data; and
forming the ground plane corresponding to at least one of the one or more planes according to the included angle of the one or more planes.

9. The mobile machine of claim 8 , wherein the obtaining the included angle of each of the one or more planes with respect to the RGB-D camera based on the depth channel data comprises:
obtaining the included angle between each of the one or more planes and the RGB-D camera through a normal vector of the plane, wherein the normal vector is obtained by fitting to pixels in the depth channel data.

10. The mobile machine of claim 8 , wherein the forming the ground plane corresponding to at least one of the one or more planes according to the included angle of the one or more planes comprises:
connecting the neighboring planes as a single plane, wherein a difference between the included angles of the neighboring planes is not larger than a predetermined difference; and
taking the largest single plane as the ground plane.

11. The mobile machine of claim 7 , wherein the obtaining the foreground of the image corresponding to the ground plane based on the depth channel data comprises:
taking a position of each point in the depth channel data distant to the ground plane with at least a minimum distance as a candidate location of the obstacle; and
taking an area in the image corresponding to all the candidate locations of the obstacle as the foreground of the image.

12. The mobile machine of claim 7 , wherein the 3D location comprises a plurality of coordinates corresponding to the obstacle.

13. A non-transitory computer readable storage medium storing one or more programs, wherein the one or more programs comprise instructions, which when executed by a mobile machine having a plurality of sensors, cause the mobile machine to:
receive an image through the RGB-D camera, wherein the image comprises depth channel data and RGB channel data;
estimate a ground plane corresponding to the floor based on the depth channel data;
obtain a foreground of the image corresponding to the ground plane based on the depth channel data;
perform a distribution modeling on the foreground of the image based on the RGB channel data to obtain a 2D location of the obstacle; and
transform the 2D location of the obstacle into a 3D location of the obstacle based on the depth channel data;
wherein the obtaining the foreground of the image corresponding to the ground plane based on the depth channel data comprises:
taking a position of each point in the depth channel data distant to the ground plane with at least a minimum distance as a candidate location of the obstacle; and
taking an area in the image corresponding to all the candidate locations of the obstacle as the foreground of the image.

14. The storage medium of claim 13 , wherein the estimating the ground plane corresponding to the floor based on the depth channel data com

Assignees

• Ubkang Qingdao Tech Co Ltd
• Ubtech North America Res And Development Center Corp
• Ubtech Robotics Corp Ltd

Inventors

• Shao Dan
• Guo Dejun
• Xiu Zhen
• Dong Chuqiao
• Tan Huan

Classifications

• G05D1/0251 Primary

  extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision (stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593) · CPC title

• G06T7/70 Primary

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G06T7/194

  involving foreground-background segmentation · CPC title

• G06T7/50

  Depth or shape recovery · CPC title

• G06T2207/10024

  Color image · CPC title