Plane detection using semantic segmentation

Patent metadata

Field	Value
Publication number	US-11610397-B2
Application number	US-202117473469-A
Country	US
Kind code	B2
Filing date	Sep 13, 2021
Priority date	Jun 25, 2018
Publication date	Mar 21, 2023
Grant date	Mar 21, 2023

Abstract

In one implementation, a method of generating a plane hypothesis is performed by a device including one or more processors, non-transitory memory, and a scene camera. The method includes obtaining an image of a scene including a plurality of pixels. The method includes obtaining a plurality of points of a point cloud based on the image of the scene. The method includes obtaining an object classification set based on the image of the scene. Each element of the object classification set includes a plurality of pixels respectively associated with a corresponding object in the scene. The method includes detecting a plane within the scene by identifying a subset of the plurality of points of the point cloud that correspond to a particular element of the object classification set.

First claim

What is claimed is:

1. A method comprising:
obtaining an image of a scene including a plurality of pixels;
obtaining a plurality of points of a point cloud based on the image of the scene;
obtaining an object classification set based on the image of the scene, wherein each element of the object classification set includes a plurality of pixels respectively associated with a corresponding object in the scene; and
detecting a plane within the scene by identifying a subset of the plurality of points of the point cloud that correspond to a particular element of the object classification set.

2. The method of claim 1 , wherein each element of the object classification set is associated with a distinct object in the scene.

3. The method of claim 1 , wherein obtaining the object classification set includes generating the object classification set via semantic segmentation, and wherein each element of the object classification set includes a semantic label associated with a corresponding object in the scene.

4. The method of claim 1 , wherein detecting the plane includes generating a plane hypothesis based on the point cloud and the object classification set.

5. The method of claim 4 , wherein generating the plane hypothesis includes:
generating a first plane hypothesis based on the point cloud;
associating the first plane hypothesis with the particular element of the object classification set;
identifying the subset of the plurality of points of the point cloud based on pixels of the particular element of the object classification set;
updating the first plane hypothesis based on the subset of points of the point cloud.

6. The method of claim 4 , wherein generating the plane hypothesis includes:
determining an initial confidence score associated with the plane hypothesis based on the object classification set;
in accordance with a determination that a number of the subset of the plurality of points is greater than a threshold number, generating an increased confidence score associated with the plane hypothesis that is greater than the initial confidence score; and
in accordance with a determination that the number of the subset of the plurality of points is less than the threshold number, generating a decreased confidence score associated with the plane hypothesis that is less than the initial confidence score.

7. The method of claim 4 , wherein generating the plane hypothesis includes:
determining an initial confidence score associated with the plane hypothesis based on the object classification set;
in accordance with a determination that a number of the subset of the plurality of points is greater than a threshold number, generating an increased confidence score associated with the plane hypothesis that is greater than the initial confidence score; and
in accordance with a determination that the number of the subset of the plurality of points is less than the threshold number, generating a decreased confidence score associated with the plane hypothesis that is less than the initial confidence score.

8. The method of claim 4 , wherein generating the plane hypothesis includes:
applying a random sample consensus (RANSAC) plane detection algorithm to the subset of the plurality of points of the point cloud; and
foregoing applying the RANSAC plane detection algorithm to a remainder subset of the plurality of points of the point cloud, wherein each of the remainder subset of the plurality of points is not included in the subset of the plurality of points of the point cloud.

9. The method of claim 1 , wherein obtaining the plurality of points of the point cloud is based on VIO (visual inertial odometry) and/or data from a depth sensor.

10. The method of claim 1 , wherein obtaining the object classification set includes generating the object classification set by applying a neural network to the image of the scene.

11. A device comprising:
a scene camera to obtain an image of a scene including a plurality of pixels; and
one or more processors to:
obtain a plurality of points of a point cloud based on the image of the scene;
obtain an object classification set based on the image of the scene, wherein each element of the object classification set includes a plurality of pixels respectively associated with a corresponding object in the scene; and
detect a plane within the scene by identifying a subset of the plurality of points of the point cloud that correspond to a particular element of the object classification set.

12. The device of claim 11 , wherein each element of the object classification set is associated with a distinct object in the scene.

13. The device of claim 11 , wherein obtaining the object classification set includes generating, via the one or more processors, the object classification set via semantic segmentation, and wherein each element of the object classification set includes a semantic label associated with a corresponding object in the scene.

14. The device of claim 11 , wherein detecting the plane includes generating, via the one or more processors, a plane hypothesis based on the point cloud and the object classification set.

15. The device of claim 14 , wherein generating the plane hypothesis includes:
generating a first plane hypothesis based on the point cloud;
associating the first plane hypothesis with the particular element of the object classification set;
identifying the subset of the plurality of points of the point cloud based on pixels of the particular element of the object classification set;
updating the first plane hypothesis based on the subset of points of the point cloud.

16. The device of claim 14 , wherein generating the plane hypothesis includes:
determining an initial confidence score associated with the plane hypothesis based on the object classification set;
in accordance with a determination that a number of the subset of the plurality of points is greater than a threshold number, generating an increased confidence score associated with the plane hypothesis that is greater than the initial confidence score; and
in accordance with a determination that the number of the subset of the plurality of points is less than the threshold number, generating a decreased confidence score associated with the plane hypothesis that is less than the initial confidence score.

17. The device of claim 14 , wherein generating the plane hypothesis includes:
determining an initial confidence score associated with the plane hypothesis based on the object classification set;
in accordance with a determination that a number of the subset of the plurality of points is greater than a threshold number, generating an increased confidence score associated with the plane hypothesis that is greater than the initial confidence score; and
in accordance with a determination that the number of the subset of the plurality of points is less than the threshold number, generating a decreased confidence score associated with the plane hypothesis that is less than the initial confidence score.

18. The device of claim 14 , wherein generating the plane hypothesis includes:
applying a random sample consensus (RANSAC) plane detection algorithm to the subset of the points of the point cloud; and
foregoing applying the RANSAC plane detection algorithm to the remaining points of the plurality of points of the point cloud.

19. The device of claim 11 , wherein obtaining the object classification set includes generating, via the or more processors, the object classification set by applying a neural netwo

Assignees

• Apple Inc

Inventors

• Ulbricht Daniel
• Blechschmidt Angela
• Baig Mohammad Haris
• Batra Tanmay
• Verma Eshan
• Kc Amit Kumar

Classifications

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T2200/24

  involving graphical user interfaces [GUIs] · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06V30/274

  Syntactic or semantic context, e.g. balancing · CPC title