Plane detection using semantic segmentation

Patent metadata

Field	Value
Publication number	US-11972607-B2
Application number	US-202318111541-A
Country	US
Kind code	B2
Filing date	Feb 18, 2023
Priority date	Jun 25, 2018
Publication date	Apr 30, 2024
Grant date	Apr 30, 2024

Abstract

In one implementation, a method of generating a plane hypothesis is performed by a device including one or more processors, non-transitory memory, and a scene camera. The method includes obtaining an image of a scene including a plurality of pixels. The method includes obtaining a plurality of points of a point cloud based on the image of the scene. The method includes obtaining an object classification set based on the image of the scene. Each element of the object classification set includes a plurality of pixels respectively associated with a corresponding object in the scene. The method includes detecting a plane within the scene by identifying a subset of the plurality of points of the point cloud that correspond to a particular element of the object classification set.

First claim

What is claimed is:

1. A method comprising:
obtaining an image of a scene including a plurality of pixels;
obtaining a plurality of points of a point cloud based on the image of the scene;
obtaining an object classification based on the image of the scene, wherein the object classification corresponds to a plurality of pixels respectively associated with a corresponding object in the scene; and
detecting a plane within the scene by identifying at least a subset of the plurality of points of the point cloud that correspond to the object classification.

2. The method of claim 1 , wherein obtaining the object classification includes generating the object classification via semantic segmentation, and wherein each of the subset of the plurality of points of the point cloud that correspond to the object classification includes a semantic label associated with the corresponding object in the scene.

3. The method of claim 1 , wherein detecting the plane includes generating a plane hypothesis based on the point cloud and the object classification.

4. The method of claim 3 , wherein generating the plane hypothesis includes:
generating a first plane hypothesis based on the point cloud;
associating the first plane hypothesis with the object classification;
identifying the subset of the plurality of points of the point cloud based on the plurality of pixels associated with the corresponding object in the scene that corresponds to the object classification; and
updating the first plane hypothesis based on the subset of the plurality of points of the point cloud.

5. The method of claim 3 , wherein generating the plane hypothesis includes:
determining an initial confidence score associated with the plane hypothesis based on the object classification;
in accordance with a determination that a count of the subset of the plurality of points of the point cloud is greater than a threshold number, generating an increased confidence score associated with the plane hypothesis that is greater than the initial confidence score; and
in accordance with a determination that the count of the subset of the plurality of points of the point cloud is less than the threshold number, generating a decreased confidence score associated with the plane hypothesis that is less than the initial confidence score.

6. The method of claim 3 , wherein generating the plane hypothesis includes:
applying a random sample consensus (RANSAC) plane detection algorithm to the subset of the plurality of points of the point cloud; and
foregoing applying the RANSAC plane detection algorithm to a remainder subset of the plurality of points of the point cloud, wherein each of the remainder subset of the plurality of points is not included in the subset of the plurality of points of the point cloud.

7. The method of claim 1 , wherein obtaining the plurality of points of the point cloud is based on VIO (visual inertial odometry) and/or data from a depth sensor.

8. The method of claim 1 , wherein obtaining the object classification includes generating the object classification by applying a neural network to the image of the scene.

9. A device comprising:
a scene camera to obtain an image of a scene including a plurality of pixels; and
one or more processors for:
obtaining a plurality of points of a point cloud based on the image of the scene;
obtaining an object classification based on the image of the scene, wherein the object classification corresponds to a plurality of pixels within the image of the scene that are respectively associated with a corresponding object in the scene; and
detecting a plane within the scene by identifying at least a subset of the plurality of points of the point cloud that correspond to the object classification.

10. The device of claim 9 , wherein obtaining the object classification includes generating the object classification via semantic segmentation, and wherein each of the subset of the plurality of points of the point cloud that correspond to the object classification includes a semantic label associated with the corresponding object in the scene.

11. The device of claim 9 , wherein detecting the plane includes generating a plane hypothesis based on the point cloud and the object classification.

12. The device of claim 11 , wherein generating the plane hypothesis includes:
generating a first plane hypothesis based on the point cloud;
associating the first plane hypothesis with the object classification;
identifying the subset of the plurality of points of the point cloud based on the plurality of pixels associated with the corresponding object in the scene that corresponds to the object classification; and
updating the first plane hypothesis based on the subset of the plurality of points of the point cloud.

13. The device of claim 11 , wherein generating the plane hypothesis includes:
determining an initial confidence score associated with the plane hypothesis based on the object classification;
in accordance with a determination that a count of the subset of the plurality of points of the point cloud is greater than a threshold number, generating an increased confidence score associated with the plane hypothesis that is greater than the initial confidence score; and
in accordance with a determination that the count of the subset of the plurality of points of the point cloud is less than the threshold number, generating a decreased confidence score associated with the plane hypothesis that is less than the initial confidence score.

14. The device of claim 11 , wherein generating the plane hypothesis includes:
applying a random sample consensus (RANSAC) plane detection algorithm to the subset of the plurality of points of the point cloud; and
foregoing applying the RANSAC plane detection algorithm to a remainder subset of the plurality of points of the point cloud, wherein each of the remainder subset of the plurality of points is not included in the subset of the plurality of points of the point cloud.

15. The device of claim 9 , wherein obtaining the object classification includes generating the object classification by applying a neural network to the image of the scene.

16. A non-transitory memory storing one or more programs, which, when executed by one or more processors of a device with one or more scene cameras, cause the device to perform operations comprising:
obtaining an image of a scene including a plurality of pixels;
obtaining a plurality of points of a point cloud based on the image of the scene;
obtaining an object classification based on the image of the scene, wherein the object classification corresponds to a plurality of pixels respectively associated with a corresponding object in the scene; and
detecting a plane within the scene by identifying at least a subset of the plurality of points of the point cloud that correspond to the object classification.

17. The non-transitory memory of claim 16 , wherein obtaining the object classification includes generating the object classification via semantic segmentation, and wherein each of the subset of the plurality of points of the point cloud that correspond to the object classification includes a semantic label associated with the corresponding object in the scene.

18. The non-transitory memory of claim 16 , wherein detecting the plane includes generating a plane hypothesis based on the point cloud and the object classification.

19. The non-transitory memory of claim 18 , wherein generating the plane hypothesis includes:
generating a first plane hypothes

Assignees

• Apple Inc

Inventors

• Ulbricht Daniel
• Blechschmidt Angela
• Baig Mohammad Haris
• Batra Tanmay
• Verma Eshan
• Kc Amit Kumar

Classifications

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T2200/24

  involving graphical user interfaces [GUIs] · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06V30/274

  Syntactic or semantic context, e.g. balancing · CPC title