Plane detection using semantic segmentation

Patent metadata

Field	Value
Publication number	US-10824864-B2
Application number	US-201916360732-A
Country	US
Kind code	B2
Filing date	Mar 21, 2019
Priority date	Jun 25, 2018
Publication date	Nov 3, 2020
Grant date	Nov 3, 2020

Abstract

In one implementation, a method of generating a plane hypothesis is performed by a head-mounted device (HMD) including one or more processors, non-transitory memory, and a scene camera. The method includes obtaining an image of a scene including a plurality of pixels. The method include obtaining a point cloud based on the image of the scene and generating an object classification set based on the image of the scene, each element of the object classification set including a respective plurality of pixels classified as a respective object in the scene. The method includes generating a plane hypothesis based on the point cloud and the object classification set.

First claim

What is claimed is:

1. A method comprising:
obtaining an image of a scene including a plurality of pixels;
obtaining a point cloud based on the image of the scene;
generating an object classification set based on the image of the scene, each element of the object classification set including a respective plurality of pixels classified as a respective object in the scene; and
generating a plane hypothesis based on the point cloud and the object classification set, wherein generating the plane hypothesis includes:
generating a first plane hypothesis based on the point cloud;
associating the first plane hypothesis with a particular element of the object classification set;
determining a subset of points of the point cloud corresponding the respective plurality of pixels of the particular element of the object classification set; and
updating the first plane hypothesis based on the subset of points of the point cloud.

2. The method of claim 1 , wherein generating the plane hypothesis includes:
determining a subset of points of the point cloud corresponding to the respective plurality of pixels of a particular element of the object classification set; and
generating the plane hypothesis based on the subset of points of the point cloud.

3. The method of claim 1 , wherein updating the first plane hypothesis based on the subset of points in the point cloud includes expanding a boundary of the first plane hypothesis to include an additional one or more points of the subset of points in the point cloud.

4. The method of claim 1 , wherein generating the plane hypothesis includes determining a confidence associated with the plane hypothesis based on the object classification set.

5. The method of claim 1 , further comprising:
obtaining a second image of the scene including a second plurality of pixels;
obtaining a second point cloud based on the second image of the scene;
generating a second object classification set based on the second image of the scene, each element of the second object classification set including a respective second plurality of pixels classified as a respective object in the scene; and
updating a plane hypothesis based on the second point cloud and the second object classification set.

6. The method of claim 1 , wherein at least one element of the object classification set further includes a label.

7. The method of claim 6 , further comprising:
associating the plane hypothesis with a particular element of the object classification set including a label;
detecting a user input that corresponds to association of a computer-generated reality object with the plane hypothesis; and
in response to the label of the particular element meeting a placement criteria, associating the computer-generated reality object with the plane hypothesis.

8. The method of claim 6 , further comprising:
associating the plane hypothesis with a particular element of the object classification set including a label; and
in response to the label of the particular element meeting a modification criteria, displaying a computer-generated reality object at the location of the plane hypothesis.

9. The method of claim 1 , wherein obtaining the point cloud is based on VIO (visual inertial odometry) and/or data from a depth sensor.

10. The method of claim 1 , wherein generating an object classification set is based on applying a neural network to the image of the scene.

11. A device comprising:
a scene camera; and
one or more processors to:
obtain, using the scene camera, an image of a scene including a plurality of pixels;
obtain a point cloud based on the image of the scene;
generate an object classification set based on the image of the scene, each element of the object classification set including a respective plurality of pixels classified as a respective object in the scene; and
generate a plane hypothesis based on the point cloud and the object classification set, wherein generating the plane hypothesis includes:
generating a first plane hypothesis based on the point cloud;
associating the first plane hypothesis with a particular element of the object classification set;
determining a subset of points of the point cloud corresponding the respective plurality of pixels of the particular element of the object classification set; and
updating the first plane hypothesis based on the subset of points of the point cloud.

12. The device of claim 11 , wherein the one or more processors are to generate the plane hypothesis by:
determining a subset of points of the point cloud corresponding to the respective plurality of pixels of a particular element of the object classification set; and
generating the plane hypothesis based on the subset of points of the point cloud.

13. The device of claim 11 , wherein updating the first plane hypothesis based on the subset of points in the point cloud includes expanding a boundary of the first plane hypothesis to include an additional one or more points of the subset of points in the point cloud.

14. The device of claim 11 , wherein at least one element of the object classification set further includes a label and the one or more processors are further to:
associate the plane hypothesis with a particular element of the object classification set including a label;
detect a user input that corresponds to association of a computer-generated reality object with the plane hypothesis; and
in response to the label of the particular element meeting a placement criteria, associate the computer-generated reality object with the plane hypothesis.

15. The device of claim 11 , wherein at least one element of the object classification set further includes a label and the one or more processors are further to:
associate the plane hypothesis with a particular element of the object classification set including a label; and
in response to the label of the particular element meeting a modification criteria, display a computer-generated reality object at the location of the plane hypothesis.

16. A non-transitory memory storing one or more programs, which, when executed by one or more processors of a device with one or more scene cameras, cause the device to perform operations comprising:
obtaining an image of a scene including a plurality of pixels;
obtaining a point cloud based on the image of the scene;
generating an object classification set based on the image of the scene, each element of the object classification set including a respective plurality of pixels classified as a respective object in the scene; and
generating a plane hypothesis based on the point cloud and the object classification set, wherein generating the plane hypothesis includes:
generating a first plane hypothesis based on the point cloud;
associating the first plane hypothesis with a particular element of the object classification set;
determining a subset of points of the point cloud corresponding the respective plurality of pixels of the particular element of the object classification set; and
updating the first plane hypothesis based on the subset of points of the point cloud.

17. The non-transitory memory of claim 16 , wherein generating the plane hypothesis includes:
determining a subset of points of the point cloud corresponding to the respective plurality of pixels of a particular element of the object classification set; and
generating the plane hypothesis based on the subset of points of the point cloud.

18. The non-transitory memory of claim 16 , wherein updating the first plane hypothesis based on the subset o

Assignees

• Apple Inc

Inventors

• Ulbricht Daniel
• Blechschmidt Angela
• Baig Mohammad Haris
• Batra Tanmay
• Verma Eshan
• Kc Amit Kumar

Classifications

• G06V20/64

  Three-dimensional [3D] objects · CPC title

• G06V30/19173

  Classification techniques · CPC title

• G06V20/20 Primary

  in augmented reality scenes · CPC title

• G06V20/10

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title

• G06F18/24

  Classification techniques · CPC title