Planar surface detection

Patent metadata

Field	Value
Publication number	US-11074749-B2
Application number	US-201916396444-A
Country	US
Kind code	B2
Filing date	Apr 26, 2019
Priority date	Apr 26, 2019
Publication date	Jul 27, 2021
Grant date	Jul 27, 2021

Abstract

The described implementations relate to processing of an environment using a plurality of sub-volumes, and specifically to generating surface representations in the plurality of sub-volumes for individual processing. One example can identify planar fragments within the plurality of sub-volumes. The example can determine that various planar fragments constitute part of a contiguous planar surface and should be aggregated. The example can also output data representing the contiguous planar surface formed from the aggregated planar fragments.

First claim

The invention claimed is:

1. A method, comprising:
partitioning a three-dimensional environment including objects into a plurality of sub-volumes;
generating a surface representation in each of the plurality of sub-volumes based on captured environment information;
detecting a plurality of planar fragments inside each of the plurality of sub-volumes, the plurality of planar fragments having semantic labels indicating respective objects associated with respective planar fragments;
determining that at least some of the plurality of planar fragments belong to a same object and are candidate planar fragments for aggregating based at least on the semantic labels associated with the candidate planar fragments being the same;
aggregating the candidate planar fragments to form a contiguous planar surface spanning multiple sub-volumes; and
outputting data representing the contiguous planar surface.

2. The method of claim 1 , wherein the generated surface representation utilizes a point cloud or a surface mesh.

3. The method of claim 1 , wherein determining that at least some of the plurality of planar fragments are candidate planar fragments further comprises:
determining whether a first planar fragment has a similar plane equation as a second planar fragment; and
marking the first planar fragment and the second planar fragment as candidate planar fragments for aggregating based at least on having a similar plane equation.

4. The method of claim 3 , wherein the candidate planar fragments are aggregated to form the contiguous planar surface when their plane equations are within a threshold level of similarity.

5. The method of claim 1 , further comprising:
capturing subsequent environment information; and
updating a subset portion of the plurality of sub-volumes corresponding to detected changes reflected in the subsequent environment information.

6. The method of claim 1 , wherein each of the plurality of sub-volumes is of a fixed size.

7. The method of claim 6 , wherein each of the plurality of sub-volumes is cubical in shape and overlaps with other sub-volumes.

8. The method of claim 1 , wherein detecting the plurality of planar fragments is performed independently and in parallel for each sub-volume.

9. A method comprising:
detecting a plurality of planar fragments that are located within neighboring sub-volumes depicting a physical environment including objects, the plurality of planar fragments having semantic labels indicating respective objects associated with respective planar fragments;
determining that a first candidate planar fragment in a first sub-volume and a second candidate planar fragment in a second sub-volume belong to a same object and should be aggregated by comparing three-dimensional points associated with the first candidate planar fragment and second candidate planar fragment and by checking that the semantic labels associated with the first candidate planar fragment and the second candidate planar fragment are the same; and
aggregating the first candidate planar fragment and the second candidate planar fragment to form a contiguous planar surface.

10. The method of claim 9 , wherein comparing the three-dimensional points further comprises:
creating a two-dimensional projection image by projecting three-dimensional points from the first candidate planar fragment and the second candidate planar fragment; and
determining whether the projected three-dimensional points associated with the first candidate planar fragment are within a threshold distance of the projected three-dimensional points associated with the second candidate planar fragment.

11. The method of claim 10 , wherein the projection image is created by rasterizing the three-dimensional points.

12. The method of claim 11 , wherein the projection image has pixels that are sized according to the threshold distance, and the rasterized three-dimensional points map to the pixels of the projection image.

13. The method of claim 9 , wherein detecting the plurality of planar fragments comprises:
determining a set of neighboring sub-volumes based on positions of the sub-volumes;
selecting, as a seed plane, a planar fragment having the highest number of inlier planes; and
selecting, as candidate planar fragments, planar fragments of the sub-volumes that have planar equations similar to the seed plane.

14. The method of claim 13 , wherein a planar fragment is an inlier plane to the seed plane based at least upon determining that an angle between surface normals is below a threshold value.

15. The method of claim 13 , wherein a planar fragment is an inlier plane to the seed plane based at least upon determining that a distance between average surface points is below a threshold value.

16. A system comprising:
a camera configured to capture prior environment information of a physical environment;
a memory storing prior planar data associated with a plurality of sub-volumes depicting the prior environment information; and
a processor configured to:
capture, via the camera, current environment information of the physical environment that has changed since the prior environment information was captured; and
determine whether to update individual sub-volumes of the prior environment information stored in the memory based on changes between the prior environment information and the current environment information by:
loading from the memory a subset portion of the prior planar data associated with the prior environment information; and
identifying the individual sub-volumes that include the changes between current planar data associated with the current environment information and the prior planar data associated with the prior environment information; and
updating the prior planar data associated with the identified individual sub-volumes stored in the memory with the current planar data associated with the identified individual sub-volumes to reflect the changes between the current environment information and the prior environment information.

17. The system of claim 16 , wherein the subset portion is a spherical area of prior planar data associated with sub-volumes that are within a set distance of a current position of the camera in the physical environment.

18. The system of claim 16 , wherein the stored prior planar data is updated with the current planar data only if the changes between the current planar data associated with the current environment information and the prior planar data associated with the prior environment information exceed a threshold.

19. The system of claim 16 , wherein the changes include a movement of a captured object in the physical environment.

20. A method, comprising:
generating point cloud information based on captured environment information of an environment including objects;
partitioning the point cloud information into a plurality of sub-volumes;
detecting a plurality of planar fragments inside individual sub-volumes, the plurality of planar fragments having semantic labels indicating respective objects associated with respective planar fragments;
determining a subset of the plurality of planar fragments that belong to a same object and are candidate planar fragments for aggregating based at least on the semantic labels associated with the candidate planar fragments being the same;
aggregating the candidate planar fragments that represent a contiguous planar surface; and
outputting data representing the contiguous planar surface.

21. The method of claim 2

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Pekelny Yuri
• Ulusoy Ali O

Classifications

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T17/20 Primary

  Finite element generation, e.g. wire-frame surface description, {tesselation} · CPC title

• G06T11/40

  Filling planar surfaces by adding surface attributes, e.g. adding colours or textures · CPC title

• G06F3/04815 Primary

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title