Method and device for detecting a vertical planar surface

Patent metadata

Field	Value
Publication number	US-11935286-B2
Application number	US-201917278935-A
Country	US
Kind code	B2
Filing date	Oct 1, 2019
Priority date	Oct 1, 2018
Publication date	Mar 19, 2024
Grant date	Mar 19, 2024

Abstract

A method of vertical planar surface detection includes identifying, at an electronic device, a plurality of feature points of an environment proximate the electronic device based on imagery captured at the electronic device and based on received non-visual pose information obtained from non-visual sensors. The electronic device queries the plurality of feature points to identify one or more planar surfaces in the imagery captured at the electronic device. Further, the electronic device computes a plurality of oriented point vectors associated with the one or more planar surfaces. A vertical planar surface is detected based at least in part on one or more of the plurality of oriented point vectors being aligned in a direction perpendicular to gravity within a predetermined threshold.

First claim

What is claimed is:

1. A method, comprising:
identifying, at an electronic device by performing raycasting, a plurality of feature points of an environment proximate the electronic device based on imagery captured at the electronic device and based on received non-visual pose information obtained from non-visual sensors, the raycasting projecting rays into the electronic device's view of the environment and returning feature points that the rays intersect, along with the pose of that intersection in world space, to determine whether a perceived plane passes through the feature points at the intersection of the rays, the raycasting resolving whether a ray along the normal from a screen of the electronic device returns a planar surface;
identifying one or more planar surfaces based on the plurality of feature points;
computing, at the electronic device, a plurality of oriented point vectors associated with the one or more planar surfaces, comprising determining a surface normal to each of the identified plurality of feature points;
detecting a vertical planar surface within the one or more planar surfaces based at least in part on one or more of the plurality of oriented point vectors being aligned in a direction perpendicular to gravity within a predetermined threshold; and
generating a floor plan of the environment proximate the electronic device based on one or more of the plurality of oriented point vectors corresponding to vertical planar surfaces.

2. The method of claim 1 , wherein the plurality of feature points includes one or more features identifiable in imagery captured at the electronic device.

3. The method of claim 1 , wherein computing the plurality of oriented point vectors comprises:
determining, at the electronic device, a surface normal for each of the identified plurality of feature points.

4. The method of claim 1 , further comprising:
discarding a subset of the plurality of oriented point vectors, where the subset of the plurality of oriented point vectors are not directed towards the electronic device within a first predetermined threshold.

5. The method of claim 1 , further comprising:
discarding a subset of the plurality of oriented point vectors, where the subset of the plurality of oriented point vectors are not aligned in the direction perpendicular to gravity.

6. The method of claim 1 , wherein generating the floor plan of the environment comprises:
receiving, at the electronic device, a first infinite plane corresponding to a first wall boundary identified as a first vertical planar surface;
receiving, at the electronic device, a second infinite plane corresponding to a second wall boundary identified as a second vertical planar surface; and
identifying a room corner at an intersection of the first infinite plane and the second infinite plane.

7. An electronic device, comprising:
an imaging camera to capture imagery of an environment proximate the electronic device;
a processor to:
identify a plurality of feature points of the environment by performing raycasting based on imagery captured by the imaging camera and further based on received non-visual pose information obtained from non-visual sensors), the raycasting projecting a ray into the electronic device's view of the environment and returning feature points that the ray intersects, along with the pose of that intersection in world space, to determine whether a perceived plane passes through the feature points at the intersection of the ray, the raycasting resolving whether a ray along the normal from a screen of the electronic device returns a planar surface;
identify one or more planar surfaces based on the plurality of feature points;
compute a plurality of oriented point vectors associated with the one or more planar surfaces, comprising determining a surface normal to each of the identified plurality of feature points;
detect a vertical planar surface within the one or more planar surfaces based at least in part on one or more of the plurality of oriented point vectors being aligned in a direction perpendicular to gravity within a predetermined threshold; and
generate a floor plan of the environment proximate the electronic device based on one or more of the plurality of oriented point vectors corresponding to vertical planar surfaces.

8. The electronic device of claim 7 , wherein the plurality of feature points includes one or more features identifiable in imagery captured at the electronic device.

9. The electronic device of claim 7 , further comprising an inertial measurement unit for determining a gravity direction.

10. The electronic device of claim 7 , wherein the processor is to compute the plurality of oriented point vectors by:
determining, at the electronic device, a surface normal for each of the identified plurality of feature points.

11. The electronic device of claim 7 , wherein the processor is to compute the plurality of oriented point vectors by:
discarding a subset of the plurality of oriented point vectors, where the subset of the plurality of oriented point vectors are not directed towards the electronic device within a first predetermined threshold.

12. The electronic device of claim 7 , wherein the processor is to compute the plurality of oriented point vectors by:
discarding a subset of the plurality of oriented point vectors, where the subset of the plurality of oriented point vectors are not aligned in the direction perpendicular to gravity.

13. The electronic device of claim 7 , wherein the processor is to generate the floor plan by:
receiving a first infinite plane corresponding to a first wall boundary identified as a first vertical planar surface;
receiving a second infinite plane corresponding to a second wall boundary identified as a second vertical planar surface; and
identifying a room corner at an intersection of the first infinite plane and the second infinite plane.

14. A method, comprising:
identifying, at an electronic device by performing raycasting, a plurality of feature points of an environment proximate the electronic device based on imagery captured at the electronic device and based on received non-visual pose information obtained from non-visual sensors, the raycasting projecting rays into the electronic device's view of the environment and returning feature points that the rays intersect, along with the pose of that intersection in world space, to determine whether a perceived plane passes through the feature points at the intersection of the rays, the raycasting resolving whether a ray along the normal from a screen of the electronic device returns a planar surface;
computing, at the electronic device, a plurality of oriented point vectors associated with one or more planar surfaces, comprising determining a surface normal to each of the identified plurality of feature points;
detecting a vertical planar surface within the one or more planar surfaces based at least in part on one or more of the plurality of oriented point vectors being aligned in a direction perpendicular to gravity within a predetermined threshold;
displaying, at a display of the electronic device, a user prompt requesting confirmation that the detected vertical planar surface corresponds to a wall position to be added to a floor plan; and
in response to receiving a user confirmation, adding a wall to the floor plan at the wall position.

15. The method of claim 14 , further comprising:
receiving a first infinite plane corresponding to a first wall boundary identified as a first vertical planar surface;
receiving a second infinite plane corresponding to a second wall boundary

Assignees

• Google Llc

Inventors

• Prasso Luca
• Raphael Seth

Classifications

• G06V20/10 Primary

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title

• G06V10/40

  Extraction of image or video features · CPC title

• G06V20/20 Primary

  in augmented reality scenes · CPC title