Capturing Environmental Features Using 2D and 3D Scans

Patent metadata

Field	Value
Publication number	US-2023083703-A1
Application number	US-202117459084-A
Country	US
Kind code	A1
Filing date	Sep 15, 2021
Priority date	Sep 15, 2021
Publication date	Mar 16, 2023
Grant date	—

Abstract

During capture of a three dimensional mesh of an environment, a combination of user inputs and machine learning is used to place annotations within the three dimensional mesh environment interactively. When the full mesh is assembled, the annotations are used to detect which portions of the three-dimensional mesh make up the features of interest. Additional features can be derived from these results, such as a 2D image of a floor plan.

First claim

We claim:

1 . A method of detecting a region of interest in an image of a space, the method comprising: obtaining a plurality of depth measurements captured by a depth sensor attached to a device, the plurality of depth measurements defining distances from the depth sensor to respective points in the space using a three dimensional grid;
declaring those depth measurements a portion of the space; obtaining a detected region of interest in the image of the portion of the space, captured by a machine learning recognition system using two dimensions of the three dimensional grid; mapping the detected region of interest onto the three dimensional grid in two dimensions to make a two dimensional detected region of interest; obtaining a finished depth measurement three dimensional grid of the space; projecting the detected region of interest onto the finished depth measurement three dimensional grid of the space to make a projected three dimensional shape; and discovering an intersection between the projected three dimensional shape and the finished depth measurement three dimensional grid of the space as a region of interest.

2 . The method of claim 1 , wherein the machine learning recognition system is associated with a 2D camera and wherein the two dimensions of the three dimensional grid are transformed from a 2D grid created by the 2D camera.

3 . The method of claim 2 , wherein obtaining a detected region of interest in the portion of the space comprises accepting an annotation placement on a surface of a representation of the 2D grid.

4 . The method of claim 3 , wherein projecting the two dimensional detected region of interest onto the finished depth measurement three dimensional grid of the space to make a projected three dimensional shape comprises creating a ray that intersects the three dimensional grid using the annotation placement, creating several intersection points.

5 . The method of claim 4 , wherein adjacent faces of the intersection point are identified to create a wall portion.

6 . The method of claim 5 , wherein the wall portion is flattened along a z axis, with a marker pointing to an inside of the space, creating a 2D marker oriented in space.

7 . The method of claim 6 , wherein there are multiple 2D markers and wherein the multiple 2-D markers oriented in space are extended to create a 2-D outline of the space.

8 . The method of claim 3 , further comprising placing a marker on the surface of a representation of the 2D grid marking a wall associated with the annotation placement.

9 . The method of claim 1 , wherein obtaining a detected region of interest in the portion of the space comprises capturing the detected region of interest using a two dimensional camera with image recognition software using a transformation of two dimensions of the three dimensional grid.

10 . The method of claim 9 , wherein capturing the detected region of interest using a two dimensional camera with image recognition software using two dimensions of the three dimensional grid further comprises capturing at least three points that correspond to object corners.

11 . A system to generate a 3D scan of a space with regions of interest marked, the system comprising:
a 3D scanner operationally able to scan a portion of a space in three dimensions creating a 3D scan portion, and operationally able to scan a whole space creating a whole 3D scan; a 2D scanner operationally connected to the 3D scanner, the 2D scanner operationally able to use machine learning to detect regions of interest within a 2D scan and operationally able to mark the regions of interest in the 2D scan using two of the three dimensions from the 3D scan portion; a combiner operationally able to mark the regions of interest within the 2D scan with two dimensions of the 3D scan portion to create a ray; and a post processor operationally able to position the ray with the whole 3D scan to create a 3D scan of a space with the regions of interest marked.

12 . The system of claim 11 , wherein the region of interest comprise walls, doors, windows, or equipment.

13 . The system of claim 12 , wherein the post processor is operationally able to determine a wall portion by intersecting the ray with the whole 3D scan along a third dimension axis.

14 . A non-transitory computer readable storage medium storing instructions for capturing of regions of interest wherein the instructions when executed by a processor, cause the processor to perform steps including:
receiving, by a LiDAR system mounted on a device, a partial 3D scan of a space; receiving by a 2D camera mounted on the device a 2D detected region of interest; marking the 2D detected region of interest within the partial 3D scan creating a 2D marked ROI; receiving a finished 3D scan; and marking an intersection of the 2D marked ROI in the finished 3D scan as a marked region of interest.

15 . The non-transitory computer readable storage medium of claim 14 , wherein receiving by a 2D camera mounted on the device a 2D detected region of interest comprises accepting 2D user input marking the 2D detected region of interest.

16 . The non-transitory computer readable storage medium of claim 15 , further comprising: receiving by a 2D camera mounted on the device a second 2D detected region of interest;
marking the second 2D detected region of interest within the partial 3D scan in two dimensions; intersecting the second 2D detected region of interest with the finished 3D scan creating a second 3D scan intersection; marking the second 3D scan intersection as a second marked 2D region of interest; extending the 2D detected region of interest and the marked second 2D detected region of interest to an intersection point to create an extended 2D detected region of interest and a second extended 2D detected region of interest; and marking the extended 2D detected region of interest and the second extended 2D detected region of interest as a wall floor plan portion.

17 . The non-transitory computer readable storage medium of claim 16 , wherein intersecting the 2D detected region of interest with the finished 3D scan creating a 3D scan intersection further comprises identifying adjacent faces with a similar orientation within the finished 3D scan; and wherein intersecting the 2D detected region of interest with the finished 3D scan creating a 3D scan intersection comprises flattening the wall floor plan portion into a 2D marker.

18 . The non-transitory computer readable storage medium of claim 17 , wherein receiving e a 2D detected region of interest comprises a machine learning image recognition system using a camera feed from the 2D camera mounted on the device to detect a region of interest.

19 . The non-transitory computer readable storage medium of claim 18 , wherein intersecting the second 2D detected region of interest with the finished 3D scan creating a second 3D scan intersection comprises:
locating a surface on the finished 3D scan; creating a 3D extension of three points of user interest marking the region around a ray at 90° to the surface of the finished 3D scan; and marking an intersection of the 3D extension of the three points of user interest with the finished 3D scan as the region of interest.

20 . The non-transitory computer readable storage medium of claim 14 , wherein the 2D detects region of interest is a 2D representation of a wall, a door, a window, a bookcase, a desk, a light fixture, a table, or a sensor.

Assignees

• Passivelogic Inc

Inventors

• Meiners Justin

Classifications

• G06V20/64

  Three-dimensional [3D] objects · CPC title

• G06V10/25

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T17/00 Primary

  Three-dimensional [3D] modelling for computer graphics · CPC title

• G06T2219/004

  Annotating, labelling · CPC title