Systems and methods for generating three-dimensional reconstructions of environments

Patent metadata

Field	Value
Publication number	US-11928830-B2
Application number	US-202117645591-A
Country	US
Kind code	B2
Filing date	Dec 22, 2021
Priority date	Dec 22, 2021
Publication date	Mar 12, 2024
Grant date	Mar 12, 2024

Abstract

Disclosed are methods and systems for generating three-dimensional reconstructions of environments. A system, for example, may include a housing having an image sensor directed in a first direction and a distance sensor directed in a second direction and a control unit including a processor and a memory storing instructions. The processor may be configured to execute the instructions to: generate a first 3D model of an environment; generate a plurality of revolved 3D models by revolving the first 3D model relative to the image sensor to a plurality of positions within a predetermined angular range; match a set of distance values to one of the revolved 3D models; determine an angular position of the second direction relative to the first direction; and generate a 3D reconstruction of the environment.

First claim

What is claimed is:

1. A system comprising:
a housing including an image sensor directed in a first direction and a distance sensor directed in a second direction, wherein during movement of the housing through an environment the image sensor is configured to generate an image sequence and the distance sensor is configured to generate a set of distance values, wherein the second direction is within a predetermined angular range of the first direction;
a control unit including a processor and a memory storing instructions, wherein the processor is configured to execute the instructions to:
generate, based on the image sequence, a first 3D model of the environment;
generate a first set of revolved 3D models by revolving the first 3D model relative to the image sensor to a first plurality of positions within the predetermined angular range;
match the set of distance values to at least one of the first set of revolved 3D models;
determine, based on the matched at least one of the first set of revolved 3D models, a first angular position of the second direction relative to the first direction; and
generate, based on the first 3D model, the set of distance values, and the first angular position, a 3D reconstruction of the environment, the 3D reconstruction including information indicative of a scale of the environment.

2. The system of claim 1 , wherein the image sensor is incorporated within a monocular camera and the distance sensor is a single beam laser distancer including a laser emitter and a laser receiver.

3. The system of claim 1 , wherein the memory further stores a machine learning model trained to learn associations between at least (i) a training set of image sequences and (ii) a training set of 3D models, each image sequences of the training set of image sequences corresponding to one or more 3D model of the training set of 3D models; and
the processor is configured to generate the first 3D model of the environment using the machine learning model.

4. The system of claim 3 , wherein the machine learning model is a convolutional neural network.

5. The system of claim 1 , wherein the revolved 3D models are generated by incrementally revolving the 3D model relative to the image sensor to a predetermined number of positions extending substantially through the predetermined angular range.

6. The system of claim 1 , wherein the revolved 3D models are generated by:
incrementally revolving the first 3D model relative to the image sensor about a first axis to a first predetermined number of positions; and
incrementally revolving the first 3D model relative to the image sensor about a second axis to a second predetermined number of positions;
wherein the first axis is orthogonal to the second axis.

7. The system of claim 6 , wherein the first 3D model and the revolved 3D models are 3D point clouds.

8. The system of claim 1 , wherein the processor is further configured to execute the instructions to determine the angular position by:
generating a second set of revolved 3D models by revolving the matched at least one of the first set of revolved 3D models relative to the image sensor to a second plurality of positions, the second plurality of positions existing in a fine angular range within the predetermined angular range; and
matching the set of distance values to at least one of the second set of revolved 3D models.

9. The system of claim 1 , wherein the processor is further configured to execute the instructions generate the 3D reconstruction in part by fusing the first 3D model and the set of distance values.

10. The system of claim 9 , wherein the processor is further configured to execute the instructions to compensate for the angular position during generation of the 3D reconstruction.

11. A method comprising:
moving a housing through an environment, the housing including an image sensor directed in a first direction and a distance sensor directed in a second direction, wherein the second direction is within a predetermined angular range of the first direction;
during movement of the housing through the environment, generating an image sequence using the image sensor and generating a set of distance values using the distance sensor;
generating, using a processor and based on the image sequence, a first 3D model of the environment;
generating, using the processor based on the first 3D model and the predetermined angular range, a plurality of revolved 3D models by revolving the first 3D model relative to the image sensor to a plurality of positions within the predetermined angular range;
matching, using the processor, the set of distance values to at least one of the revolved 3D models;
determining, using the processor and based on the matched at least one of the revolved 3D models, an angular position of the second direction relative to the first direction; and
generating, using the processor and based on the first 3D model, the set of distance values, and the angular position, a 3D reconstruction of the environment, the 3D reconstruction including information indicative of a scale of the environment.

12. The method of claim 11 , wherein the image sensor is incorporated within a monocular camera and the distance sensor is a single beam laser distancer including a laser emitter and a laser receiver.

13. The method of claim 11 , wherein the processor is operatively connected to a memory storing instructions and a machine learning model trained to learn associations between at least (i) a training set of image sequences and (ii) a training set of 3D models, each image sequence of the training set of image sequences corresponding to one or more 3D model of the training set of 3D models; and
generating the first 3D model of the environment is performed by the processor using the machine learning model.

14. The method of claim 11 , wherein generating the revolved 3D models further comprises incrementally revolving the 3D model relative to the image sensor to a predetermined number of positions extending substantially through the predetermined angular range.

15. The method of claim 11 , wherein generating the revolved 3D models further comprises:
incrementally revolving the first 3D model relative to the image sensor about a first axis to a first predetermined number of positions; and
incrementally revolving the first 3D model relative to the image sensor about a second axis to a second predetermined number of positions;
wherein the first axis is orthogonal to the second axis.

16. The method of claim 15 , wherein the first 3D model and the revolved 3D models are 3D point clouds.

17. The method of claim 11 , wherein determining the angular position comprises:
generating, using the processor, a second set of revolved 3D models by revolving the matched at least one of the first set of revolved 3D models relative to the image sensor to a second plurality of positions, the second plurality of positions existing in a fine angular range within the predetermined angular range; and
matching, using the processor, the set of distance values to at least one of the second set of revolved 3D models.

18. The method of claim 11 , wherein generating the 3D reconstruction comprises:
fusing the first 3D model and the set of distance values.

19. The method of claim 18 , wherein generating the 3D reconstruction further comprises:
compensating for the angular position during generation of the 3D reconstruction.

20. A non-transitory computer readable medium storing instructions that, when executed by a pro

Assignees

• Honeywell Int Inc

Inventors

• Ren Zhiguo
• Speranzon Alberto
• Dins Carl
• Hu Juan
• Dai Zhiyong
• Venkataraman Vijay

Classifications

• G06T7/521 Primary

  from laser ranging, e.g. using interferometry; from the projection of structured light · CPC title

• G06T7/593

  from stereo images · CPC title

• G06T17/10

  Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T2207/20081

  Training; Learning · CPC title