Three-dimensional room model generation using panorama paths and photogrammetry

Patent metadata

Field	Value
Publication number	US-11024079-B1
Application number	US-201916298934-A
Country	US
Kind code	B1
Filing date	Mar 11, 2019
Priority date	Mar 11, 2019
Publication date	Jun 1, 2021
Grant date	Jun 1, 2021

Abstract

Systems and methods related to an image capture process using panorama paths may include traversing a user device among a plurality of image capture locations of a room, sweeping the user device at each of the image capture locations, capturing imaging data using the user device during the traversal and/or during the sweep, and processing the imaging data using photogrammetry. The imaging data may be captured using an imaging sensor associated with the user device, and the imaging data may be processed based on data received from position and orientation sensors associated with the user device. In addition, a three-dimensional model of the room may be generated based on the imaging data.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
receiving, from a user device, measurement data associated with a room;
receiving, from the user device, movement data of the user device within the room;
generating for presentation, via a display of the user device, a panorama path comprising a plurality of locations within the room based on the measurement data and the movement data;
instructing traversal of the user device in a defined order among the plurality of locations of the panorama path;
during traversal of the user device, capturing, via an imaging sensor of the user device, a first plurality of images of the room;
receiving, from a sensor of the user device, position and orientation data of the user device associated with respective ones of the first plurality of images captured during the traversal of the user device;
at individual locations of the plurality of locations, instructing sweep of the user device in a defined direction, the sweep comprising a movement of the user device that is separate from the traversal of the user device among the plurality of locations of the panorama path, and the sweep further comprising an arc from a sweep starting point to a sweep ending point associated with respective individual locations;
during sweep of the user device, capturing, via the imaging sensor of the user device, a second plurality of images of the room;
receiving, from the sensor of the user device, position and orientation data of the user device associated with respective ones of the second plurality of images captured during the sweep of the user device;
processing the first plurality of images and the second plurality of images of the room using photogrammetry; and
generating a three-dimensional model of the room based on the first plurality of images, the second plurality of images, the position and orientation data of the user device associated with respective ones of the first plurality of images, and the position and orientation data of the user device associated with respective ones of the second plurality of images.

2. The computer-implemented method of claim 1 , further comprising:
receiving dimensions associated with the room;
wherein the plurality of locations are generated based at least in part on the dimensions associated with the room.

3. The computer-implemented method of claim 1 , wherein the defined order for traversal of the user device among the plurality of locations is, from a top-down view of the room, a counterclockwise order; and
wherein the defined direction for sweep of the user device is, from a user viewpoint, a left-to-right direction.

4. The computer-implemented method of claim 1 , further comprising:
at individual locations of the plurality of locations, instructing a plurality of sweeps of the user device.

5. A method, comprising:
generating for presentation, via a display associated with a user device, a plurality of locations within a space;
instructing traversal of the user device among the plurality of locations;
during traversal of the user device among the plurality of locations, capturing, via a first sensor associated with the user device, a first plurality of images of the space;
receiving, from a second sensor associated with the user device, data associated with position and orientation of the user device associated with respective ones of the first plurality of images captured during the traversal of the user device among the plurality of locations;
at individual locations of the plurality of locations, capturing, via the first sensor associated with the user device, a second plurality of images of the space during sweep of the user device at individual locations, the sweep comprising a movement of the user device that is separate from the traversal of the user device among the plurality of locations, and the sweep further comprising a rotation between a sweep starting point and a sweep ending point associated with respective individual locations; and
causing a three-dimensional model of the space to be generated based on the first plurality of images, the second plurality of images, and the data associated with position and orientation of the user device associated with respective ones of the first plurality of images.

6. The method of claim 5 , wherein generating the plurality of locations within the space further comprises:
receiving dimensions associated with the space; and
generating the plurality of locations based at least in part on the dimensions associated with the space.

7. The method of claim 5 , wherein generating the plurality of locations within the space further comprises:
generating a defined order for traversal of the user device among the plurality of locations;
wherein the defined order is at least one of, from a top-down view of the space, a counterclockwise order or a clockwise order.

8. The method of claim 5 , wherein individual locations of the plurality of locations are associated with at least one of an edge, a corner, a side, or a vertical boundary of the space.

9. The method of claim 5 , further comprising:
at individual locations of the plurality of locations, instructing sweep of the user device to capture the second plurality of images.

10. The method of claim 5 , further comprising:
at individual locations of the plurality of locations, receiving, from the second sensor associated with the user device, data associated with position and orientation of the user device associated with respective ones of the second plurality of images captured during the sweep of the user device; and
wherein the three-dimensional model of the space is further generated based on the data associated with position and orientation of the user device associated with respective ones of the second plurality of images.

11. The method of claim 5 , wherein generating the plurality of locations within the space further comprises:
generating a defined direction for sweep of the user device at individual locations of the plurality of locations;
wherein the defined direction is at least one of, from a user viewpoint, a left-to-right direction or a right-to-left direction.

12. The method of claim 5 , wherein capturing the second plurality of images of the space during sweep of the user device further comprises:
capturing the second plurality of images between a first adjacent vertical boundary in a first direction and a second adjacent vertical boundary in a second, opposite direction.

13. The method of claim 5 , wherein capturing the second plurality of images of the space during sweep of the user device further comprises:
capturing the second plurality of images such that individual ones of the second plurality of images includes at least approximately 30% overlap with at least one other image by modifying at least one of a frame rate of the first sensor or a rate of movement of the first sensor.

14. The method of claim 5 , wherein capturing the second plurality of images of the space during sweep of the user device further comprises:
receiving, from the second sensor associated with the user device, movement data associated with the sweep of the user device; and
capturing individual ones of the second plurality of images at angular orientations that are at least approximately five degrees different from at least one other image of the second plurality of images.

15. The method of claim 9 , wherein instructing sweep of the user device further comprises:
at individual locations of the plurality of locations, instructing a plurality of sweeps of the user device.

16. The me

Assignees

• Amazon Tech Inc

Inventors

• Chuah Joonhao
• Baker Dillon
• Kane Kristian
• Kim Hai Quang
• Mount Brian
• Thomas Michael

Classifications

• G06T17/00

  Three-dimensional [3D] modelling for computer graphics · CPC title

• G06T7/246 Primary

  using feature-based methods, e.g. the tracking of corners or segments · CPC title

• G06T2200/24

  involving graphical user interfaces [GUIs] · CPC title

• G06T17/20 Primary

  Finite element generation, e.g. wire-frame surface description, {tesselation} · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title