Automated classification based on photo-realistic image/model mappings

Patent metadata

Field	Value
Publication number	US-10534962-B2
Application number	US-201715626104-A
Country	US
Kind code	B2
Filing date	Jun 17, 2017
Priority date	Jun 17, 2017
Publication date	Jan 14, 2020
Grant date	Jan 14, 2020

Abstract

Techniques are provided for increasing the accuracy of automated classifications produced by a machine learning engine. Specifically, the classification produced by a machine learning engine for one photo-realistic image is adjusted based on the classifications produced by the machine learning engine for other photo-realistic images that correspond to the same portion of a 3D model that has been generated based on the photo-realistic images. Techniques are also provided for using the classifications of the photo-realistic images that were used to create a 3D model to automatically classify portions of the 3D model. The classifications assigned to the various portions of the 3D model in this manner may also be used as a factor for automatically segmenting the 3D model.

First claim

What is claimed is:

1. A method comprising:
generating a 3D-model of a real-world space based on a collection of photo-realistic images, of the real-world space, that were captured in the real world;
based on spatial location and orientation information associated with a particular photorealistic image, determining a portion of the 3D-model to which a target region of the particular photo-realistic image maps;
based on spatial location and orientation information associated with photo-realistic images in the collection of photo-realistic images, determining a plurality of source-regions that map to the portion of the 3D-model; and
assigning a particular classification to the target region based, at least in part, on classifications assigned to the plurality of source-regions;
wherein the method is performed by one or more computing devices.

2. The method of claim 1 wherein assigning the particular classification to the target region comprises:
aggregating the classifications assigned to the plurality of source-regions to generate an aggregate classification;
assigning the aggregate classification to the portion of the 3D model; and
determining the particular classification based, at least in part, on the aggregate classification.

3. The method of claim 1 wherein at least one source-region of the plurality of source regions is a set of pixels within a photo-realistic image from the collection of photo-realistic Images.

4. The method of claim 1 wherein the real-world space is a building and the portion of the 3D-model is the portion of the 3D-model that represents a particular room of the house.

5. The method of claim 4 wherein:
the classifications assigned to the plurality of source regions are room-type classifications; and
the particular classification assigned to the target region is a room-type classification.

6. The method of claim 5 further comprising determining, based at least in part on the particular classification, that the particular photo-realistic image should be used as a best photo of the particular room.

7. The method of claim 6 further comprising including the particular photo-realistic image in a virtual tour of the building based on the particular photo-realistic image being determined to be a best photo of the particular room.

8. The method of claim 1 further comprising assigning a caption to the particular photorealistic image based on the particular classification.

9. The method of claim 1 wherein the target region is the entirety of the particular photorealistic image.

10. The method of claim 1 wherein the target region is a set of pixels within the particular photo-realistic image that has been assigned a particular initial classification.

11. The method of claim 1 wherein at least one source-region of the plurality of source regions is the entirety of a photo-realistic image from the collection of photo-realistic images.

12. The method of claim 1 wherein the particular photo-realistic image is a synthetic viewpoint rendered based on the 3D model.

13. The method of claim 1 further comprising:
automatically segmenting the 3D model based, at least in part, on aggregate classifications respectively assigned to a plurality of portions of the 3D model.

14. The method of claim 13 wherein:
the aggregate classification assigned to each of the plurality of portions is a room-type classification; and
automatically segmenting the 3D model comprises determining whether a particular part of the 3D model should be treated as a single room or multiple rooms based, at least in part, on the aggregate classifications assigned to the plurality of portions of the 3D model.

15. One or more non-transitory computer-readable media storing instructions which, when executed by one or more computing devices, cause:
generating a 3D-model of a real-world space based on a collection of photo-realistic images, of the real-world space, that were captured in the real world;
based on spatial location and orientation information associated with a particular photorealistic image, determining a portion of the 3D-model to which a target region of the particular photo-realistic image maps;
based on spatial location and orientation information associated with photo-realistic images in the collection of photo-realistic images, determining a plurality of source-regions that map to the portion of the 3D-model; and
assigning a particular classification to the target region based, at least in part, on classifications assigned to the plurality of source-regions.

16. The one or more non-transitory computer-readable media of claim 15 wherein assigning the particular classification to the target region comprises:
aggregating the classifications assigned to the plurality of source-regions to generate an aggregate classification;
assigning the aggregate classification to the portion of the 3D model; and
determining the particular classification based, at least in part, on the aggregate classification.

17. The one or more non-transitory computer-readable media of claim 15 wherein the instructions, when executed by one or more computing devices, further cause:
automatically segmenting the 3D model based, at least in part, on aggregate classifications respectively assigned to a plurality of portions of the 3D model.

18. The one or more non-transitory computer-readable media of claim 17 wherein:
the aggregate classification assigned to each of the plurality of portions is a room-type classification; and
automatically segmenting the 3D model comprises determining whether a particular part of the 3D model should be treated as a single room or multiple rooms based, at least in part, on the aggregate classifications assigned to the plurality of portions of the 3D model.

Assignees

• Matterport Inc

Inventors

• Hovden Gunnar
• Kurinnyy Mykhaylo
• Bell Matthew

Classifications

• G06V10/7796 Primary

  based on specific statistical tests · CPC title

• G06V20/35

  Categorising the entire scene, e.g. birthday party or wedding scene · CPC title

• G06T19/003

  Navigation within 3D models or images · CPC title

• G06T2210/04

  Architectural design, interior design · CPC title

• G06T17/00

  Three-dimensional [3D] modelling for computer graphics · CPC title