Automated determination of image acquisition locations in building interiors using determined room shapes

Patent metadata

Field	Value
Publication number	US-11481925-B1
Application number	US-202117201996-A
Country	US
Kind code	B1
Filing date	Mar 15, 2021
Priority date	Nov 23, 2020
Publication date	Oct 25, 2022
Grant date	Oct 25, 2022

Abstract

Techniques are described for computing devices to perform automated operations to determine the acquisition locations of images, such as within a building interior based on automatically determined shapes of rooms of the building, and for using the determined image acquisition location information in further automated manners. The image may be a panorama image or of another type (e.g., a rectilinear perspective image) and acquired at an acquisition location in a multi-room building's interior, and the determined acquisition location for such an image may be at least a location on the building's floor plan and optionally an orientation/direction for at least a part of the image—in addition, the automated image acquisition location determination may be further performed without having or using information from any depth sensors or other distance-measuring devices about distances from an image's acquisition location to walls or other objects in the surrounding building.

First claim

What is claimed is:

1. A computer-implemented method comprising:
receiving, by one or more computing systems, a target image acquired in a building that has multiple rooms, wherein the target image is a panorama image including 360 degrees of horizontal visual coverage around a vertical axis;
determining, by the one or more computing systems, an acquisition location of the target image on a floor plan of the building, including:
determining, by the one or more computing systems, three-dimensional (“3D”) room shapes for the multiple rooms, wherein each determined 3D room shape identifies at least walls and corners of an associated one of the multiple rooms;
generating, by the one or more computing systems and based at least in part on analyzing visual contents of the target image, an estimated 3D room shape for the target image that identifies at least walls and corners visible in the target image, and an indicated position within the estimated 3D room shape of the target image, and information about vanishing line angles for the target image that are based at least in part on the walls visible in the target image;
determining, by the one or more computing systems, a plurality of alternative shape matches for the estimated 3D room shape that each includes a placement of one of the determined 3D room shapes relative to the estimated 3D room shape, the placement using the vanishing line angles for the target image and additional vanishing line angles associated with the one determined 3D room shape to match at least one wall and corner of the estimated 3D room shape to at least one other wall and corner of the one determined 3D room shape;
selecting, by the one or more computing systems, one of the plurality of alternative shape matches based at least in part on a measurement of a fit for the selected one alternative shape match of the walls and corners of the estimated 3D room shape to the walls and corners of its one determined 3D room shape using the placement for the selected one alternative shape match, and identifying the room of the multiple rooms whose determined room shape is the one determined 3D room shape for the selected one alternative shape match;
locating, by the one or more computing systems and within the identified room by using the placement for the selected one alternative shape match of the one determined 3D room shape relative to the estimated 3D room shape, the indicated position within the estimated 3D room shape on that placement, and identifying the acquisition location of the target image on the floor plan based on the located indicated position within the identified room; and

displaying, by the one or more computing systems, the floor plan of the building with a visual indication of the determined acquisition location of the target image in the identified room.

2. The computer-implemented method of claim 1 wherein the determining of the 3D room shapes for the multiple rooms includes obtaining two-dimensional (“2D”) outlines of the multiple rooms from a floor plan for the building, and generating the 3D room shapes for the multiple rooms based at least in part on the obtained 2D outlines and on information about heights of walls of the multiple rooms.

3. The computer-implemented method of claim 1 wherein the determining of the acquisition location of the target image on the floor plan includes determining, by the one or more computing systems, an orientation of the target image at the determined acquisition location that identifies a direction in at least two dimensions of at least a starting view from the determined acquisition location, and wherein the displaying of the floor plan further includes displaying, by the one or more computing systems and on a client device, a visual indication of the determined orientation at the determined acquisition location.

4. A computer-implemented method comprising:
receiving, by one or more computing systems, a target image acquired in a building that has multiple rooms, wherein the target image is a panorama image having visual coverage of multiple surrounding walls;
determining, by the one or more computing systems, an acquisition location of the target image on a floor plan of the building, including:
obtaining, by the one or more computing systems, determined three-dimensional (“3D”) room shapes for the multiple rooms;
generating, by the one or more computing systems and based at least in part on analyzing visual contents of the target image, an estimated 3D room shape visible in the target image and an indicated position within the estimated 3D room shape of the target image;
selecting, by the one or more computing systems, one of a plurality of alternative shape matches for the estimated 3D room shape based at least in part on a degree of match determined for the selected one alternative shape match, wherein the alternative shape matches each includes a placement of one of the determined 3D room shapes relative to the estimated 3D room shape and has a determined degree of match that is based at least in part on a fit in the placement of the one determined 3D room shape to the estimated 3D room shape, and wherein the one determined 3D room shape for the selected one alternative shape match is for an identified room of the multiple rooms;
locating, by the one or more computing systems, and within the identified room by using the placement for the selected one alternative shape match of the determined 3D room shape relative to the estimated 3D room shape, the indicated position within the estimated 3D room shape on that placement, and identifying the acquisition location of the target image on the floor plan based on the located indicated position within the identified room; and

providing, by the one or more computing systems, information about the determined acquisition location of the target image in the identified room, to enable display of the determined acquisition location on the floor plan of the building.

5. The computer-implemented method of claim 4 wherein the determining of the acquisition location of the target image on the floor plan includes determining, by the one or more computing systems, an orientation of the target image at the determined acquisition location that identifies a direction in at least two dimensions of at least a starting view from the determined acquisition location, and wherein the providing of the information includes presenting, by the one or more computing systems and on a client device, a display of at least some of the floor plan of the building that includes the identified room and that is overlaid with visual indications of the determined acquisition location and of the determined orientation.

6. The computer-implemented method of claim 4 wherein the generating of the estimated 3D room shape includes determining, by the one or more computing systems, room shape information of a surrounding room around the acquisition location of the target image that includes at least one of a 3D point cloud modeling at least some walls of the surrounding room, or planar surfaces representing the at least some walls of the surrounding room, or wireframe structural lines representing boundaries of the at least some walls of the surrounding room, and that further includes structural wall elements of the surrounding room including at least windows and doorways and non-doorway wall openings, and using the determined room shape information for the estimated 3D room shape.

7. The computer-implemented method of claim 6 wherein the generating of the estimated 3D room shape includes using information about an acquisition time of the target image and further includes determining, by the one or more computing systems, additional room shape information that includes at least one of 3D bounding b

Assignees

• Zillow Inc

Inventors

• Li Yuguang
• Hutchcroft Will Adrian
• Khosravan Naji
• Boyadzhiev Ivaylo

Classifications

• H04N23/698

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/30244

  Camera pose · CPC title

• G06T7/74 Primary

  involving reference images or patches · CPC title