Automated determination of image acquisition locations in building interiors using multiple data capture devices

Patent metadata

Field	Value
Publication number	US-11632602-B2
Application number	US-202217573465-A
Country	US
Kind code	B2
Filing date	Jan 11, 2022
Priority date	Jan 8, 2021
Publication date	Apr 18, 2023
Grant date	Apr 18, 2023

Abstract

Techniques are described for using multiple devices to automatically determine the acquisition location of an image from a camera device, such as within a building interior and by using data from the multiple devices (e.g., visual data from the camera device's image, and additional data captured by a separate mobile computing device in the same area as the camera device), and for subsequently using determined image acquisition location information in one or more further automated manners. The image may be a panorama image or of another type, and the determined acquisition location for such an image may be at least a location on the building's floor plan—in addition, the automated image acquisition location determination may be further performed without having or using information from any depth sensors or other distance-measuring devices about distances from an image's acquisition location to walls or other objects in the surrounding building.

First claim

What is claimed is:

1. A computer-implemented method comprising:
obtaining, by one or more computing devices, one or more panorama images having visual data captured by a camera device in one or more rooms of a building, and one or more additional images having additional visual data captured in the one or more rooms by a mobile computing device separate from the camera device;
determining, by the one or more computing devices and for each of the one or more rooms, a shape of that room by using a combination of the visual data of the one or more panorama images captured by the camera device and the additional visual data of the one or more additional images captured by the mobile computing device;
determining, by the one or more computing devices and for each of the one or more rooms,
a first location of the mobile computing device in that room at which at least one of the additional images was captured based at least in part on additional visual data of the at least one additional image, and
a second location of the camera device in that room at which at least one of the panorama images was captured, wherein the second location in that room of the camera device is determined relative to the first location in that room of the mobile computing device and based at least in part on some or all of the camera device being visible in the additional visual data of the at least one additional image captured in that room by the mobile computing device;

generating, by the one or more computing devices, mapping information for the one or more rooms that includes, for each of the one or more rooms, a position within the determined shape of that room of the at least one panorama image captured in that room, including using the second location of the camera device in that room as part of determining the position of the at least one panorama image captured in that room; and
displaying, by the one or more computing devices, the generated mapping information for the one or more rooms, including the determined shape of each of the one or more rooms with one or more overlaid visual indications to show the position of the at least one panorama image in each of the one or more rooms.

2. A non-transitory computer-readable medium having stored contents that cause one or more computing devices to perform automated operations including at least:
obtaining, by the one or more computing devices, one or more panorama images having visual data captured by a camera device in one or more rooms of a building, and one or more additional images having additional visual data captured in the one or more rooms by a mobile computing device separate from the camera device, wherein movements of the camera device and the mobile computing device are independent of each other;
determining, by the one or more computing devices and for each of the one or more rooms, a shape of that room by using at least one of the visual data of the one or more panorama images captured by the camera device or the additional visual data of the one or more additional images captured by the mobile computing device;
determining, by the one or more computing devices and for each of the one or more rooms,
a first location of the mobile computing device in that room at which at least one of the additional images was captured based at least in part on additional visual data of the at least one additional image, and
a second location of the camera device in that room at which at least one of the panorama images was captured, wherein the second location in that room of the camera device is determined relative to the first location in that room of the mobile computing device;

generating, by the one or more computing devices, mapping information for the one or more rooms that includes, for each of the one or more rooms, a position within the determined shape of that room of the at least one panorama image captured in that room, including using the second location of the camera device in that room as part of determining the position of the at least one panorama image captured in that room; and
providing, by the one or more computing devices, the generated mapping information for the one or more rooms, to enable display of the determined shape of each of the one or more rooms with one or more overlaid visual indications to show the position of the at least one panorama image in each of the one or more rooms.

3. The non-transitory computer-readable medium of claim 2 wherein the obtaining of the one or more panorama images involves obtaining multiple panorama images by the camera device that include at least one panorama image in each of two or more rooms of the building, wherein the obtaining of the one or more additional images involves obtaining multiple additional images by the mobile computing device that include at least one additional image in each of the two or more rooms, and wherein the generating of the mapping information includes generating, by the one or more computing devices, at least a partial floor plan for the building that includes determined shapes of the two or more rooms positioned relative to each other.

4. The non-transitory computer-readable medium of claim 3 wherein the mobile computing device is the one or more computing devices performing the automated operations, and wherein the stored contents include software instructions that, when executed by the mobile computing device, cause the mobile computing device to perform further automated operations including at least:
capturing, by the mobile computing device, movement data during travel between the two or more rooms; and
performing, by the mobile computing device, positioning of the determined shapes of the two or more rooms relative to each other based at least in part on the captured movement data,
and wherein the providing of the generated mapping information includes displaying, by the mobile computing device, the generated at least partial floor plan for the building to a user, to enable identification of additional areas of the building at which to acquire additional panorama images.

5. The non-transitory computer-readable medium of claim 2 wherein the generating of the mapping information includes generating at least a partial floor plan for the building having representations of at least two rooms of the building, wherein the providing of the generated mapping information includes displaying, by the one or more computing devices, the generated at least partial floor plan to a user, and wherein the automated operations further include, after the displaying of the generated at least partial floor plan:
receiving, by the one or more computing devices, an indication from the user of an additional room of the building separate from the at least two rooms;
obtaining, by the one or more computing devices, a further panorama image that is captured in the additional room by the camera device, and a further additional image that is captured in the additional room by the mobile computing device, and further motion data that is captured in the additional room from one or more inertial measurement unit (IMU) sensors of the mobile computing device;
determining, by the one or more computing devices, a position of the further panorama image within a shape of the additional room that is determined based on a combination of visual data of the further panorama image and of additional visual data of the further additional image and of the further motion data that is captured in the additional room;
updating, by the one or more computing devices, the at least partial floor plan to position the shape of the additional room relative to at least one of the positioned determined shapes of the at least two rooms; and
displaying, by the one or more computing devices, the updated at least partial floor plan for the bui

Assignees

• Mftb Holdco Inc
• Mfib Holdco Inc

Inventors

• Cier Sean P
• Kang Sing Bing
• Dawson Mitchell David
• Moulon Pierre

Classifications

• G06T2207/20081

  Training; Learning · CPC title

• G06T7/579

  from motion · CPC title

• H04N23/6812

  based on additional sensors, e.g. acceleration sensors · CPC title

• G06T7/73

  using feature-based methods · CPC title

• H04N23/698 Primary

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title