Exterior hybrid photo mapping

Patent metadata

Field	Value
Publication number	US-2016307328-A1
Application number	US-201615195921-A
Country	US
Kind code	A1
Filing date	Jun 28, 2016
Priority date	Sep 27, 2013
Publication date	Oct 20, 2016
Grant date	—

Abstract

Embodiments disclosed pertain to the use of user equipment (UE) for the generation of a 3D exterior envelope of a structure based on captured images and a measurement set associated with each captured image. In some embodiments, a sequence of exterior images of a structure is captured and a corresponding measurement set comprising Inertial Measurement Unit (IMU) measurements, wireless measurements (including Global Navigation Satellite (GNSS) measurements) and/or other non-wireless sensor measurements may be obtained concurrently. A closed-loop trajectory of the UE in global coordinates may be determined and a 3D structural envelope of the structure may be obtained based on the closed loop trajectory and feature points in a subset of images selected from the sequence of exterior images of the structure.

First claim

What is claimed is:

1 . A method on a device, the method comprising:
capturing a plurality of images of a structure when traversing a plurality of locations in a vicinity of the structure; capturing a plurality of measurement sets, wherein each measurement set corresponds to at least one image in the plurality of images; estimating, based, in part, on the plurality of images and the corresponding plurality of measurement sets, a 3D structural envelope of the structure and a trajectory of the device; transmitting information comprising the estimated trajectory of the device, the 3D structural envelope, the plurality of images, and the corresponding measurement sets, to a first server wirelessly coupled to the device; and receiving, based, in part, on the transmitted information, a corrected trajectory of the device registered to absolute coordinates and a 3D structural envelope of the structure registered to the absolute coordinates.

2 . The method of claim 1 , wherein each measurement set comprises Inertial Measurement Unit (IMU) measurements.

3 . The method of claim 2 , wherein each measurement set comprises available wireless measurements with correction information for the available wireless measurements.

4 . The method of claim 3 , wherein the available wireless measurements comprise:
Global Navigation Satellite System (GNSS) measurements comprising differentially corrected code and carrier phase observables and wherein the correction information for the wireless measurements comprises one or more of: GNSS code and carrier differential corrections, GNSS precise orbital and clock information, or GNSS atmospheric corrections.

5 . The method of claim 4 , wherein estimating the trajectory of the device comprises:
applying, for each location in a subset of the plurality of locations, one or more of: the GNSS code and carrier differential corrections, the GNSS precise orbital and clock information, or the GNSS atmospheric corrections to the GNSS measurements corresponding to the location to obtain a corresponding corrected GNSS measurement; determining a set of corrected locations, wherein, for each location in the subset of the plurality of images, a corresponding corrected location in the set of corrected locations is obtained based on the corrected GNSS measurements; and determining the estimated trajectory based, in part, on the set of corrected locations.

6 . The method of claim 3 , wherein the available wireless measurements further comprise
Wireless Wide Area Network (WWAN) measurements comprising one or more of:
Observed Time Difference of Arrival (OTDOA) measurements, or
Reference Signal Time Difference (RSTD) measurements, or
Advanced Forward Link Trilateralation (AFLT) measurements, or
hybrid-AFLT measurements.

7 . The method of claim 3 , wherein the correction information for the available wireless measurements comprises antenna pattern information for the device.

8 . The method of claim 1 , wherein the corrected trajectory of the device registered to the absolute coordinates is determined based, in part, on a closed-loop trajectory of the device.

9 . The method of claim 1 , wherein receiving the corrected trajectory of the device registered to the absolute coordinates and the 3D structural envelope of the structure registered to the absolute coordinates comprises:
receiving an outdoor map comprising the corrected trajectory and the 3D structural envelope, wherein the outdoor map comprises a plurality of map layers registered to the absolute coordinates.

10 . The method of claim 9 , wherein the plurality of map layers includes at least two of:
a plan view map layer; or a 3D structural map layer, with external 3D structural envelope information for neighboring structures; a map layer with attenuation or delay of WWAN signals associated with the structure, or a map layer with annotations for any variation in WWAN signal strength in the vicinity of the structure; or a Line Of Sight (LOS) map layer, indicating, for each absolute coordinate in a first plurality of the absolute coordinates on the LOS map layer, corresponding antennas in a line of sight relative to the absolute coordinate.

11 . The method of claim 1 , wherein estimating the trajectory of the device and the 3D structural envelope of the structure comprises:
applying Visual Simultaneous Localization and Mapping (VSLAM) techniques to a subset of the plurality of images of the structure to determine a scene geometry and a set of first 6 Degrees Of Freedom (6DOF) poses of the device, wherein each first 6DOF pose of the device in the set of first 6DOF poses corresponds to a distinct image in the subset of the plurality of images each first 6DOF pose being determined based on keypoints in the subset of the plurality of images; and determining the external 3D structural envelope of the structure based, in part, on the scene geometry; and determining the estimated trajectory based, in part, on the set of first 6DOF poses.

12 . The method of claim 11 , wherein determining the estimated trajectory based on the set of first 6DOF poses comprises:
providing each first 6DOF pose in the set of first 6DOF poses to an Extended Kalman Filter (EKF), wherein for each first 6DOF pose, the EKF determines a corresponding second 6DOF pose of the device in the absolute coordinates, based in part, on the first 6DOF pose, Inertial Measurement Unit (IMU) measurements, and available wireless measurements, wherein each measurement set comprises IMU measurements and available wireless measurements with correction information for the available wireless measurements.

13 . A device comprising:
a camera configured to capture a plurality of images of a structure when traversing a plurality of locations in a vicinity of the structure; one or more sensors; a processor, coupled to the camera and the one or more sensors wherein the processor is configured to: determine a plurality of measurement sets, wherein each measurement set comprises measurements by the one or more sensors and each measurement set corresponds to at least one image in the plurality of images; estimate, based, in part, on the plurality of images and the corresponding plurality of measurement sets, a 3D structural envelope of the structure and a trajectory of the device; transmitting information comprising the estimated trajectory of the device, and the 3D structural envelope, the plurality of images, and the corresponding measurement sets, to a first server wirelessly coupled to the device; and receive, based, in part, on the transmitted information, a corrected trajectory of the device registered to absolute coordinates and a 3D structural envelope of the structure registered to the absolute coordinates.

14 . The device of claim 13 , wherein:
the one or more sensors comprise an Inertial Measurement Unit (IMU); and each measurement set comprises IMU measurements.

15 . The device of claim 14 , further comprising:
a wireless measurement circuit coupled to the processor, wherein the wireless measurement circuit is configured to take wireless measurements of available wireless signals, and acquire correction information for the wireless measurements; and each measurement set further comprises the wireless measurements with the correction information for the wireless measurements.

16 . The device of claim 15 , wherein the wireless measurements comprise:
Global Navigation Satellite System (GNSS) measurements comprising differentially corrected code and carrier phase observables and wherein the correction information for

Assignees

• Qualcomm Inc

Inventors

• Moeglein Mark Leo
• Brunner Christopher
• Ramanandan Arvind
• Ramachandran Mahesh
• Tyagi Abhishek
• Chari Murali Ramaswamy

Classifications

• G06T7/579

  from motion · CPC title

• H04W4/021

  Services related to particular areas, e.g. point of interest [POI] services, venue services or geofences · CPC title

• G06T2207/30244

  Camera pose · CPC title

• G01S5/0294 Primary

  Trajectory determination or predictive filtering, e.g. target tracking or Kalman filtering · CPC title

• H04W4/029

  Location-based management or tracking services · CPC title