Methods for localization using geotagged photographs and three-dimensional visualization

Patent metadata

Field	Value
Publication number	US-9892558-B2
Application number	US-201615047655-A
Country	US
Kind code	B2
Filing date	Feb 19, 2016
Priority date	Feb 19, 2016
Publication date	Feb 13, 2018
Grant date	Feb 13, 2018

Abstract

Methods for identifying parts of a target object (e.g., an airplane) using geotagged photographs captured on site by a hand-held imaging device. The geotagged photographs contain GPS location data and camera setting information. The embedded image metadata from two or more photographs is used to estimate the location (i.e., position and orientation) of the imaging device relative to the target object, which location is defined in the coordinate system of the target object. Once the coordinates of the area of interest on the target object are known, the part number and other information associated with the part can be determined when the imaging device viewpoint information is provided to a three-dimensional visualization environment that has access to three-dimensional models of the target object.

First claim

The invention claimed is:

1. A method for identifying a part of a target object, comprising:
(a) acquiring reference location image data, first geolocation data and first heading angle data using an imaging device directed toward the target object and located at a known reference location relative to a coordinate system of the target object;
(b) acquiring feature location image data, second geolocation data and second heading angle data using an imaging device directed toward an area on the target object and located at an unknown feature location relative to the coordinate system of the target object;
(c) computing an offset in Cartesian coordinates between the feature location and the reference location, wherein the offset is a function of coordinates in said first and second geolocation data and heading angles in said first and second heading angle data;
(d) retrieving three-dimensional model data of the target object from a three-dimensional model database;
(e) displaying a three-dimensional model image of at least a portion of the target object with a viewpoint that is a function of the offset computed in step (c) and the camera field-of-view angle, said three-dimensional model image representing said three-dimensional model data retrieved from the three-dimensional model database;
(f) displaying a feature location image of the same portion of the target object as an overlay or background to said three-dimensional model image, said feature location image representing at least portions of said feature location image data;
(g) selecting a point in the displayed feature location image;
(h) retrieving additional three-dimensional model data from the three-dimensional model database in response to said selecting; and
(i) displaying symbology representing the retrieved additional three-dimensional model data.

2. The method as recited in claim 1 , wherein the additional three-dimensional model data comprises part identification data that identifies a part of the target object associated with an area of the three-dimensional model image containing the selected point.

3. The method as recited in claim 1 , wherein the additional three-dimensional model data comprises coordinates of the selected point defined in the coordinate system of the target object.

4. The method as recited in claim 1 , wherein step (c) comprises computing an offset transformation matrix that represents a location offset between the imaging device at the reference location and the imaging device at the feature location, wherein the offset transformation matrix is a function of at least coordinates in said first and second geolocation data and heading angles in said first and second heading angle data, and the viewpoint is a function of the offset transformation matrix.

5. The method as recited in claim 1 , further comprising adjusting a position of the three-dimensional model image to align the three-dimensional model image with the feature location image on the display screen.

6. The method as recited in claim 1 , wherein said displayed feature location image comprises an outline extracted from said feature location image data using edge detection.

7. The method as recited in claim 1 , wherein the target object is an airplane.

8. The method as recited in claim 1 , wherein a position of the reference location is collocated with an origin of the coordinate system of the target object.

9. The method as recited in claim 1 , wherein steps (c)-(f), (h) and (i) are performed by the imaging device.

10. The method as recited in claim 1 , further comprising sending said reference and feature location image data, said first and second geolocation data, and said first and second heading angle data from the imaging device to a computer system, wherein steps (c)-(f), (h) and (i) are performed by the computer system.

11. A method for identifying a part of a target object, comprising:
(a) acquiring reference location image data, first geolocation data and first heading angle data using an imaging device directed toward the target object and located at a known reference location relative to a coordinate system of the target object;
(b) constructing a reference location data file containing at least said reference location image data, said first geolocation data and said first heading angle data;
(c) acquiring feature location image data, second geolocation data and second heading angle data using an imaging device directed toward an area on the target object and located at an unknown feature location relative to the coordinate system of the target object;
(d) constructing a feature location data file containing at least said feature location image data, said second geolocation data and said second heading angle data;
(e) sending the reference and feature location data files from the imaging device to a computer system;
(f) computing an offset between the feature location and the reference location, wherein the offset is a function of coordinates in said first and second geolocation data and heading angles in said first and second heading angle data;
(g) retrieving three-dimensional model data of the target object from a three-dimensional model database;
(h) displaying a three-dimensional model image of at least a portion of the target object with a viewpoint that is a function of the offset computed in step (f) and the field-of-view angle of the imaging device, said three-dimensional model image representing said three-dimensional model data retrieved from the three-dimensional model database; and
(i) displaying a feature location image of the same portion of the target object as an overlay or background to said three-dimensional model image, said feature location image representing at least portions of said feature location image data,
wherein steps (f) through (i) are performed by the computer system.

12. The method as recited in claim 11 , further comprising:
(g) selecting a point in said displayed feature location image;
(h) retrieving additional three-dimensional model data from the three-dimensional model database in response to said selecting; and
(i) displaying symbology representing the retrieved additional three-dimensional model data.

13. The method as recited in claim 12 , wherein the additional three-dimensional model data comprises part identification data that identifies a part of the target object associated with an area of the three-dimensional model image containing the selected point.

14. The method as recited in claim 12 , wherein the additional three-dimensional model data comprises coordinates of the selected point in the coordinate system of the target object.

15. The method as recited in claim 12 , wherein step (f) comprises computing an offset transformation matrix that represents a location offset between the imaging device at the reference location and the imaging device at the feature location, wherein the offset transformation matrix is a function of at least coordinates in said first and second geolocation data and heading angles in said first and second heading angle data, and the viewpoint is a function of the offset transformation matrix.

16. The method as recited in claim 11 , wherein said feature location image comprises an outline extracted from said feature location image data using edge detection.

17. A system comprising a digital camera, a GPS receiver, a digital compass, a display screen, a non-transitory tangible computer-readable storage medium storing three-dimensional model data of a target object, a three-dimensional visualization software application, and an offset

Assignees

• Boeing Co

Inventors

• Troy James J
• Karakusevic Vladimir
• Esposito Christopher D

Classifications

• G06F16/587

  using geographical or spatial information, e.g. location · CPC title

• G06Q10/20 Primary

  Administration of product repair or maintenance · CPC title

• G06T2219/2004

  Aligning objects, relative positioning of parts · CPC title

• B64F5/0081

  Operations & Transport · mapped topic

• G06F17/30268

  Physics · mapped topic