System and method for real-time overlay of map features onto a video feed

Patent metadata

Field	Value
Publication number	US-2017039765-A1
Application number	US-201514703633-A
Country	US
Kind code	A1
Filing date	May 4, 2015
Priority date	May 5, 2014
Publication date	Feb 9, 2017
Grant date	—

Abstract

A method is provided for augmenting video feed obtained by a camera of a aerial vehicle to a user interface. The method can include obtaining a sequence of video images with or without corresponding sensor metadata from the aerial vehicle; obtaining supplemental data based on the sequence of video images and the sensor metadata; correcting an error in the sensor metadata using a reconstruction error minimization technique; creating a geographically-referenced scene model based on a virtual sensor coordinate system that is registered to the sequence of video images; overlaying the supplemental information onto the geographically-referenced scene model by rendering geo-registered data from a 3D perspective that matches a corrected camera model; creating a video stream of a virtual representation from the scene from the perspective of the camera based on the overlaying; and providing the video stream to a UI to be render onto a display.

First claim

What is claimed is:

1 . A method for providing an augmented video feed obtained by a camera of a manned or unmanned aerial vehicle (“UAV”) to a user interface (“UI”), the method comprising:
obtaining a sequence of video images with or without corresponding sensor metadata from the aerial vehicle;
obtaining supplemental data based on the sequence of video images and the sensor metadata;
correcting, by a processor, an error in the sensor metadata using a reconstruction error minimization technique;
creating, by a processor, a geographically-referenced scene model based on a virtual sensor coordinate system that is registered to the sequence of video images;
overlaying the supplemental information onto the geographically-referenced scene model by rendering geo-registered data from a 3D perspective that matches a corrected camera model;
creating a video stream of a virtual representation from the scene from the perspective of the camera based on the overlaying; and
providing the video stream to a UI to be render onto a display.

2 . The method of claim 1 , wherein the supplemental data comprises one or more of: static geo-referenced datasets, dynamical geo-referenced datasets, traffic conditions, elevation data, terrain data, social media information, waypoint data, light detection and ranging (“LIDAR”) data, airspace symbology data, 3D model data, and road maps.

3 . The method of claim 1 , wherein the error in the sensor metadata comprises one or more of: missing data, temporal drift, and spatial drift.

4 . The method of claim 1 , wherein the correcting the error in the sensor metadata is performed for each frame in the sequence of video images, wherein a correction of one video image is used to refine the correction for a subsequent video image.

5 . The method of claim 1 , wherein the correcting the error in the sensor metadata for one image frame is based on another image frame or map data corresponding to a scene that is represented in the one image.

6 . The method of claim 1 , further comprising:
constructing a depth map using light detection and ranging (“LIDAR”) or digital elevation maps (“DEM”); determining that one or more pixels representing overlay objects has low or no visibility for a video frame based the depth map; and rendering the one or more pixels in a manner to represent that the one or more pixels are occluded.

7 . The method of claim 1 , wherein the correcting the error in the sensor metadata further comprises:
performing a first registration between a video image in the sequence of video images and a corresponding map data; determining an anchor frame from the sequence of video images; and performing a second registration between the video image and the corresponding map data using the anchor frame by minimizing a reprojection error.

8 . The method of claim 7 , wherein the minimizing the reprojection error is performed using the reconstruction error minimization technique according to:
min( H ) R g(v,m,H) =min( H )Σ j=1 n (| p j v −Hp j m |+|p j m −H −1 p j v |)/ n , where
subject to H to be close to rigid body geometry, and v is a frame of input image, m is cropped map imagery, R g is reconstruction error of feature points, j where j=1, . . . , n, where n is a number of corresponding points.

9 . The method of claim 1 , wherein the reconstruction error is minimized using a Lagrange optimization technique to obtain a final homography between map data and the video images.

10 . The method of claim 1 , wherein the supplemental data comprises information beyond a field-of-view of the camera of the aerial vehicle.

11 . The method of claim 1 , further comprising:
obtaining a series of waypoint data, wherein the series comprises a beginning waypoint, one or more intermediate waypoints, and a destination waypoint; generating a flight path based on the series of waypoint data; and outputting the flight path to the UI.

12 . The method of claim 1 , further comprising generating the UI that shows the supplemental data overlaid over the video stream and the sensor metadata.

13 . A device for providing an augmented video feed obtained by a camera of a manned or unmanned aerial vehicle (“UAV”) to a user interface (“UI”), the device comprising:
a memory containing instructions; and
at least one processor, operably connected to the memory, the executes the instructions to perform a method for providing an augmented video feed obtained by a camera of a manned or unmanned aerial vehicle (“UAV”) to a user interface (“UI”), comprising:
obtaining a sequence of video images with or without corresponding sensor metadata from the aerial vehicle;
obtaining supplemental data based on the sequence of video images and the sensor metadata;
correcting, by a processor, an error in the sensor metadata using a reconstruction error minimization technique;
creating, by a processor, a geographically-referenced scene model based on a virtual sensor coordinate system that is registered to the sequence of video images;
overlaying the supplemental information onto the geographically-referenced scene model by rendering geo-registered data from a 3D perspective that matches a corrected camera model;
creating a video stream of a virtual representation from the scene from the perspective of the camera based on the overlaying; and
providing the video stream to a UI to be render onto a display.

14 . The device of claim 13 , wherein the at least one processor is further operable to perform the method comprising:
constructing a depth map using light detection and ranging (“LIDAR”) or digital elevation maps (“DEM”); determining that one or more pixels representing overlay objects has low or no visibility for a video frame based the depth map; and rendering the one or more pixels in a manner to represent that the one or more pixels are occluded.

15 . The device of claim 14 , wherein the correcting the error in the sensor metadata further comprises:
performing a first registration between a video image in the sequence of video images and a corresponding map data; determining an anchor frame from the sequence of video images; and performing a second registration between the video image and the corresponding map data using the anchor frame by minimizing a reprojection error.

16 . The device of claim 15 , wherein the minimizing the reprojection error is performed using the reconstruction error minimization technique according to:
min( H ) R g(v,m,H) =min( H )Σ j=1 n (| p j v −Hp j m |+|p j m −H −1 p j v |)/ n , where
subject to H to be close to rigid body geometry, and v is a frame of input image, m is cropped map imagery, R g is reconstruction error of feature points, j where j=1, . . . , n, where n is a number of corresponding points.

17 . The device of claim 16 , wherein the reconstruction error is minimized using a Lagrange optimization technique to obtain a final homography between map data and the video images.

18 . A computer readable storage medium comprising instructions for causing one or more processors to perform a method, the method for discerning a vehicle at an access control point, the device comprising:
obtaining a sequence of video images with or without corresponding sensor metadata from the aerial vehicle; obtaining supplemental data based on the sequence of video images and the sensor metadata; correcting, by a processor, an error in the sensor metadata using a reconstruction error minimization technique; creating,

Assignees

• Avigilon Fortress Corp

Inventors

• Zhou Shirley
• Madden Don
• Choe Tae Eun
• Scanlon Andrew W

Classifications

• G06T7/344

  involving models · CPC title

• G06T7/521

  from laser ranging, e.g. using interferometry; from the projection of structured light · CPC title

• G06T7/55

  from multiple images · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T2207/10032

  Satellite or aerial image; Remote sensing · CPC title