Augmented video system providing enhanced situational awareness

Patent metadata

Field	Value
Publication number	US-9380275-B2
Application number	US-201313754795-A
Country	US
Kind code	B2
Filing date	Jan 30, 2013
Priority date	Jan 30, 2013
Publication date	Jun 28, 2016
Grant date	Jun 28, 2016

Abstract

A facility, comprising systems and methods, for providing enhanced situational awareness to captured image data is disclosed. The disclosed techniques are used in conjunction with image data, such as a real-time or near real-time image stream captured by a camera attached to an unmanned system, previously captured image data, rendered image data, etc. The facility enhances situational awareness by projecting overlays onto captured video data or “wrapping” captured image data with previously-captured and/or “synthetic world” information, such as satellite images, computer-generated images, wire models, textured surfaces, and so on. The facility also provides enhanced zoom techniques that allow a user to quickly zoom in on an object or area of interest using a combined optical and digital zoom technique. Additionally, the facility provides a digital lead indicator designed to reduce operator-induced oscillations in commanding an image capturing device.

First claim

We claim:

1. A method for providing enhanced situational awareness to image data captured by an image capturing device carried by an unmanned aircraft vehicle system, the method comprising:
storing, for at least one of a plurality of objects of interest, position data for the at least one object of interest;
storing, for at least one of a plurality of geographic areas, topographical data for the at least one geographic area;
receiving image data captured by the image capturing device carried by the unmanned aircraft vehicle, wherein at least one of a plurality of frames of the received image data has associated metadata comprising a timestamp, an orientation of the unmanned aircraft vehicle, an orientation of the image capturing device relative to the unmanned aircraft vehicle, configuration information for the image capturing device, a position for the unmanned aircraft vehicle, and a velocity for the unmanned aircraft vehicle; and
for at least one of a plurality of frames of the received image data,
determining a transformation matrix based on metadata associated with the frame, and
for each of at least one overlay,
applying the determined transformation matrix data to the overlay to transform the overlay, and
providing the transformed overlay for display in association with a display of the frame.

2. The method of example 1 wherein the topographical data includes at least one of wire frame data, map data, or terrain contour data.

3. The method of example 1, further comprising:
storing, for at least one of a plurality of geographic areas, bathymetric data for the at least one geographic area.

4. The method of claim 1 , further comprising:
in response to receiving a request to change the orientation of the image capturing device,
determining a horizontal field of view for the image capturing device,
determining a vertical field of view for the image capturing device,
determining a horizontal slew rate for changing the orientation of the image capturing device,
determining a vertical slew rate for changing the orientation of the image capturing device,
determining a width of a displayed image,
determining a height of a displayed image,
determining an advance period,
calculating a coordinate based on the determined horizontal field of view, the determined vertical field of view, the determined horizontal slew rate, the determined vertical slew rate, the determined width, the determined height, and the determined advance period, and
providing for display an indication of a digital lead indicator at the calculated coordinate.

5. The method of claim 1 , further comprising:
in response to receiving a request to perform an optical zoom operation,
digitally zooming a video frame captured by the image capturing device,
providing for display at least a portion of the digitally-zoomed video frame in place of a displayed video frame,
determining whether the image capturing device has completed the requested optical zoom operation, and
in response to determining that the image capturing device has completed the requested optical zoom operation, replacing a digitally-zoomed video frame with a video frame captured by the image capturing device.

6. The method of claim 1 wherein the at least one overlay includes an object of interest overlay, a map annotation overlay, a heading and groundspeed overlay, a flight path overlay, and a waypoint and route overlay.

7. The method of claim 1 wherein the at least one overlay includes an engine out boundaries overlay, a terrain contour overlay, a map overlay, a building overlay, and a city block overlay.

8. The method of claim 1 wherein the at least one overlay includes a 3D airspace overlay, a detectability overlay, an automatic identification system (AIS) overlay, and an approach information overlay.

9. A computer-readable medium that is not a transitory, propagating signal, storing instructions that, when executed by a computing system having a processor, cause the computing system to perform operations comprising:
storing terrain information for a geographic region;
identifying a first image stream captured by a first image capturing device and including images of a first area within the geographic region;
identifying a second image stream other than the first image stream captured by a second image capturing device other than the first image capturing device and including images of a second area within the geographic region other than the first area within the geographic region;
rendering a hybrid environment comprising at least a portion of the stored terrain information for the geographic region, at least one image from the first image stream, and at least one image from the second image stream, wherein the rendering comprises:
determining a position of a virtual camera relative to the hybrid environment,
determining a position of the first image capturing device relative to the geographic region,
generating a first transformation matrix based on the determined position of the virtual camera relative to the hybrid environment and the determined position of the first image capturing device relative to the geographic region,
applying the generated first transformation matrix to images of the first stream to generate first transformed frames,
determining a position of the second image capturing device relative to the geographic region,
generating a second transformation matrix based on the determined position of the virtual camera relative to the hybrid environment and the determined position of the second image capturing device relative to the geographic region,
applying the generated second transformation matrix to images of the second image stream to generate second transformed frames,
adding at least one of the generated first transformed frames and at least one of the generated second transformed frames to a rendering of the stored terrain information for the geographic region.

10. The computer-readable medium that is not a transitory, propagating signal, of claim 9 wherein the first image capturing device is carried by an aircraft vehicle system and wherein the rendered hybrid environment includes a graphical representation of the aircraft vehicle system at a location that approximates the actual location of the aircraft vehicle system relative to the geographic region.

11. The computer-readable medium that is not a transitory, propagating signal, of claim 9 , the operations further comprising:
for each of a plurality of overlays,
for each of a plurality of objects of the overlay,
transforming a position of the object into screen coordinates, and
providing an indication of the object for display at the screen coordinates.

12. The computer-readable medium that is not a transitory, propagating signal, of claim 11 , the operations further comprising:
for at least one overlay,
for at least one object of the overlay,
receiving, from a user, a label for the object,
storing the received label, and
providing for display an indication of the received label.

13. A computing system comprising:
at least one or more processors;
a display device configured to display at least a portion of a video frame captured by an image capturing device;
at least one processor, of the one or more processors, configured to receive a request to perform an optical zoom-in operation of the image capturing device;
at least one processor, of the one or more processors, configured to initiate the requested optical zoom-in operation of the image capturing device;
at least one processor, of the one or more processors, configured to while the image capturing device is performing

Assignees

• Insitu Inc

Inventors

• Davidson Jr Darcy
• Trowbridge Theodore T

Classifications

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• H04N7/183 Primary

  for receiving images from a single remote source · CPC title

• G06T17/05 Primary

  Geographic models · CPC title

• G06T11/00

  Two-dimensional [2D] image generation · CPC title

• G06T11/60

  Creating or editing images; Combining images with text · CPC title