Geospatial image processing for targeted data acquisition

Patent metadata

Field	Value
Publication number	US-11967041-B2
Application number	US-202117323358-A
Country	US
Kind code	B2
Filing date	May 18, 2021
Priority date	May 18, 2021
Publication date	Apr 23, 2024
Grant date	Apr 23, 2024

Abstract

A computer implemented method includes obtaining data for raw image frames captured by a moving camera. The raw image frames are indexed geographically, and a graph is created from the multiple raw image frames. The graph includes image frames as vertices and edges that represent image frames having overlapping image information. The method further includes skipping frames based on the amount of overlap, determining a frame having an interesting feature, using the graph to find additional raw image frames that have the interesting feature, combining multiple raw image frames to form a unique image frame, and transmitting the unique image frame.

First claim

The invention claimed is:

1. A computer implemented method comprising:
obtaining at an edge device, data for raw image frames captured by a moving camera supported by a drone controlled to survey an area of interest, the data including geographic information for each raw image frame, the edge device performing:
indexing the raw image frames geographically based on the geographic information;
creating a graph from the multiple raw image frames, the graph comprising indexed image frames as vertices and edges connecting the vertices representing the image frames having overlapping image information, each edge including a weight representative of an amount of image overlap between connected vertices, wherein the weight is determined based on the geographic information;
skipping frames based on an amount of overlap threshold;
determining a remaining frame having an interesting feature;
using the graph to find additional raw image frames that have the interesting feature;
combining multiple of the additional raw image frames with the remaining frame to form a unique image frame; and
transmitting the unique image frame.

2. The method of claim 1 wherein a domain specific inferencing model is used to determine a frame having the interesting feature.

3. The method of claim 1 wherein combining multiple raw image frames to form the unique image frame includes stitching the multiple raw image frames together.

4. The method of claim 1 wherein the edges represent whether or not the raw image frame has already been used to form a unique image frame.

5. The method of claim 4 wherein determining a frame having the interesting feature comprises discovering a frame from an unexplored area of interest by performing a random walk on an area tracking data structure based on the edge representation of whether or not the raw image frame has already been used.

6. The method of claim 4 and further comprising determining additional unprocessed frames having interesting features and using such additional frames to generate further unique image frames.

7. The method of claim 6 and further comprising using a domain specific inferencing model to sort the unique image frames in order of relative importance for an application.

8. The method of claim 1 and further comprising receiving drone control actions at the edge device in response to the transmitted unique frame.

9. The method of claim 1 wherein determining a frame having an interesting feature comprises discovering a frame from an unexplored area of interest by performing a random walk on an area tracking data structure.

10. The method of claim 1 wherein skipping frames comprises processing skipped frames as a background task to perform:
determining a skipped frame having an interesting feature;
using the graph to find additional raw image frames that have the interesting feature;
combining multiple raw image frames to form a unique image frame; and
transmitting the unique image frame.

11. A machine-readable storage device having instructions for execution by a processor of a machine to cause the processor to perform operations to perform a method, the operations comprising:
obtaining at an edge device, data for raw image frames captured by a moving camera supported by a drone controlled to survey an area of interest, the data including geographic information for each raw image frame, the edge device performing:
indexing the raw image frames geographically based on the geographic information;
creating a graph from the multiple raw image frames, the graph comprising indexed image frames as vertices and edges connecting the vertices representing the image frames having overlapping image information, each edge including a weight representative of an amount of image overlap between connected vertices, wherein the weight is determined based on the geographic information;
skipping frames based on an amount of overlap threshold;
determining a remaining frame having an interesting feature;
using the graph to find additional raw image frames that have the interesting feature;
combining multiple of the additional raw image frames with the remaining frame to form a unique image frame; and
transmitting the unique image frame.

12. The device of claim 11 wherein a domain specific inferencing model is used to determine a frame having the interesting feature.

13. The device of claim 11 wherein combining multiple raw image frames to form the unique image frame includes stitching the multiple raw image frames together.

14. The device of claim 11 wherein the edges represent whether or not the raw image frame has already been used to form a unique image frame.

15. The device of claim 11 wherein determining a frame having the interesting feature comprises discovering a frame from an unexplored area of interest by performing a random walk on an area tracking data structure based on the edge representation of whether or not the raw image frame has already been used.

16. The device of claim 11 and wherein the operations further comprise determining additional unprocessed frames having interesting features;
using such additional frames to generate further unique image frames; and
using a domain specific inferencing model to sort the unique image frames in order of relative importance for an application.

17. A device comprising:
a processor; and
a memory device coupled to the processor and having a program stored thereon for execution by the processor to perform operations comprising:
obtaining at an edge device, data for raw image frames captured by a moving camera supported by a drone controlled to survey an area of interest, the data including geographic information for each raw image frame, the edge device performing:
indexing the raw image frames geographically based on the geographic information;
creating a graph from the multiple raw image frames, the graph comprising indexed image frames as vertices and edges connecting the vertices representing the image frames having overlapping image information, each edge including a weight representative of an amount of image overlap between connected vertices, wherein the weight is determined based on the geographic information;
skipping frames based on an amount of overlap threshold;
determining a remaining frame having an interesting feature;
using the graph to find additional raw image frames that have the interesting feature;
combining multiple of the additional raw image frames with the remaining frame to form a unique image frame; and
transmitting the unique image frame.

18. The device of claim 17 wherein a domain specific inferencing model is used to determine a frame having the interesting feature, wherein combining multiple raw image frames to form the unique image frame includes stitching the multiple raw image frames together, and wherein the edges represent whether or not the raw image frame has already been used to form a unique image frame.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Badam Anirudh
• Chandra Ranveer

Classifications

• G06T11/26

  Drawing of charts or graphs · CPC title

• G06T3/4038 Primary

  Image mosaicing, e.g. composing plane images from plane sub-images · CPC title

• B64C39/024

  of the remote controlled vehicle type, i.e. RPV · CPC title

• B64D47/08

  Arrangements of cameras · CPC title

• G06T11/206

  Physics · mapped topic