Detection and replacement of transient obstructions from high elevation digital images

Patent metadata

Field	Value
Publication number	US-2019392596-A1
Application number	US-201816016495-A
Country	US
Kind code	A1
Filing date	Jun 22, 2018
Priority date	Jun 22, 2018
Publication date	Dec 26, 2019
Grant date	—

Abstract

Implementations relate to detecting/replacing transient obstructions from high-elevation digital images. A digital image of a geographic area includes pixels that align spatially with respective geographic units of the geographic area. Analysis of the digital image may uncover obscured pixel(s) that align spatially with geographic unit(s) of the geographic area that are obscured by transient obstruction(s). Domain fingerprint(s) of the obscured geographic unit(s) may be determined across pixels of a corpus of digital images that align spatially with the one or more obscured geographic units. Unobscured pixel(s) of the same/different digital image may be identified that align spatially with unobscured geographic unit(s) of the geographic area. The unobscured geographic unit(s) also may have domain fingerprint(s) that match the domain fingerprint(s) of the obscured geographic unit(s). Replacement pixel data may be calculated based on the unobscured pixels and used to generate a transient-obstruction-free version of the digital image.

First claim

What is claimed is:

1 . A method implemented using one or more processors, comprising:
obtaining a digital image of a geographic area captured from an elevated vantage point, wherein the digital image comprises a plurality of pixels that align spatially with a respective plurality of geographic units of the geographic area; identifying one or more obscured pixels of the digital image that align spatially with one or more obscured geographic units of the geographic area that are obscured in the digital image by one or more transient obstructions; determining, across pixels of a corpus of digital images that align spatially with the one or more obscured geographic units, one or more spectral-temporal data fingerprints of the one or more obscured geographic units; identifying one or more unobscured pixels of the same digital image or a different digital image that align spatially with one or more unobscured geographic units that are unobscured by transient obstructions, wherein the unobscured geographic units have one or more spectral-temporal data fingerprints that match the one or more spectral-temporal data fingerprints of the one or more obscured geographic units; calculating replacement pixel data based on the one or more unobscured pixels; and generating a transient-obstruction-free version of the digital image in which data associated with the one or more obscured pixels is replaced with the replacement pixel data.

2 . The method of claim 1 , wherein the digital image is captured by a satellite, and the elevated vantage point lies on a trajectory of the satellite.

3 . The method of claim 1 , wherein the digital image is captured by an unmanned aerial drone or manned aircraft, and the elevated vantage point lies on a trajectory of the unmanned aerial drone or manned aircraft.

4 . The method of claim 1 , wherein the one or more spectral-temporal data fingerprints of the one or more obscured geographic units include infrared.

5 . The method of claim 1 , wherein the one or more spectral-temporal data fingerprints of the one or more obscured geographic units include panchromatic.

6 . The method of claim 1 , further comprising applying the one or more spectral-temporal data fingerprints of the one or more obscured geographic units across a trained machine learning model to determine a terrain classification of the one or more obscured geographic units.

7 . The method of claim 6 , wherein identifying one or more unobscured pixels comprises determining that the terrain classification of the one or more obscured geographic units matches a terrain classification of the one or more unobscured geographic units.

8 . The method of claim 1 , wherein one or more of the operations of determining, identifying, and calculating includes sequentially applying data indicative of each digital image of the corpus of digital images as input across a recurrent neural network.

9 . The method of claim 1 , wherein the calculating is based at least in part on application of a generator model, and wherein the generator model is part of a generative adversarial network that also includes a discriminator model that is trained in conjunction with the generator model.

10 . The method of claim 9 , wherein the generator model and the discriminator model are trained using at least one training example comprising a high elevation digital image with synthetic transient obstructions added, wherein the synthetic transient obstructions are added by a transient obstruction generation model that is trained as part of another generative adversarial network.

11 . At least one non-transitory computer-readable medium comprising instructions that, in response to execution of the instructions by one or more processors, cause the one or more processors to perform the following operations:
obtaining a digital image of a geographic area captured from an elevated vantage point, wherein the digital image comprises a plurality of pixels that align spatially with a respective plurality of geographic units of the geographic area; identifying one or more obscured pixels of the digital image that align spatially with one or more obscured geographic units of the geographic area that are obscured in the digital image by one or more transient obstructions; determining, across pixels of a corpus of digital images that align spatially with the one or more obscured geographic units, one or more spectral-temporal data fingerprints of the one or more obscured geographic units; identifying one or more unobscured pixels of the same digital image or a different digital image that align spatially with one or more unobscured geographic units that are unobscured by transient obstructions, wherein the unobscured geographic units have one or more spectral-temporal data fingerprints that match the one or more spectral-temporal data fingerprints of the one or more obscured geographic units; calculating replacement pixel data based on the one or more unobscured pixels; and generating a transient-obstruction-free version of the digital image in which data associated with the one or more obscured pixels is replaced with the replacement pixel data.

12 . The at least one non-transitory computer-readable medium of claim 11 , wherein the digital image is captured by a satellite, and the elevated vantage point lies on a trajectory of the satellite.

13 . The at least one non-transitory computer-readable medium of claim 11 , wherein the digital image is captured by an unmanned aerial drone, and the elevated vantage point lies on a trajectory of the unmanned aerial drone.

14 . The at least one non-transitory computer-readable medium of claim 11 , wherein the one or more spectral-temporal data fingerprints of the one or more obscured geographic units include infrared.

15 . The at least one non-transitory computer-readable medium of claim 11 , wherein the one or more spectral-temporal data fingerprints of the one or more obscured geographic units include panchromatic.

16 . The at least one non-transitory computer-readable medium of claim 11 , further comprising applying the one or more spectral-temporal data fingerprints of the one or more obscured geographic units across a trained machine learning model to determine a terrain classification of the one or more obscured geographic units.

17 . The at least one non-transitory computer-readable medium of claim 16 , wherein identifying one or more unobscured pixels comprises determining that the terrain classification of the one or more obscured geographic units matches a terrain classification of the one or more unobscured geographic units.

18 . The at least one non-transitory computer-readable medium of claim 11 , wherein one or more of the operations of determining, identifying, and calculating include sequentially applying data indicative of each digital image of the corpus of digital images as input across a recurrent neural network.

19 . The at least one non-transitory computer-readable medium of claim 11 , wherein the calculating is based at least in part on application of a generator model, and wherein the generator model is part of a generative adversarial network that also includes a discriminator model that is trained in conjunction with the generator model.

20 . A system comprising one or more processors and memory operably coupled with the one or more processors, wherein the memory stores instructions that, in response to execution of the instructions by one or more processors, cause the one or more processor

Assignees

• X Dev Llc

Inventors

• Yang Jie
• Guo Cheng-En
• Grant Elliott

Classifications

• G06T7/337 Primary

  involving reference images or patches · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06F18/241

  relating to the classification model, e.g. parametric or non-parametric approaches · CPC title

• G06T2207/30188

  Vegetation; Agriculture · CPC title

• G06T2207/10036

  Multispectral image; Hyperspectral image · CPC title