Detection and replacement of transient obstructions from high elevation digital images

Patent metadata

Field	Value
Publication number	US-10878588-B2
Application number	US-201816016495-A
Country	US
Kind code	B2
Filing date	Jun 22, 2018
Priority date	Jun 22, 2018
Publication date	Dec 29, 2020
Grant date	Dec 29, 2020

Abstract

Implementations relate to detecting/replacing transient obstructions from high-elevation digital images. A digital image of a geographic area includes pixels that align spatially with respective geographic units of the geographic area. Analysis of the digital image may uncover obscured pixel(s) that align spatially with geographic unit(s) of the geographic area that are obscured by transient obstruction(s). Domain fingerprint(s) of the obscured geographic unit(s) may be determined across pixels of a corpus of digital images that align spatially with the one or more obscured geographic units. Unobscured pixel(s) of the same/different digital image may be identified that align spatially with unobscured geographic unit(s) of the geographic area. The unobscured geographic unit(s) also may have domain fingerprint(s) that match the domain fingerprint(s) of the obscured geographic unit(s). Replacement pixel data may be calculated based on the unobscured pixels and used to generate a transient-obstruction-free version of the digital image.

First claim

What is claimed is:

1. A method implemented using one or more processors, comprising:
obtaining a first digital image of a geographic area captured from an elevated vantage point, wherein the first digital image comprises a plurality of pixels that correspond geographically to a respective plurality of geographic units of the geographic area;
identifying one or more obscured pixels of the first digital image that correspond geographically to one or more obscured geographic units of the geographic area that are obscured in the first digital image by one or more transient obstructions;
determining, across pixels of a temporal sequence of digital images that correspond geographically to the one or more obscured geographic units, one or more spectral-temporal data fingerprints of the one or more obscured geographic units, wherein for each obscured geographic unit of the one or more obscured geographic units, the corresponding spectral-temporal fingerprint comprises a sequence of pixel values for the obscured geographic unit across the temporal sequence of digital images;
identifying, from the first digital image or a second digital image, one or more unobscured pixels that correspond geographically to one or more unobscured geographic units that are unobscured by transient obstructions in the first or second digital image, wherein the one or more unobscured geographic units are spatially different than the one or more obscured geographic units of the geographic area, and wherein the one or more unobscured pixels are identified based on the one or more unobscured pixels having one or more spectral-temporal data fingerprints that match the one or more spectral-temporal data fingerprints of the one or more obscured geographic units;
calculating replacement pixel data based on the one or more unobscured pixels; and
generating a transient-obstruction-free version of the first digital image in which data associated with the one or more obscured pixels is replaced with the replacement pixel data.

2. The method of claim 1 , wherein the first digital image is captured by a satellite, and the elevated vantage point lies on a trajectory of the satellite.

3. The method of claim 1 , wherein the first digital image is captured by an unmanned aerial drone or manned aircraft, and the elevated vantage point lies on a trajectory of the unmanned aerial drone or manned aircraft.

4. The method of claim 1 , wherein the one or more spectral-temporal data fingerprints of the one or more obscured geographic units include infrared.

5. The method of claim 1 , wherein the one or more spectral-temporal data fingerprints of the one or more obscured geographic units include panchromatic.

6. The method of claim 1 , further comprising applying the one or more spectral-temporal data fingerprints of the one or more obscured geographic units across a trained machine learning model to determine a terrain classification of the one or more obscured geographic units.

7. The method of claim 6 , wherein identifying one or more unobscured pixels comprises determining that the terrain classification of the one or more obscured geographic units matches a terrain classification of the one or more unobscured geographic units.

8. The method of claim 1 , comprising sequentially applying data indicative of each digital image of the temporal sequence of digital images as input across a recurrent neural network.

9. The method of claim 1 , wherein the calculating is based at least in part on application of a generator model, and wherein the generator model is part of a generative adversarial network that also includes a discriminator model that is trained in conjunction with the generator model.

10. The method of claim 9 , wherein the generator model and the discriminator model are trained using at least one training example comprising a high elevation digital image with synthetic transient obstructions added, wherein the synthetic transient obstructions are added by a transient obstruction generation model that is trained as part of another generative adversarial network.

11. At least one non-transitory computer-readable medium comprising instructions that, in response to execution of the instructions by one or more processors, cause the one or more processors to perform the following operations:
obtaining a first digital image of a geographic area captured from an elevated vantage point, wherein the first digital image comprises a plurality of pixels that correspond geographically to a respective plurality of geographic units of the geographic area;
identifying one or more obscured pixels of the first digital image that correspond geographically to one or more obscured geographic units of the geographic area that are obscured in the first digital image by one or more transient obstructions;
determining, across pixels of a temporal sequence of digital images that correspond geographically to the one or more obscured geographic units, one or more spectral-temporal data fingerprints of the one or more obscured geographic units, wherein for each obscured geographic unit of the one or more obscured geographic units, the corresponding spectral-temporal fingerprint comprises a sequence of pixel values for the obscured geographic unit across the temporal sequence of digital images;
identifying, from the first digital image or a second digital image, one or more unobscured pixels that correspond geographically to one or more unobscured geographic units that are unobscured by transient obstructions in the first or second digital image, wherein the one or more unobscured geographic units are spatially different than the one or more obscured geographic units of the geographic area, and wherein the one or more unobscured pixels are identified based on the one or more unobscured pixels having one or more spectral-temporal data fingerprints that match the one or more spectral-temporal data fingerprints of the one or more obscured geographic units;
calculating replacement pixel data based on the one or more unobscured pixels; and
generating a transient-obstruction-free version of the first digital image in which data associated with the one or more obscured pixels is replaced with the replacement pixel data.

12. The at least one non-transitory computer-readable medium of claim 11 , wherein the first digital image is captured by a satellite, and the elevated vantage point lies on a trajectory of the satellite.

13. The at least one non-transitory computer-readable medium of claim 11 , wherein the first digital image is captured by an unmanned aerial drone, and the elevated vantage point lies on a trajectory of the unmanned aerial drone.

14. The at least one non-transitory computer-readable medium of claim 11 , wherein the one or more spectral-temporal data fingerprints of the one or more obscured geographic units include infrared.

15. The at least one non-transitory computer-readable medium of claim 11 , wherein the one or more spectral-temporal data fingerprints of the one or more obscured geographic units include panchromatic.

16. The at least one non-transitory computer-readable medium of claim 11 , further comprising applying the one or more spectral-temporal data fingerprints of the one or more obscured geographic units across a trained machine learning model to determine a terrain classification of the one or more obscured geographic units.

17. The at least one non-transitory computer-readable medium of claim 16 , wherein identifying one or more unobscured pixels comprises determining that the terrain classification of the one or more obscured ge

Assignees

• X Dev Llc

Inventors

• Yang Jie
• Guo Cheng-En
• Grant Elliott

Classifications

• G06T7/337 Primary

  involving reference images or patches · CPC title

• G06F18/241

  relating to the classification model, e.g. parametric or non-parametric approaches · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V20/188

  Vegetation · CPC title

• G06T2207/20081

  Training; Learning · CPC title