Recovering occluded image data using machine learning

Patent metadata

Field	Value
Publication number	US-11625812-B2
Application number	US-202016786257-A
Country	US
Kind code	B2
Filing date	Feb 10, 2020
Priority date	Nov 1, 2019
Publication date	Apr 11, 2023
Grant date	Apr 11, 2023

Abstract

Examples disclosed herein are related to using a machine learning model to generate image data. One example provides a system, comprising one or more processors, and storage comprising instructions executable by the one or more processors to obtain image data comprising an image with unoccluded features, apply a mask to the unoccluded features in the image to form partial observation training data comprising a masked region that obscures at least a portion of the unoccluded features, and train a machine learning model comprising a generator and a discriminator at least in part by generating image data for the masked region and comparing the image data generated for the masked region to the image with unoccluded features.

First claim

The invention claimed is:

1. A system, comprising
one or more processors; and
storage comprising instructions executable by the one or more processors to:
obtain image data comprising an image with occluded features and unoccluded features, the occluded features being occluded by clouds,
apply a random mask as synthetic clouds to the unoccluded features in the image to form partial observation training data comprising a masked region that obscures at least a portion of the unoccluded features, and
train a machine learning model comprising a generator and a discriminator at least in part by generating image data for the masked region and comparing the image data generated for the masked region to the unoccluded features that were masked.

2. The system of claim 1 , wherein the instructions are further executable to receive image data acquired via an imaging technique that penetrates clouds, and train the machine learning model by generating image data for the masked region based upon the image data acquired via the imaging technique that penetrates clouds.

3. The system of claim 2 , wherein the image data acquired via the imaging technique that penetrates clouds comprises one or more of synthetic aperture radar (SAR) image data, microwave image data, and infrared image data.

4. The system of claim 2 , wherein the instructions are further executable to interpolate the image data acquired via the imaging technique that penetrates clouds to generate interpolated image data for training the machine learning model.

5. The system of claim 1 , wherein the instructions are executable to use the machine learning model to generate image data for an occluded feature in deployment-phase image data.

6. The system of claim 1 , wherein the random mask simulates an occluding feature in deployment-phase image data.

7. The system of claim 1 , wherein the image data comprises one or more of visible satellite image data and multispectral satellite image data.

8. The system of claim 1 , wherein the machine learning model comprises a generative adversarial network (GAN).

9. The system of claim 8 , wherein the GAN comprises an attention mechanism.

10. The system of claim 1 , wherein the image data comprises an image with features occluded by clouds.

11. A method comprising:
receiving training data comprising satellite optical images and radar image data for a geographical area, the satellite images comprising occluded image data due to clouds in the image data and also comprising unoccluded image data,
applying a random mask as synthetic clouds to the unoccluded image data to form partial observation training data comprising a masked region to simulate clouds blocking the unoccluded image data that is masked, and
training a machine learning model comprising a generator and a discriminator at least in part by generating image data for the masked region based upon corresponding image data acquired via an imaging technique that penetrates clouds, and comparing the image data generated for the masked region with the unoccluded image data that were masked.

12. The method of claim 11 , wherein the corresponding image data comprises one or more of SAR data, microwave data, and infrared data.

13. The method of claim 11 , wherein the machine learning model comprises a GAN.

14. The method of claim 13 , wherein the GAN comprises an attention mechanism.

15. The method of claim 13 , further comprising interpolating the image data acquired via the imaging technique that penetrates clouds to generate interpolated image data for training the machine learning model.

16. The method of claim 11 , wherein the image data comprises one or more of multispectral image data and visible image data.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Chandra Ranveer
• Olsen Peder Andreas
• Zhao Mingmin

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/094

  Adversarial learning · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0475

  Generative networks · CPC title

• G06T2207/20081

  Training; Learning · CPC title