Synthesizing training data for broad area geospatial object detection

Patent metadata

Field	Value
Publication number	US-9767565-B2
Application number	US-201615194541-A
Country	US
Kind code	B2
Filing date	Jun 27, 2016
Priority date	Aug 26, 2015
Publication date	Sep 19, 2017
Grant date	Sep 19, 2017

Abstract

A system for broad area geospatial object recognition, identification, classification, location and quantification, comprising an image manipulation module to create synthetically-generated images to imitate and augment an existing quantity of orthorectified geospatial images; together with a deep learning module and a convolutional neural network serving as an image analysis module, to analyze a large corpus of orthorectified geospatial images, identify and demarcate a searched object of interest from within the corpus, locate and quantify the identified or classified objects from the corpus of geospatial imagery available to the system. The system reports results in a requestor's preferred format.

First claim

What is claimed is:

1. A system for broad area geospatial object detection using synthetically-generated training images for improved training of a deep learning model comprising a computing device comprising a processor, a memory, and a plurality of programming instructions stored in the memory and operable on the processor, wherein the plurality of programming instructions:
(a) retrieves a 3-dimensional model of an object of interest from an established data store;
(b) creates a flattened, 2-dimensional modeled image from the 3-dimensional model;
(c) compares the flattened modeled image to a real geospatial image comprising an instance of the object of interest and associated background;
(d) scales the flattened modeled image to align with the real geospatial image of the instance of the object of interest and upon successful alignment, separates the flattened modeled image from the background of the real image in order to fine tune components of the flattened modeled image, which include smoothing edges or color matching to simulate the real image;
(e) applies a plurality of environmental effects to replicate seasonal, time of day, associated brightness, and environmental factors consistent with a geographic location of the real background image to create a plurality of modified synthetic images;
(f) creates a plurality of shadowed, modified 2-dimensional synthetic images for the 3-dimensional object as if it were physically located and oriented where it would be affected by real-time and real-world shadowing;
(g) adjusts the shadowed, modified synthetic 2-dimensional images by pixelating and blurring or focusing to resemble the real image;
(h) identifies and demarcates a footprint associated with each of the shadowed, modified synthetic 2-dimensional images;
(i) overlays the demarcated footprint onto a real image and masks the background colors surrounding the synthetic image to become transparent such that overlay onto the real image does not obscure existing images to create a manipulated synthetic image;
(j) generates a labeled corpus of manipulated synthetic training data comprising a plurality of modified images; and
(k) trains a deep learning model comprising a convolutional neural network to recognize objects of the same type as the object of interest in geospatial images.

2. The system of claim 1 , further comprising an image analysis server comprising a second processor, a second memory, and a second plurality of programming instructions stored in the second memory and operable on the second processor, wherein the second plurality of programming instructions:
(a) uses the deep learning model to automatically identify and label all objects of interest in a received data set comprising a plurality of unanalyzed orthorectified geospatial imagery, regardless of the orientation or scale of the feature item within the section, and accounting for differences in item scale by using a multi-scale sliding window algorithm; and
(b) outputs the locations of the identified objects of interest.

3. A method for identifying objects of interest in geospatial images using a deep learning model and synthetically-generated training images the method comprising the steps of:
(a) automatically generating, using an image manipulation computer, a 2-dimensional image of an object of interest from a three-dimensional model of the object of interest;
(b) manipulating the generated 2-dimensional image to create a plurality of synthetic images comprising at least one of the object of interest, placing the object of interest in the plurality of synthetic images in a plurality of locations, environments, orientations, scales, exposures, and foci in order to create a large corpus of synthetically-generated training images;
(c) training, using the large corpus of synthetically-generated training images, a deep learning model comprising a convolutional neural network to recognize objects of the same type as the object of interest in a plurality of unlabeled geospatial images;
(d) analyzing the plurality of unlabeled geospatial images using the deep learning model to identify objects of interest; and
(e) generating an output file comprising location, classification, and quantity of the object of interest in each of the plurality of geospatial images.

Assignees

• Digitalglobe Inc

Inventors

• Estrada Adam
• Burd Christopher
• Jenkins Andrew
• Newbrough Joseph
• Szoko Scott
• Vinton Melanie

Classifications

• G06N3/08

  Learning methods · CPC title

• G06V10/774 Primary

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06T7/75

  involving models · CPC title

• G06T15/50

  Lighting effects · CPC title

• G06T7/90

  Determination of colour characteristics · CPC title