Method and system for detecting multiple moving objects from real-time aerial images

Patent metadata

Field	Value
Publication number	US-9940724-B2
Application number	US-201615263761-A
Country	US
Kind code	B2
Filing date	Sep 13, 2016
Priority date	Sep 13, 2016
Publication date	Apr 10, 2018
Grant date	Apr 10, 2018

Abstract

In accordance with various embodiments of the disclosed subject matter, a method and a system for detecting multiple objects from real-time images are provided. The method comprises: performing, using a CPU host, an image segmentation process to divide real-time input images into a plurality of image partitions; performing, by multiple GPUs, a fast block-wise registration process, a mark setting process, a background generation process, a foreground generation process based on a Hyper-Q computation infrastructure, and a support vector machine classification process; and generating, by the CPU host, visualization classification images.

First claim

What is claimed is:

1. A method for detecting multiple objects from real-time images, comprising:
performing, using a central processing unit (CPU) host, an image segmentation process to divide real-time input images into a plurality of image partitions;
allocating a multitude of image memory according to the plurality of image partitions for parallel computation in a plurality of graphics processing units (GPUs);
performing, by registration kernels of the plurality of GPUs, a fast block-wise registration process to register the plurality of image partitions;
performing, by mask setting kernels of the plurality of GPUs, a mark setting process for the registered image partitions to stitch the registered images into output images;
performing, by background generation kernels of the plurality of GPUs, a background generation process based on the output images to generate background images through median filter;
performing, by foreground generation kernels of the plurality of GPUs, a foreground generation process based on the output images to generate foreground images based on a Hyper-Q computation framework;
performing, by classification kernels of the plurality of GPUs, a support vector machine classification process to classify a plurality of objects of the real-time input images; and
generating, by the CPU host, visualization classification images based on the background images, foreground images and the plurality of objects.

2. The method of claim 1 , wherein:
the real-time input images are generated from a Wide Area Motion Imagery system on an aerial platform;
a frame rate of the real-time input images is at least two frames per second; and
a scale of each real-time input image has a magnitude of million orders of pixels in resolution.

3. The method of claim 1 , wherein:
the fast block-wise registration process is a Compute Unified Device Architecture (CUDA) based parallel computing infrastructure, and comprises:
performing a block-wise speeded up robust features extraction process for each image partition;
performing a point matching process for each image partition;
using a random sample consensus algorithm to remove outlier points from the plurality of image partitions; and
performing a transformation estimation process of the image partitions to generate block-wise homography matrices.

4. The method of claim 1 , wherein each registration kernel is configured to have one computation node integrated with eight groups of four image partitions at a time instant.

5. The method of claim 3 , wherein the mark setting process comprises:
stitching portions of the registered image partitions based on the block-wise homography matrices generated from the transformation estimation process;
wherein a number of threads per block is consistent with available shared memory of the plurality of GPUs.

6. The method of claim 3 , wherein the point matching process is based on Brute-force (BF) or Fast Library for Approximate Nearest Neighbors (FLANN).

7. The method of claim 1 , wherein the background generation process:
comprises a background setting step for setting each image partition to a mask with zero pixel values, an image averaging step, and a background extraction step; and
is a parallelized process implemented based on the plurality of GPUs using data CUDA structure dim3.

8. The method of claim 1 , wherein the foreground generation process:
comprises a pixel value comparison step, a value assigning step, and a foreground extraction step;
implements the Hyper-Q computation infrastructure to enable multiple CPU cores to launch work on a single GPU simultaneously for increasing GPUs utilization and slashing CPU idle time; and
introduces Grid Management Unit to create multiple hardware work queues to reduce a synchronization time.

9. The method of claim 1 , wherein the support vector machine classification process comprises:
implementing histogram of oriented gradients (HOG) to compute color gradients and obtain gradient magnitudes and orientations via convolution; and
calculating probabilities or confidence levels of the multiple objects based on the gradient magnitudes and orientations.

10. The method of claim 1 , further comprising:
asynchronously transmitting the classified plurality of objects from the multiple GPUs to the CPU host to reduce or eliminate a synchronization time.

11. The method of claim 1 , further comprising:
generating a graphical user interface through an open source computer vision library associated with the CPU host for monitoring the multiple objects on the visualization classification images in real-time;
wherein the multiple objects include at least vehicles and people.

12. A system for detecting multiple objects from real-time images, comprising:
a CPU host configured for:
performing an image segmentation process to divide real-time input images into a plurality of image partitions, and
allocating a multitude of image memory according to the plurality of image partitions for parallel computation; and

a plurality of graphics processing units (GPUs) comprising:
registration kernels for performing a fast block-wise registration process to register the plurality of image partitions,
mask setting kernels for performing a mark setting process for the registered image partitions to stitch the registered images into output images,
background generation kernels for performing a background generation process based on the output images to generate background images through median filter,
foreground generation kernels for performing a foreground generation process based on the output images to generate foreground images based on a Hyper-Q computation infrastructure, and
classification kernels for performing a support vector machine classification process to classify a plurality of objects of the real-time input images;

wherein the CPU host is further configured for generating visualization classification images based on the background images, foreground images and the plurality of objects.

13. The system of claim 12 , wherein:
the real-time input images are generated from a Wide Area Motion Imagery system on an aerial platform;
a frame rate of the real-time input images is at least two frames per second;
a scale of each real-time input image has a magnitude of million orders of pixels in resolution; and
the multiple objects include at least vehicles and people.

14. The system of claim 12 , wherein:
the registration kernels are configured for performing the fast block-wise registration process based a Compute Unified Device Architecture (CUDA) based parallel computing infrastructure, by:
performing a block-wise speeded up robust features extraction process for each image partition;
performing a point matching process for each image partition;
using a random sample consensus (RANSAC) algorithm to remove outlier points from the plurality of image partitions; and
performing a transformation estimation process of the image partitions to generate block-wise homography matrices;

wherein each registration kernel is configured to have one cluster integrated with eight groups of four image partitions at a time instant.

15. The system of claim 14 , wherein the mark setting kernels are configured for:
stitching portions of the registered image partitions based on the block-wise homography matrices generated from the transformation estimation process;
wherein a number of threads per block is consistent with available shared memory of the plurality of GPUs.

16. The system of claim 12 , wherein the background generatio

Assignees

• Intelligent Fusion Tech Inc
• Intelligent Fusion Tech Inc

Inventors

• Jia Bin
• Liu Kui
• Wei Sixiao
• Blasch Erik
• Sheaff Carolyn
• Ling Haibin
• Chen Genshe

Classifications

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V10/50

  by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis · CPC title

• G06T3/4038 Primary

  Image mosaicing, e.g. composing plane images from plane sub-images · CPC title

• G06F18/2411

  based on the proximity to a decision surface, e.g. support vector machines · CPC title

• G06V10/757

  Matching configurations of points or features · CPC title