Methods and systems for object recognition in low illumination conditions

Patent metadata

Field	Value
Publication number	US-11461592-B2
Application number	US-201917251327-A
Country	US
Kind code	B2
Filing date	Aug 7, 2019
Priority date	Aug 10, 2018
Publication date	Oct 4, 2022
Grant date	Oct 4, 2022

Abstract

Described herein is an object recognition system in low illumination conditions. A 3D InIm system can be trained in the low illumination levels to classify 3D objects obtained under low illumination conditions. Regions of interest obtained from 3D reconstructed images are obtained by de-noising the 3D reconstructed image using total-variation regularization using an augmented Lagrange approach followed by face detection. The regions of interest are then inputted into a trained CNN. The CNN can be trained using 3D InIm reconstructed under low illumination after TV-denoising. The elemental images were obtained under various low illumination conditions having different SNRs. The CNN can effectively recognize the 3D reconstructed faces after TV-denoising.

First claim

The invention claimed is:

1. An object recognition system operable in a range of illumination conditions including in low illumination conditions using 3D integral imaging, the system comprising:
a computing system hosting a deep learning engine and an object detection application; and
a camera array in communication with the computing system, the camera array including a plurality of cameras each positioned to capture a field of view from a different perspective, the camera array configured to
capture a plurality of elemental images of a 3D scene in low illumination conditions having low light flux, wherein each elemental image of the plurality of elemental images is captured by a different camera of the camera array, the 3D scene including one or more objects, and
transmit the plurality of elemental images to the computing system;

wherein the computing system is configured to train the deep learning engine by
retrieving a set of training data, the set of training data including a plurality of sets of elemental images of an imaging subject, wherein each set of elemental images of the imaging subject is captured by the camera array under a different low illumination condition of a plurality of different low illumination conditions,
reconstructing a plurality of depth images from the plurality of sets of elemental images of the imaging subject, the plurality of depth images including a depth image at each of a plurality of different distances from the camera array for each of the plurality of different low illumination conditions, wherein reconstructing a depth image includes back-propagating image data from each elemental image of a set of elemental images as light rays through a virtual pinhole array to a depth plane corresponding to a defined distance from the camera array and defining the depth image as a sum of the back-propagated image data from all of the elemental images of the set of elemental images at the depth plane corresponding to the defined distance from the camera array, and
training the deep learning engine to produce as an output an identification of the imaging subject in response to receiving as an input one or more depth images from each set of elemental images corresponding to a position of the imaging subject relative to the camera array, wherein the one or more depth images from each set of elemental images corresponding to the position of the imaging subject relative to the camera array includes at least one selected from a group consisting of a single depth image and a subset of depth images corresponding to a sequence of adjacent distances from the camera array;

wherein the computing system is further configured to classify a detected object in the plurality of elemental images of the 3D scene captured by the camera array by
receiving the plurality of elemental images of the 3D scene captured in the low illumination conditions;
reconstructing a plurality of depth images of the 3D scene using the plurality of elemental images of the 3D scene;
de-noising the plurality of depth images of the 3D scene;
applying the object detection application to the plurality of depth images of the 3D scene to identify one or more regions of interest in the one or more depth images of the plurality of depth images of the 3D scene;
providing image data for a region of interest in the one or more depth images of the plurality of depth images of the 3D scene as input to the trained deep learning engine; and
identifying, based on the output of the trained deep learning engine generated in response to receiving the image data for the region of interest as input, the one or more objects in the one or more regions of interest.

2. The system of claim 1 , wherein the deep learning engine implements a Convolutional Neural Network (CNN).

3. The system of claim 1 , wherein the set of training data further includes (i) multiple sets of elemental images for each of a plurality of different imaging subject captured in various low illumination levels and having various levels of low signal-to-noise ratio (SNR), and (ii) augmented data obtained by perturbing elemental images with Gaussian noise.

4. The system of claim 1 , wherein the computing system is configured to de-noise the plurality of depth images of the 3D scene by total-variation regularization using an augmented Lagrange approach.

5. The system of claim 1 , wherein the computing system is configured to reconstruct the plurality of depth images of the 3D scene using the plurality of elemental images of the 3D scene by back-propagating captured light rays from the plurality of elemental images through virtual pinholes of a virtual pinhole array to each of a plurality of specified depth planes.

6. The system of claim 1 , wherein the camera array operates in multiple bands across the electromagnetic spectrum.

7. The system of claim 6 , wherein the camera array operates in the visible spectrum and at least one additional spectrum selected from the group consisting of infrared, ultraviolet and x-ray, and combinations thereof.

8. The system of claim 1 , wherein the computing system does not receive any input indicating the illumination conditions associated with the elemental images of the 3D scene.

9. The system of claim 1 , wherein the computing system is configured to provide the image data for the region of interest in the one or more depth images of the plurality of depth images of the 3D scene as input to the trained deep learning engine by providing as input to the trained deep learning engine a new subset of depth images corresponding to a sequence of adjacent distances from the camera array including at least one depth image identified by the object detection application as including the object and a defined number of adjacent depth images.

10. An object recognition method operable in a range of illumination conditions including in low illumination conditions using 3D integral imaging, the method comprising:
capturing, via a camera array in communication with a computing system, a plurality of elemental images of a 3D scene in low illumination conditions having low light flux, the 3D scene including one or more objects, wherein the camera array includes a plurality of cameras each positioned to capture a field of view from a different perspective;
receiving, via the computing system, the plurality of elemental images of the 3D scene;
reconstructing, via the computing system, a plurality of depth images of the 3D scene using the plurality of elemental images of the 3D scene, wherein reconstructing the plurality of depth images of the 3D scene includes back-propagating captured light rays from each elemental image through a virtual pinhole array to each of a plurality of different depth planes;
de-noising, via the computing system, the plurality of depth images of the 3D scene;
applying, via the computing system, an object detection application to each depth image of the plurality of depth images of the 3D scene, wherein the object detection application is configured to identify regions of interest in the depth images of the 3D scene that contain an object of the one or more objects;
applying, via the computing system, a deep learning engine to each region of interest identified by the object detection application, wherein the deep learning engine is trained to receive as input image data from one or more depth images corresponding to a region of interest and to produce as output a classification of the object in the region of interest, wherein the deep learning engine is trained to classify the object based on a set of training data including a plurality of sets of depth images of the object reconstructed from a plurality of sets of elemental images of t

Assignees

• Univ Connecticut

Inventors

• Javidi Bahram
• Markman Adam

Classifications

• G06V10/30

  Noise filtering · CPC title

• G06T7/55 Primary

  from multiple images · CPC title

• G06V40/165

  using facial parts and geometric relationships · CPC title

• G06V40/166

  using acquisition arrangements · CPC title

• G06V20/64

  Three-dimensional [3D] objects · CPC title