Cross-trained convolutional neural networks using multimodal images

Patent metadata

Field	Value
Publication number	US-9633282-B2
Application number	US-201514813233-A
Country	US
Kind code	B2
Filing date	Jul 30, 2015
Priority date	Jul 30, 2015
Publication date	Apr 25, 2017
Grant date	Apr 25, 2017

Abstract

Embodiments of a computer-implemented method for training a convolutional neural network (CNN) that is pre-trained using a set of color images are disclosed. The method comprises receiving a training dataset including multiple multidimensional images, each multidimensional image including a color image and a depth image; performing a fine-tuning of the pre-trained CNN using the depth image for each of the plurality of multidimensional images; obtaining a depth CNN based on the pre-trained CNN, wherein the depth CNN is associated with a first set of parameters; replicating the depth CNN to obtain a duplicate depth CNN being initialized with the first set of parameters; and obtaining a depth-enhanced color CNN based on the duplicate depth CNN being fine-tuned using the color image for each of the plurality of multidimensional images, wherein the depth-enhanced color CNN is associated with a second set of parameters.

First claim

What is claimed is:

1. A computer-implemented method for training a convolutional neural network (CNN) that is pre-trained using a set of color images, the method comprising:
receiving, using an input module of a system memory, a training dataset including a plurality of multidimensional images, each multidimensional image including a color image and a depth image;
performing, using a processor, a fine-tuning of the pre-trained CNN using the depth image for each of the plurality of multidimensional images;
obtaining, using a cross-trained CNN module in the system memory, a depth CNN based on the pre-trained CNN, the depth CNN includes at least one convolutional layer in communication with an ultimate fully-connected layer via a penultimate fully-connected-layer, wherein the depth CNN is associated with a first set of parameters;
replicating, using the cross-trained CNN module, the depth CNN to obtain a duplicate depth CNN being initialized with the first set of parameters; and
obtaining, using the cross-trained CNN module, a depth-enhanced color CNN based on the duplicate depth CNN being fine-tuned using the color image for each of the plurality of multidimensional images, the depth-enhanced color CNN includes at least one convolutional layer in communication with an ultimate fully-connected layer of the depth-enhanced color CNN via a penultimate fully-connected-layer of the depth-enhanced color CNN, wherein the depth-enhanced color CNN is associated with a second set of parameters.

2. The computer-implemented method of claim 1 , wherein the second set of parameters are obtained by adjusting the first set of parameters based on the duplicate depth CNN being fine-tuned using the color image for one or more of the plurality of multidimensional images.

3. The computer-implemented method of claim 1 , wherein the depth image is encoded across a plurality of channels being applied to the at least one convolutional layer of the depth CNN for each image pixel, wherein the plurality of channels include horizontal disparity, height above ground, and the angle the pixel's local surface normal makes with the inferred gravity direction.

4. The computer-implemented method of claim 1 , wherein the depth image is replicated across one or more channels being applied to the at least one convolutional layer of the depth CNN for each image pixel.

5. The computer-implemented method of claim 1 , wherein the color image is an RGB (Red, Green, Blue) image having three image channels, namely, a red image channel, a green image channel, and a blue image channel.

6. The computer-implemented method of claim 1 , wherein each of the color image and the depth image is represented as a two-dimensional matrix.

7. The computer-implemented method of claim 1 , wherein the first set of parameters includes neuron weights and biases.

8. The computer-implemented method of claim 1 , further comprising testing the obtained depth CNN and the depth-enhanced color for object recognition, wherein the testing comprises:
receiving the depth CNN, the depth-enhanced color CNN, and a testing dataset including a second plurality of multidimensional images, each of the second plurality of multidimensional images including a test color image and a test depth image;
applying the test depth image to the depth CNN and the test color image to the depth-enhanced color CNN for each of the second plurality of multidimensional images;
extracting depth features from the penultimate fully-connected layer of the depth CNN for each of the second plurality of multidimensional images;
extracting color features from the penultimate fully-connected layer of the depth-enhanced color CNN for each of the second plurality of multidimensional images; and
concatenating the extracted depth features and color features to generate a combine feature vector for each of the second plurality of multidimensional images.

9. The computer-implemented method of claim 8 , wherein the testing dataset further includes at least one color image being applied to the depth-enhanced color CNN for extracting color features from the penultimate fully-connected layer of the depth-enhanced color CNN.

10. A device for training a convolutional neural network (CNN) that is pre-trained using a set of color images, the device comprising one or more processors configured to:
receive using an input module a training dataset including a plurality of multidimensional images, each of the multidimensional images including a color image and a depth image;
perform using a cross-trained CNN module a fine-tuning of the pre-trained CNN using the depth image for each of the plurality of multidimensional images;
obtain using the cross-trained CNN module a depth CNN based on the pre-trained CNN, the depth CNN includes at least one convolutional layer in communication with an ultimate fully-connected layer via a penultimate fully-connected-layer, wherein the depth CNN is associated with a first set of parameters;
replicate using the cross-trained CNN module the depth CNN to obtain a duplicate depth CNN being initialized with the first set of parameters; and

obtain using the cross-trained CNN module a depth-enhanced color CNN based on the duplicate depth CNN being fine-tuned using the color image for each of the plurality of multidimensional images, the depth-enhanced color CNN includes at least one convolutional layer in communication with an ultimate fully-connected layer of the depth-enhanced color CNN via a penultimate fully-connected-layer the depth-enhanced color CNN, wherein the depth-enhanced color CNN is associated with a second set of parameters.

11. The device of claim 10 , wherein the second set of parameters are obtained by adjusting the first set of parameters based on the duplicate depth CNN being fine-tuned using the color image for one or more of the plurality of multidimensional images.

12. The device of claim 10 , wherein the depth image is encoded across a plurality of channels being applied to the at least one convolutional layer of the depth CNN for each image pixel, wherein the plurality of channels include horizontal disparity, height above ground, and the angle the pixel's local surface normal makes with the inferred gravity direction.

13. The device of claim 10 , wherein the depth image is replicated across three channels being applied to the at least one convolutional layer of the depth CNN for each image pixel.

14. The device of claim 10 , wherein the color image is an RGB (Red, Green, Blue) image having three image channels, namely, red image channel, green image channel, and blue image channel.

15. The device of claim 10 , wherein each of the color image and the depth image is represented as a two-dimensional matrix.

16. The device of claim 10 , wherein the first set of parameters includes neuron weights and biases.

17. The device of claim 10 , further comprising testing the obtained depth CNN and the depth-enhanced color CNN for object recognition, wherein the device comprises one or more processors configured to:
receive using the input module a testing dataset including a second plurality of multidimensional images, each of the second plurality of multidimensional images including a test color image and a test depth image;
input using a classification module the test depth image to the depth CNN and the test color image to the depth-enhanced color CNN for each of the second plurality of multidimensional images;
extract using the classification module depth features from the penultimate fully-connected layer of the depth CNN for each of the second pl

Assignees

• Xerox Corp

Inventors

• Sharma Arjun
• Kompalli Pramod Sankar

Classifications

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V10/56 Primary

  relating to colour · CPC title

• G06V30/194

  References adjustable by an adaptive method, e.g. learning · CPC title

• G06V10/449

  Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters · CPC title

• G06K9/6232

  Physics · mapped topic