Deep learning architectures for the classification of objects captured with a light-field camera

Patent metadata

Field	Value
Publication number	US-10657425-B2
Application number	US-201815917473-A
Country	US
Kind code	B2
Filing date	Mar 9, 2018
Priority date	Mar 9, 2018
Publication date	May 19, 2020
Grant date	May 19, 2020

Abstract

Light-field data is masked to identify regions of interest, before applying to deep learning models. In one approach, a light-field camera captures a light-field image of an object to be classified. The light-field image includes a plurality of views of the object taken simultaneously from different viewpoints. The light-field image is pre-processed, with the resulting data provided as input to a deep learning model. The pre-processing includes determining and then applying masks to select regions of interest within the light-field data. In this way, less relevant data can be excluded from the deep learning model. Based on the masked data, the deep learning model produces a decision classifying the object.

First claim

What is claimed is:

1. A method implemented on a computer system comprising a processor, the processor executing instructions to effect a method for classifying an object, the method comprising:
receiving a light-field image of the object, the light-field image comprising a plurality of views of the object taken simultaneously from different viewpoints;
pre-processing the light-field image to produce derivative image data, the pre-processing comprising:
determining masking to select regions of interest within the light-field image, and
applying the masking; and

applying the derivative image data as input to a deep learning model, the deep learning model producing a decision classifying the object, wherein producing the decision classifying the object comprises:
the deep learning model producing a plurality of intermediate decisions classifying the object; and
applying a voting schema to the intermediate decisions to produce the decision classifying the object.

2. The method of claim 1 wherein determining masking comprises:
extracting depth information for the object from the plurality of views; and
determining the masking based on the extracted depth information.

3. The method of claim 1 wherein applying masking consists of:
applying masking to only one of the views, wherein the derivative image data is not based on any other of the views.

4. The method of claim 1 wherein applying masking comprises:
applying masking to each of the plurality of views.

5. The method of claim 4 wherein:
determining masking comprises determining separately for each of the plurality of views, the masking to be applied to that view; and
applying masking comprises, for each of the views, applying the masking for that view.

6. The method of claim 1 wherein:
pre-processing the light-field image comprises producing epipolar images from light-field data in the light-field image; and
applying masking comprises applying masking to the epipolar images.

7. The method of claim 1 wherein applying masking comprises:
applying masking directly to the light-field image.

8. The method of claim 1 wherein:
pre-processing the light-field image comprises reducing light-field data in the light-field image with respect to image, view and/or channel dimensions; and
applying masking comprises applying masking to the reduced light-field data.

9. The method of claim 1 wherein the light-field image comprises different color channels, and applying masking comprises applying masking separately to each of the color channels.

10. The method of claim 1 wherein the masking is binary.

11. The method of claim 1 wherein the masking is continuous.

12. The method of claim 1 wherein the voting schema weights the intermediate decisions, and the weight for each intermediate decision is related to the masking applied to produce that intermediate decision.

13. The method of claim 1 wherein the voting schema uses a machine learning model to produce the decision classifying the object.

14. The method of claim 1 wherein the object is an eardrum of a patient, and the decision is a decision classifying a condition of the eardrum.

15. The method of claim 1 wherein the deep learning model comprises a plurality of layers, each layer including a set of filters applied to a set of inputs for the layer, and a non-linear function applied to an output of the filtering.

16. A method implemented on a computer system comprising a processor, the processor executing instructions to effect a method for classifying an object, the method comprising:
receiving a light-field image of the object, the light-field image comprising a plurality of views of the object taken simultaneously from different viewpoints;
pre-processing the light-field image to produce derivative image data, the pre-processing comprising:
determining masking to select regions of interest within the light-field image, wherein determining masking comprises determining masking based on image intensity of light-field data in the light-field image; and
applying the masking; and

applying the derivative image data as input to a deep learning model, the deep learning model producing a decision classifying the object.

17. A method implemented on a computer system comprising a processor, the processor executing instructions to effect a method for classifying an object, the method comprising:
receiving a light-field image of the object, the light-field image comprising a plurality of views of the object taken simultaneously from different viewpoints;
pre-processing the light-field image to produce derivative image data, the pre-processing comprising:
determining masking to select regions of interest within the light-field image, wherein determining masking comprises using a machine learning model to determine the masking; and
applying the masking; and

applying the derivative image data as input to a deep learning model, the deep learning model producing a decision classifying the object.

Assignees

• Martinello Manuel
• Spinoulas Leonidas
• Ricoh Co Ltd

Inventors

• Martinello Manuel
• Spinoulas Leonidas

Classifications

• G06N5/02

  Knowledge representation; Symbolic representation · CPC title

• A61B1/227

  for ears, i.e. otoscopes · CPC title

• G06T7/557

  from light fields, e.g. from plenoptic cameras · CPC title

• G06K9/66 Primary

  Physics · mapped topic

• G06K9/6232

  Physics · mapped topic