Object space calibration of plenoptic imaging systems

Patent metadata

Field	Value
Publication number	US-9544583-B2
Application number	US-201514594019-A
Country	US
Kind code	B2
Filing date	Jan 9, 2015
Priority date	Jan 9, 2015
Publication date	Jan 10, 2017
Grant date	Jan 10, 2017

Abstract

A collimated object is adjustable to produce collimated light propagating along different propagation directions. The plenoptic imaging system under calibration captures plenoptic images of the object adjusted to different propagation directions. The captured plenoptic images includes superpixels, each of which includes subpixels. Each subpixel captures light from a corresponding light field viewing direction. Based on the captured plenoptic images, a calibration module calculates which propagation directions map to which subpixels. The mapping defines the light field viewing directions for the subpixels. This can be used to improve processing of plenoptic images captured by the plenoptic imaging system.

First claim

What is claimed is:

1. A method for calibrating a plenoptic imaging system, the plenoptic imaging system including primary imaging optics, a secondary imaging array and a sensor array, the method comprising:
the plenoptic imaging system capturing plenoptic images of an object that produces collimated light propagating along different propagation directions, each plenoptic image comprising a plurality of superpixels, each superpixel comprising a plurality of subpixels, each subpixel capturing light from at least one corresponding light field viewing direction, the light field viewing directions corresponding to the different propagation directions;
automatically adjusting the object to produce collimated light propagating along different propagation directions, the plenoptic imaging system automatically capturing plenoptic images for each of the different propagation directions;
based on the captured plenoptic images, calculating a mapping of propagation directions from the object to subpixels in the plenoptic image, the mapping defining the light field viewing directions for the subpixels; and
after calculating the mapping, the plenoptic imaging system capturing a second plenoptic image of a second object, and processing the second plenoptic image using the calculated mapping to account for the light field viewing directions for the subpixels.

2. A method for calibrating a plenoptic imaging system, the plenoptic imaging system including primary imaging optics, a secondary imaging array and a sensor array, the method comprising:
the plenoptic imaging system capturing plenoptic images of an object that produces collimated light propagating along different propagation directions, each plenoptic image comprising a plurality of superpixels, each superpixel comprising a plurality of subpixels, each subpixel capturing light from at least one corresponding light field viewing direction, the light field viewing directions corresponding to the different propagation directions;
based on the captured plenoptic images, calculating a mapping of propagation directions from the object to subpixels in the plenoptic image, the mapping defining the light field viewing directions for the subpixels, wherein calculating the mapping of propagation directions to subpixels comprises, for each collimated object:
calculating a centroid of light captured by each superpixel; and
mapping the propagation direction of light from the collimated object to the subpixels corresponding to the calculated centroids; and

after calculating the mapping, the plenoptic imaging system capturing a second plenoptic image of a second object, and processing the second plenoptic image using the calculated mapping to account for the light field viewing directions for the subpixels.

3. The method of claim 1 wherein calculating a mapping comprises creating a lookup table that maps light field viewing directions to subpixels.

4. The method of claim 1 wherein calculating a mapping comprises creating a lookup table that maps subpixels to light field viewing directions.

5. The method of claim 1 wherein each superpixel contains N subpixels and the object produces collimated light propagating along at least N different propagation directions.

6. The method of claim 1 wherein the plenoptic imaging system is not object-space telecentric.

7. The method of claim 1 wherein the plenoptic imaging system is not image-space telecentric.

8. The method of claim 1 wherein the plenoptic imaging system has finite conjugates.

9. The method of claim 1 further comprising:
storing data about the light field viewing directions for the subpixels;
after storing the data, capturing another plenoptic image; and
processing said plenoptic image, using the stored data to account for the light field viewing directions for the subpixels.

10. A system for calibrating a plenoptic imaging system, the system comprising:
an adjustable collimated object, the object adjustable to produce collimated light propagating along different propagation directions;
a plenoptic imaging system that captures plenoptic images of the object adjusted to different propagation directions, each plenoptic image comprising a plurality of superpixels, each superpixel comprising a plurality of subpixels, each subpixel capturing light from a corresponding light field viewing direction; and
a calibration module that, based on the captured plenoptic images, calculates a mapping of propagation directions to subpixels, the mapping defining the light field viewing directions for the subpixels.

11. The system of claim 10 wherein the calibration module is further for:
calculating a centroid of light captured by each superpixel; and
mapping the propagation direction of light from the collimated object to the subpixels corresponding to the calculated centroids.

12. The system of claim 10 wherein the calibration module further creates a lookup table that maps light field viewing directions to subpixels.

13. The system of claim 10 wherein the calibration module further creates a lookup table that maps subpixels to light field viewing directions.

14. The system of claim 10 further comprising a controller that:
automatically adjusts the object to produce collimated light propagating along different propagation directions; and
automatically causes the plenoptic imaging system to capture plenoptic images corresponding to each of the different propagation directions.

15. The system of claim 10 wherein the plenoptic imaging system is not object-space telecentric.

16. The system of claim 10 wherein the plenoptic imaging system is not image-space telecentric.

17. The system of claim 10 wherein the plenoptic imaging system has finite conjugates.

18. The system of claim 10 further comprising:
a processing module that processes a plenoptic image of a different object, said processing taking into account the light field viewing directions for the subpixels as determined by the calibration module.

19. A computer program product for calibrating a plenoptic imaging system, the computer program product comprising a non-transitory machine-readable medium storing computer program code for performing a method, the method comprising:
automatically adjusting an object to produce collimated light propagating along different propagation directions, a plenoptic imaging system automatically capturing plenoptic images for each of the different propagation directions;
accessing the plenoptic images captured by the plenoptic imaging system of the object that produces collimated light propagating along different propagation directions, each plenoptic image comprising a plurality of superpixels, each superpixel comprising a plurality of subpixels, each subpixel capturing light from a corresponding light field viewing direction, the light field viewing directions corresponding to the different propagation directions;
based on the captured plenoptic images, calculating a mapping of propagation directions from the object to subpixels in the plenoptic image, the mapping defining the light field viewing directions for the subpixels; and
after calculating the mapping, the plenoptic imaging system capturing a second plenoptic image of a second object, and processing the second plenoptic image using the calculated mapping to account for the light field viewing directions for the subpixels.

Assignees

• Meng Lingfei
• Lu Liyang
• Bedard Noah
• Berkner Kathrin
• Ricoh Co Ltd

Inventors

• Meng Lingfei
• Lu Liyang
• Bedard Noah
• Berkner Kathrin

Classifications

• H04N23/55

  Optical parts specially adapted for electronic image sensors; Mounting thereof · CPC title

• G06T2207/10052

  Images from lightfield camera · CPC title

• H04N17/002 Primary

  for television cameras · CPC title

• G06T7/0018

  Physics · mapped topic

• G06T7/80 Primary

  Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration · CPC title