Object space calibration of plenoptic imaging systems

Patent metadata

Field	Value
Publication number	US-9918077-B2
Application number	US-201615365671-A
Country	US
Kind code	B2
Filing date	Nov 30, 2016
Priority date	Jan 9, 2015
Publication date	Mar 13, 2018
Grant date	Mar 13, 2018

Abstract

A collimated object is adjustable to produce collimated light propagating along different propagation directions. The plenoptic imaging system under calibration captures plenoptic images of the object adjusted to different propagation directions. The captured plenoptic images includes superpixels, each of which includes subpixels. Each subpixel captures light from a corresponding light field viewing direction. Based on the captured plenoptic images, a calibration module calculates which propagation directions map to which subpixels. The mapping defines the light field viewing directions for the subpixels. This can be used to improve processing of plenoptic images captured by the plenoptic imaging system.

First claim

What is claimed is:

1. A method for calibrating a plenoptic imaging system, the plenoptic imaging system including primary imaging optics, a secondary imaging array and a sensor array, the method comprising:
the plenoptic imaging system capturing a plurality of plenoptic images of a light source that produces non-collimated light that is imaged through a pinhole located at a pupil of the primary imaging optics, each plenoptic image comprising a plurality of superpixels, each superpixel comprising a plurality of subpixels;
automatically adjusting a lateral position of at least one of the light source and the pinhole, the plenoptic imaging system automatically capturing plenoptic images for the different lateral positions;
based on the captured plenoptic images, calculating a mapping of spatial locations and/or propagation directions in object space of the plenoptic imaging system to subpixels in the plenoptic image, the mapping defining light field viewing regions and/or light field viewing directions for the subpixels; and
after calculating the mapping, the plenoptic imaging system capturing a second plenoptic image of a second object, and processing the second plenoptic image using the calculated mapping to account for the light field viewing regions and/or light field viewing directions for the subpixels.

2. The method of claim 1 wherein automatically adjusting a lateral position of at least one of the light source and the pinhole comprises automatically adjusting a lateral position of the light source while maintaining the pinhole in a fixed lateral position.

3. The method of claim 1 wherein automatically adjusting a lateral position of at least one of the light source and the pinhole comprises automatically adjusting a lateral position of the pinhole while maintaining the light source in a fixed lateral position.

4. The method of claim 1 wherein automatically adjusting a lateral position of at least one of the light source and the pinhole comprises automatically adjusting a lateral position of the light source and automatically adjusting a lateral position of the pinhole, wherein each combination of lateral position of the light source and lateral position of the pinhole provides a sampling of the spatial location and of the propagation direction in object space.

5. The method of claim 1 wherein calculating the mapping comprises, for each captured plenoptic image, constructing a correspondence between (a) a spatial location of the light source and a propagation direction from the light source to the pinhole, and (b) a subpixel in the plenoptic image; wherein the aggregate correspondences for all captured plenoptic images defines light field viewing regions and/or light field viewing directions for the subpixels.

6. The method of claim 1 wherein calculating a mapping comprises creating a lookup table that maps light field viewing regions and/or light field viewing directions to subpixels.

7. The method of claim 1 wherein calculating a mapping comprises creating a lookup table that maps subpixels to light field viewing regions and/or light field viewing directions.

8. A system for calibrating a plenoptic imaging system, the system comprising:
a light source that produces non-collimated light;
a plenoptic imaging system including primary imaging optics, a secondary imaging array and a sensor array;
a pinhole located at a pupil of the primary imaging optics, wherein at least one of the light source and the pinhole is adjustable in lateral position;
the plenoptic imaging system capturing plenoptic images of the light source imaged through the pinhole at different lateral positions of at least one of the light source and the pinhole, each plenoptic image comprising a plurality of superpixels, each superpixel comprising a plurality of subpixels; and
a calibration module that, based on the captured plenoptic images, calculates a mapping of spatial locations and/or propagation directions in object space of the plenoptic imaging system to subpixels in the plenoptic image, the mapping defining light field viewing regions and/or light field viewing directions for the subpixels.

9. The method of claim 8 wherein the plenoptic imaging system captures plenoptic images of the light source imaged through the pinhole at different lateral positions of the light source with the pinhole in a fixed lateral position.

10. The method of claim 8 wherein the plenoptic imaging system captures plenoptic images of the light source imaged through the pinhole at different lateral positions of the pinhole with the light source in a fixed lateral position.

11. The method of claim 8 wherein the plenoptic imaging system captures plenoptic images of the light source imaged through the pinhole at different combinations of lateral positions of the light source and lateral positions of the pinhole, wherein each combination provides a sampling of the spatial location and of the propagation direction in object space.

12. The method of claim 11 wherein the spatial location in object space is determined by the lateral position of the light source, and the propagation direction in object space is determined by the direction from the light source to the pinhole.

13. The method of claim 8 wherein the calibration module creates a lookup table that maps light field viewing regions and/or light field viewing directions to subpixels.

14. The method of claim 8 wherein the calibration module creates a lookup table that maps subpixels to light field viewing regions and/or light field viewing directions.

15. The method of claim 8 wherein the plenoptic imaging system is not object-space telecentric.

16. The method of claim 8 wherein the plenoptic imaging system is not image-space telecentric.

17. The method of claim 8 wherein the plenoptic imaging system has finite conjugates.

18. The method of claim 8 further comprising:
a processing module that processes a plenoptic image of an object, said processing taking into account the light field viewing regions and/or light field viewing directions for the subpixels as determined by the calibration module.

19. A computer program product for calibrating a plenoptic imaging system, the computer program product comprising a non-transitory machine-readable medium storing computer program code for performing a method, the method comprising:
automatically adjusting at least one of (a) a lateral position of a light source that produces non-collimated light and (b) a lateral position of a pinhole located at a pupil of primary imaging optics of a plenoptic imaging system, the plenoptic imaging system automatically capturing plenoptic images of the light source imaged through the pinhole for the different lateral positions;
accessing the plenoptic images captured by the plenoptic imaging system, each plenoptic image comprising a plurality of superpixels, each superpixel comprising a plurality of subpixels;
based on the captured plenoptic images, calculating a mapping of spatial locations and/or propagation directions in object space of the plenoptic imaging system to subpixels in the plenoptic image, the mapping defining light field viewing regions and/or light field viewing directions for the subpixels; and
after calculating the mapping, processing a second plenoptic image captured by the plenoptic imaging system, said processing using the calculated mapping to account for the light field viewing regions and/or light field viewing directions for the subpixels.

20. The computer program product claim 19 wherein:
aut

Assignees

• Meng Lingfei
• Lu Liyang
• Bedard Noah
• Berkner Kathrin
• Ricoh Co Ltd

Inventors

• Meng Lingfei
• Lu Liyang
• Bedard Noah
• Berkner Kathrin

Classifications

• H04N23/55

  Optical parts specially adapted for electronic image sensors; Mounting thereof · CPC title

• G06T2207/10052

  Images from lightfield camera · CPC title

• G06T7/80 Primary

  Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration · CPC title

• H04N17/002 Primary

  for television cameras · CPC title

• G01J3/0208 Primary

  using focussing or collimating elements, e.g. lenses or mirrors; performing aberration correction · CPC title