Multi-aperture imaging device

Patent metadata

Field	Value
Publication number	US-11244434-B2
Application number	US-202016878771-A
Country	US
Kind code	B2
Filing date	May 20, 2020
Priority date	Aug 24, 2015
Publication date	Feb 8, 2022
Grant date	Feb 8, 2022

Abstract

A multi-aperture imaging device that is, on the one hand, able to provide image information on a scene and, on the other hand, allows obtaining high lateral resolution and/or a wide total field of view, is described. The multi-aperture imaging device is provided with a first plurality of optical channels for projecting overlapping first partial fields of view of a total field of view on first image sensor areas of an image sensor of the multi-aperture imaging device, as well as with a second arrangement of optical channels for projecting at least a part of of the total field of view on a second image sensor area of the image sensor.

First claim

The invention claimed is:

1. A multi-aperture imaging device for providing information on a scene, comprising:
an image sensor;
a first arrangement of optical channels having a first plurality of optical channels for projecting overlapping first partial fields of view of a total field of view on first image sensor areas of the image sensor;
a second arrangement of optical channels for projecting at least a second partial field of view of the total field of view that overlaps with the first partial fields of view on a second image sensor area of the image sensor;
wherein the first partial fields of view together cover the total field of view;
a processor for fusing images that have been captured by the first arrangement of optical channels to a first total image, wherein the processor is configured to perform fusion at least in an overlap area of first partial fields of view of a pair of optical channels of the first plurality of optical channels by using disparities in a pair of images, wherein at least one of them has been captured by the second arrangement of optical channels whose projected second partial field of view overlaps at least with the overlap area; wherein the 3D multi-aperture imaging device is adapted to at least one of:
wherein a number of the first plurality of optical channels is between 2 and 100, both inclusive, and the overlap area of the first partial fields of view of the pair of optical channels of the first plurality of optical channels is, as regards to the area, between ½ and 1/1000 of an average image size of the images that are captured by the first plurality of optical channels:
wherein a number of the second plurality of optical channels is between 1 and 100, both inclusive, and an overlap area of the second partial fields of view of a pair of optical channels of the second plurality of optical channels is, as regards to the area, between ½ and 1/1000 of an average image size of the images that are captured by the first plurality of optical channels; and
wherein the partial fields of view of the pair of optical channels of the first plurality of optical channels project into one another by a width of the overlap area of at least 20 pixels of the images that have been captured by the pair of optical channels of the first plurality of optical channels, at an image distance of 10 m.

2. The multi-aperture imaging device according to claim 1 , wherein the second arrangement of optical channels comprises exactly one optical channel.

3. The multi-aperture imaging device according to claim 2 , wherein the exactly one optical channel of the second arrangement is arranged between two optical channels of the first plurality of optical channels.

4. The multi-aperture imaging device of claim 2 , wherein the exactly one optical channel of the second arrangement is adapted to capture the at least part of the total field of view without being deflected by a beam-deflecting means; wherein the multi-aperture imaging device comprises a beam deflecting means for deflecting the first plurality of optical channels.

5. The multi-aperture imaging device according to claim 2 , wherein a field of view of the exactly one optical channel of the second arrangement corresponds to a field of view of one of the pair of optical channels within a tolerance range.

6. The multi-aperture imaging device according to claim 2 , wherein a field of view of the exactly one optical channel of the second arrangement corresponds to the total field of view.

7. The multi-aperture imaging device according to claim 6 , wherein a scene captured by the second arrangement of optical channels is larger than the total field of view.

8. The multi-aperture imaging device according to claim 1 , wherein the first arrangement of optical channel comprises exactly two optical channels.

9. The multi-aperture imaging device according to claim 1 , wherein the second arrangement of optical channels covers the total field of view.

10. The multi-aperture imaging device according to claim 1 , wherein the first arrangement comprises exactly two optical channels that are arranged along a line extension direction, wherein to project a first partial field of view and a second partial field of view that are arranged along an image direction in the object area that is perpendicular to the line extension direction.

11. The multi-aperture imaging device of claim 1 being 3D multi-aperture imaging device.

12. The multi-aperture imaging device according to claim 1 , wherein a base distance between the first arrangement of optical channels and the second arrangement of optical channels is greater than a single channel base distance between two adjacent optical channels of the first plurality of optical channels.

13. The multi-aperture imaging device according to claim 1 , wherein the processor is configured to
fuse images that have been captured by the second arrangement of optical channels to a second total image and to use, in an overlap area of second partial fields of view of a pair of optical channels of the second arrangement of optical channels, disparities in a further pair of images, wherein at least one of them has been captured by one of the first plurality of optical channels whose projected first partial field of view overlaps with the overlap area of the second partial fields of view.

14. The multi-aperture imaging device according to claim 1 , wherein the processor is configured to obtain a second total image from the second arrangement of optical channels; and to
generate a depth map by using disparities between the first and second total images.

15. The multi-aperture imaging device according to claim 13 , wherein the processor is configured to
warp the first or the second total image into a virtual optical center or to warp and to fuse the first and second total images into the virtual view in order to acquire a final total image to which the depth map belongs.

16. The multi-aperture imaging device according to claim 1 , wherein the processor is configured to
determine, for a depth map, a depth value for each image point of the first total image.

17. The multi-aperture imaging device according to claim 1 , wherein the processor is configured such that the first partial fields of view to which the first plurality of optical channels is allocated, are mutually different and congruent to the at least one second partial field of view to which the second arrangement of optical channels is allocated.

18. The multi-aperture imaging device according to claim 1 , wherein the processor is configured to
generate a depth map for the first total image by using disparities in pairs of images comprising, at least for each of the first plurality of optical channels, at least one pair comprising an image that has been captured by the respective one of the first plurality of optical channels and a further image that has been captured by an optical channel of the second arrangement of optical channels.

19. The multi-aperture imaging device according to claim 1 , wherein the processor is configured to determine the disparities by using a cross-correlation of local sections of the pair of images.

20. A method for fusing images that have been captured by a multi-aperture imaging device comprising an image sensor, a first arrangement of optical channels having a first plurality of optical channels for projecting overlapping first partial fields of view of a total field of view on first image sensor areas of the image sensor and a second arrangement of optical chann

Assignees

• Fraunhofer Ges Forschung

Inventors

• Wippermann Frank
• Brückner Andreas
• BRäUER ANDREAS
• Oberdörster Alexander

Classifications

• H04N23/698

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• G06T5/50 Primary

  using two or more images, e.g. averaging or subtraction · CPC title

• H04N23/45

  for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images · CPC title

• H04N23/95

  Computational photography systems, e.g. light-field imaging systems · CPC title

• G06T2200/21

  involving computational photography · CPC title