Disparity estimation for multiview imaging systems

Patent metadata

Field	Value
Publication number	US-9888229-B2
Application number	US-201414312586-A
Country	US
Kind code	B2
Filing date	Jun 23, 2014
Priority date	Jun 23, 2014
Publication date	Feb 6, 2018
Grant date	Feb 6, 2018

Abstract

This invention presents an approach to first estimate a depth/disparity map using spectrally coded plenoptic camera, and then based on the disparity map the parallax between different spectral channels is rectified. Based on our new technique, we can reconstruct not only multispectral images but also a depth/disparity map. Moreover, the quality of reconstructed spectral images is significantly improved using our parallax rectification.

First claim

What is claimed is:

1. A computer-implemented method for estimating disparities from multiview images, the method comprising:
receiving N images of a scene, N≧3, the images taken from different viewpoints;
modeling disparity between any pair of the N images as a function of (a) a disparity parameter applicable to all pairs of images, and (b) a viewpoint difference between the viewpoints of the images in the pair; and
estimating the disparity parameter based on a set of pairs of images from the N images, based on the modeling of disparity for each pair in the set, and based on the viewpoint difference between viewpoints for each pair in the set.

2. The computer-implemented method of claim 1 wherein estimating the disparity parameter comprises:
estimating the disparity parameter further based on increasing correlations between the pairs in the set after parallax rectification.

3. The computer-implemented method of claim 1 wherein the viewpoint difference between two viewpoints for a pair of images is defined by a baseline between the two viewpoints, the baseline having a baseline distance and an angle of the baseline.

4. The computer-implemented method of claim 3 wherein the disparity between two viewpoints for a pair of images is modeled as (disparity between the two viewpoints)/(baseline distance between the two viewpoints)=a known function.

5. The computer-implemented method of claim 1 wherein one of the N images is selected as a reference image, the set of pairs are pairs consisting of the reference image and each of the other N images, and estimating the disparity parameter is based on the viewpoint differences between the reference image and each of the other images.

6. The computer-implemented method of claim 1 wherein the disparity between any pair of the N images is further a function of a location in the image.

7. The computer-implemented method of claim 1 wherein different images contain different spectral content and not all objects are visible in all images due to the different spectral content of the images.

8. The computer-implemented method of claim 1 further comprising:
estimating a disparity map for the images based on the disparity parameter and the viewpoint differences, the disparity map defining disparity as a function of location in the images.

9. The computer-implemented method of claim 1 further comprising:
estimating a depth map for the images based on the disparity parameter and the viewpoint differences and on a relationship between disparity and depth, the depth map defining depth as a function of location in the images.

10. The computer-implemented method of claim 1 further comprising:
applying parallax rectification to the images, the parallax rectification based on the disparity parameter and the viewpoint differences.

11. The computer-implemented method of claim 10 further comprising:
combining the parallax rectified images into a multispectral image.

12. The computer-implemented method of claim 1 wherein the images include red, green and blue filtered images.

13. The computer-implemented method of claim 1 wherein the viewpoints for the images are arranged in a grid.

14. The computer-implemented method of claim 13 wherein N=4 and the viewpoints for the four images are arranged in a square.

15. The computer-implemented method of claim 1 wherein the N images are captured by N separate cameras.

16. The computer-implemented method of claim 1 wherein the N images are captured by a single plenoptic camera.

17. A non-transitory computer readable medium configured to store program code, the program code comprising instructions for estimating disparities from multiview images, the instructions when executed by a processor cause the processor to execute a method comprising:
receiving N images of a scene, N≧3, the images taken from different viewpoints;
modeling disparity between any pair of the N images as a function of (a) a disparity parameter applicable to all pairs of images, and (b) a viewpoint difference between the viewpoints of the pair of images; and
estimating the disparity parameter based on a set of pairs of images from the N images, based on the modeling of disparity for each pair in the set, and based on the viewpoint differences between viewpoints for each pair in the set.

18. The non-transitory computer readable medium of claim 17 wherein different images contain different spectral content.

19. A multiview camera system comprising:
one or more cameras that capture N images of a scene, N≧3, the images taken from different viewpoints; and
a processing module configured to:
receive the N images;
model disparity between any pair of the N images as a function of (a) a disparity parameter applicable to all pairs of images, and (b) a viewpoint difference between the viewpoints of the pair of images; and
estimate the disparity parameter based on a set of pairs of images from the N images, based on the modeling of disparity for each pair in the set, and based on the viewpoint differences between viewpoints for each pair in the set.

20. The multiview camera system of claim 19 wherein different images contain different spectral content.

Assignees

• Meng Lingfei
• Berkner Kathrin
• Ricoh Co Ltd

Inventors

• Meng Lingfei
• Berkner Kathrin

Classifications

• G06T7/557

  from light fields, e.g. from plenoptic cameras · CPC title

• G06T2207/10052

  Images from lightfield camera · CPC title

• H04N13/282 Primary

  for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems · CPC title

• H04N13/232

  using fly-eye lenses, e.g. arrangements of circular lenses · CPC title

• H04N13/0282 Primary

  Electricity · mapped topic