Hardware-oriented dynamically adaptive disparity estimation algorithm and its real-time hardware

Patent metadata

Field	Value
Publication number	US-9756312-B2
Application number	US-201414267140-A
Country	US
Kind code	B2
Filing date	May 1, 2014
Priority date	May 1, 2014
Publication date	Sep 5, 2017
Grant date	Sep 5, 2017

Abstract

A real-time stereo camera disparity estimation device comprises input means arranged to input measured data corresponding to rows of left and right images; a plurality of on-chip memories arranged to buffer the input measured data; a vertical rotator hardware module configured to align the rows of left and right images in a same column; a reconfigurable data allocation hardware module; a reconfigurable computation of metrics hardware module; and an adaptive disparity selection hardware module configured to select disparity values with the minimum matching costs.

First claim

The invention claimed is:

1. A real-time stereo camera disparity estimation device comprising:
an input device arranged to input measured data corresponding to rows of left and right images;
a memory device arranged to buffer the input measured data;
a processor for rotating data to align the rows of left and right images in a same column;
a processor for data allocation to create variable window sizes to adapt a window size based on a local texture on the left and right images;
a processor for metrics calculation to compute stereo matching costs for disparities between the left and right images within a block within the adapted window size within a block within the adapted window size; and
a processor for adaptive disparity selection configured to select disparity values with minimum matching costs from the stereo matching costs as disparity results.

2. The device of claim 1 , further comprising:
a processor for an iterative disparity refinement configured to iteratively refine the disparity values of the disparity results of the processor for adaptive disparity selection.

3. The device of claim 1 , wherein the processor for data allocation is further configured to create variable window sizes to adapt the window size to a local texture on the at least one of left and right image.

4. The device of claim 1 , wherein the processor for metrics calculation comprises:
a plurality of processing devices for multiple processed pixels in a two dimensional block to compute their stereo matching costs for candidate disparities in parallel.

5. The device of claim 1 , wherein the memory device comprises:
dual-ports configured to write and read concurrently;
a connection of read address ports to a same read address request of the processors to allow the processors to read multiple rows and a same column of the image in parallel, the memory device configured to store YCbCr or RGB data for the pixels.

6. The device of claim 5 , wherein pixels of different rows are stored in separate block RAMs of the memory device to be able to access multiple pixels in the same column in parallel.

7. The device of claim 6 , wherein the data in the block RAMs are overwritten by new rows of at least one of the left and right image after they are processed by at least one of the processors.

8. The device of claim 1 , wherein the processor for rotating data is further configured to rotate either Y, Cb or Cr, either R, G or B to make disparity estimation in any of the selected pixel data channel, and to rotate and align either left image pixels or right image pixels.

9. The device of claim 3 , wherein the processor for data allocation to create variable window sizes comprises,
a flip-flop array configured to store and shift aligned outputs of the processor for rotating data;
wires connected to the flip-flops array arranged to sample the pixels while pixels are flowing inside the flip-flops array;
a plurality of first sampling schemes to provide the variable window sizes;
a plurality of second sampling schemes to provide constant number of contributing pixels in the variable window sizes to provide constant computational load for the variable window sizes; and
a plurality of multiplexers configured to select windows to a selected window size to be used in disparity estimation process of multiple pixels in a block according to the selected window size.

10. The device of claim 9 , wherein the selection of window size is determined depending on a variance of neighboring pixels for variable window sizes.

11. The device of claim 9 , wherein a same one of the selected window size is applied to multiple searched pixels in a block.

12. The device of claim 9 , wherein for every searched block of pixels, window size is dynamically re-determined.

13. The device of claim 4 , wherein the plurality of processing devices are configured for a computation of metrics and comprises,
a plurality of census, Hamming, SAD and BW-SAD cost computation devices for concurrent and independent disparity search of multiple pixels in the two dimensional block; and
a selection device configured for a configurability through selection either of SAD or BW-SAD cost computations.

14. The device of claim 13 , wherein the plurality of processing devices are configured to perform:
SAD and BW-SAD computations for sampled pixels in a searched block to reduce an overall computational complexity;
interpolation of SAD and BW-SAD values of the sampled pixels in the block to compute and estimate the SAD and BW-SAD values of all the remaining pixels in the searched block for which SAD and BW-SAD are not computed; and
Hamming computations for all the pixels in the searched block.

15. The device of claim 13 , wherein the processor for disparity selection comprises:
a multiplier to normalize a hamming cost using adaptive penalties; and
an adder for adding multiplied hamming value with SAD result to compute hybrid cost.

16. The device of claim 15 , wherein the adaptive penalties are in an order of two to simplify an implementation of multipliers with shifters.

17. The device of claim 15 , wherein small penalty values are used for small window size, and big penalty values are used for big window size.

18. The device of claim 2 , wherein the processor for iterative disparity refinement to refine the disparity values comprises:
a flip-flop array to store and shift the disparity results; and
a highest frequency selection device configured to determine most frequent disparity values to replace processed disparity values with the most frequent ones to establish the disparity results.

19. The device of claim 18 , wherein the highest frequency selection hardware device is configured to determine the most frequent disparity values and refine the disparities using the color similarity of neighboring pixels.

20. The device of claim 18 , wherein multiple rows are refined in parallel using a plurality of highest frequency selection devices to determine the most frequent disparity values.

21. The device of claim 18 , wherein the disparity results are iteratively refined.

22. The device of claim 18 , wherein the disparity results are iteratively refined by processing multiple consecutive columns using a plurality of highest frequency selection devices.

23. The device of claim 18 , wherein the refined disparity values are written back to the flip flop array to iteratively use refined disparity values for further refinements.

24. The device of claim 18 , wherein final shifted disparity values at an end of the flip flop array are used as the output of the disparity estimation device.

Assignees

• Ecole Polytechnique Fed De Lausanne (Epfl)
• Ecole Polytechnique Fed Lausanne Epfl

Inventors

• Akin Abdulkadir
• Leblebici Yusuf
• Schmid Alexandre
• Baz Ipek
• Boybat Irem
• Atakan Huseyin Baris

Classifications

• G06T7/97

  Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85) · CPC title

• G06T2207/10021

  Stereoscopic video; Stereoscopic image sequence · CPC title

• G06T2207/10012

  Stereo images · CPC title

• G06T1/20

  Processor architectures; Processor configuration, e.g. pipelining · CPC title

• H04N13/0207

  Electricity · mapped topic