Stereo matching method and apparatus

Patent metadata

Field	Value
Publication number	US-11417006-B2
Application number	US-202017028195-A
Country	US
Kind code	B2
Filing date	Sep 22, 2020
Priority date	Jan 26, 2017
Publication date	Aug 16, 2022
Grant date	Aug 16, 2022

Abstract

A stereo matching method includes extracting feature points of a first image and feature points of a second image, the first image and the second image together constituting a stereo image, determining reference points by matching the feature points of the second image to the feature points of the first image, classifying the reference points, and performing stereo matching on pixels of which disparities are not determined in the first image and the second image based on disparities of the reference points in the pixels determined based on a result of the classifying.

First claim

What is claimed is:

1. A stereo matching method, comprising:
extracting feature points of a first image and feature points of a second image, the first image and the second image together constituting a stereo image;
determining reference points including a disparity value of each of the reference points by matching the feature points of the second image to the feature points of the first image;
classifying classes of the reference points based on at least one of a consistency and a correlation of the reference points; and
performing, based on a reliability of the disparities of the reference points for each class, stereo matching on pixels of which disparities are not determined in the first image and the second image by propagating a first disparity value of a reference point or propagating a second disparity value of another reference point adjacent to the reference point.

2. The stereo matching method of claim 1 , wherein the performing of stereo matching comprises performing stereo matching on the pixels by updating disparity values of the reference points based on the reliability of the disparities of the reference points.

3. The stereo matching method of claim 2 , wherein the performing of stereo matching on the pixels by updating the disparity values further comprises:
based on the reliability of the disparities of the reference points,
propagating a disparity value of a reference point classified into a third class to a disparity value of a reference point classified into at least one of a first class or a second class, the reference point classified into the third class being adjacent to the reference point classified into at least one of the first class or the second class among the reference points; and
performing stereo matching on the pixels based on the propagated disparity value.

4. The stereo matching method of claim 3 , wherein the performing stereo matching on the pixels based on the propagated disparity value further comprises:
determining a polygon that uses the reference points respectively classified into the first class, the second class, and the third class as vertices;
determining a search range for calculating disparities of pixels present in the polygon based on the propagated disparity value; and
performing stereo matching on the pixels based on the search range.

5. The stereo matching method of claim 2 , wherein the performing of stereo matching on the pixels by updating the disparity values of the reference points comprises:
resetting a window that extracts the disparity values of the reference points classified into the first class and the second class from each of the first image and the second image; and
performing stereo matching on the pixels based on the reset window.

6. The stereo matching method of claim 5 , wherein the resetting of the window comprises:
resetting the window based on at least one of a shifted window having an adjustable angle and multiple windows having identical sizes with respect to a reference point of which a depth is discontinued from adjacent reference points among the reference points classified into the first class and the second class; and
resetting the window based on an extension window obtained by extending a size of the window with respect to a reference point of which a depth is continued from the adjacent reference points among the reference points classified into the first class and the second class, based on a result of the classifying.

7. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1 .

8. A stereo matching apparatus, comprising:
a processor; and
a memory configured to store a computer-readable instruction,
wherein, in response to execution of the instruction at the processor, the processor is configured to:
extract feature points of a first image and feature points of a second image, the first image and the second image together constituting a stereo image;
determine reference points including a disparity value of each of the reference points by matching the feature points of the second image to the feature points of the first image;
classify classes of the reference points based on at least one of a consistency and a correlation of the reference points; and
perform, based on a reliability of the disparities of the reference points for each class, stereo matching on pixels of which disparities are not determined in the first image and the second image by propagating a first disparity value of a reference point or propagating a second disparity value of another reference point adjacent to the reference point.

9. The stereo matching apparatus of claim 8 , wherein the processor is further configured to perform stereo matching on the pixels by updating disparity values of the reference points based on the reliability of the disparities of the reference points.

10. The stereo matching apparatus of claim 9 , wherein the processor is further configured to, based on the reliability of the disparities of the reference points,
propagate a disparity value of a reference point classified into a third class to a disparity value of a reference point classified into at least one of a first class or a second class, the reference point classified into the third class being adjacent to the reference point classified into at least one of the first class or the second class among the reference points, and
perform stereo matching on the pixels based on the propagated disparity value.

11. The stereo matching apparatus of claim 10 , wherein the processor is further configured to:
determine a polygon that uses the reference points respectively classified into the first class, the second class, and the third class as vertices,
determine a search range for calculating disparities of pixels in the polygon based on the propagated disparity value, and
perform stereo matching on the pixels based on the search range.

12. The stereo matching apparatus of claim 9 , wherein the performing of stereo matching on the pixels by updating the disparity values of the reference points comprises:
resetting a window that extracts the disparity values of the reference points classified into the first class and the second class from each of the first image and the second image; and
performing stereo matching on the pixels based on the reset window.

13. The stereo matching apparatus of claim 12 , wherein the resetting of the window comprises:
resetting the window based on at least one of a shifted window having an adjustable angle and multiple windows having identical sizes with respect to a reference point of which a depth is discontinued from adjacent reference points among the reference points classified into the first class and the second class; and
resetting the window based on an extension window obtained by extending a size of the window with respect to a reference point of which a depth is continued from the adjacent reference points among the reference points classified into the first class and the second class, based on a result of the classifying.

Assignees

• Samsung Electronics Co Ltd

Inventors

• Park Seungin
• Ahn Minsu
• Kim Jiyeon

Classifications

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• H04N13/122

  Improving the three-dimensional [3D] impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues (H04N13/128 takes precedence) · CPC title

• G06T2207/10012

  Stereo images · CPC title

• H04N2013/0081

  Depth or disparity estimation from stereoscopic image signals · CPC title

• G06V30/413

  Classification of content, e.g. text, photographs or tables · CPC title