Optical flow estimation using 4-dimensional cost volume processing

Patent metadata

Field	Value
Publication number	US-10467768-B2
Application number	US-201715482270-A
Country	US
Kind code	B2
Filing date	Apr 7, 2017
Priority date	Apr 7, 2017
Publication date	Nov 5, 2019
Grant date	Nov 5, 2019

Abstract

Techniques are provided for estimation of optical flow between images using 4-dimensional cost volume processing. A methodology implementing the techniques according to an embodiment includes extracting a first set of feature vectors from a first image and extracting a second set of feature vectors from a second image. Each feature vector of the first set is associated with a pixel of the first image and each feature vector of the second set is associated with a pixel of the second image. The method further includes constructing a 4-dimensional (4D) cost volume to store a distance metric between each feature vector of the first set of feature vectors and a selected subset of feature vectors of the second set of feature vectors. The method further includes performing a flow-semi-global matching (Flow-SGM) on the 4D cost volume to estimate an optical flow vector for pixels of the first image.

First claim

What is claimed is:

1. A processor-implemented method for optical flow estimation, the method comprising:
extracting, by a processor-based system, a first set of feature vectors from a first image, each feature vector of the first set associated with a pixel of the first image;
extracting, by the processor-based system, a second set of feature vectors from a second image, each feature vector of the second set associated with a pixel of the second image;
constructing, by the processor-based system, a 4-dimensional (4D) cost volume to store distance metrics between one or more feature vectors of the first set of feature vectors and one or more feature vectors of the second set of feature vectors; and
performing, by the processor-based system, a flow-semi-global matching (Flow-SGM) on the 4D cost volume to estimate an optical flow vector for pixels of the first image.

2. The method of claim 1 , further comprising normalizing the extracted feature vectors to unity length, and calculating each of the distance metrics as a Euclidean distance using a vector dot product operation applied to the normalized extracted feature vectors.

3. The method of claim 1 , further comprising rescaling and binning the distance metrics of the 4D cost volume to a selected integer range.

4. The method of claim 1 , wherein the feature vector extraction is performed by a trained convolutional neural network (CNN), the training based on training data comprising pairs of training images and associated ground truth optical flow vectors.

5. The method of claim 4 , wherein the training further comprises performing a stochastic gradient descent operation on the training data.

6. The method of claim 1 , further comprising:
down-sampling the first image and the second image, from an original resolution to a selected lower resolution;
generating an estimated optical flow field comprising the estimated optical flow vector; and
up-sampling the estimated optical flow field to the original resolution using interpolation.

7. The method of claim 6 , further comprising post processing of the up-sampled estimated optical flow field to in-fill occluded regions, the in-filling based on extrapolation performed within homography fitted segments of the up-sampled estimated optical flow field.

8. The method of claim 1 , further comprising providing the estimated optical flow vector to at least one of a video segmentation application, a motion detection application, an object tracking application, an action recognition application, an autonomous driving system, a computer navigation application, and a computer vision application.

9. The method of claim 1 , wherein the constructing comprises constructing a 4-dimensional (4D) cost volume to store a distance metric between each feature vector of the first set of feature vectors and a selected subset of feature vectors of the second set of feature vectors.

10. A system for optical flow estimation, the system comprising:
a feature extraction circuit to extract a first set of feature vectors from a first image, each feature vector of the first set associated with a pixel of the first image; and to extract a second set of feature vectors from a second image, each feature vector of the second set associated with a pixel of the second image;
a cost volume construction circuit to construct a 4-dimensional (4D) cost volume to store a distance metric between each feature vector of the first set of feature vectors and a selected subset of feature vectors of the second set of feature vectors; and
a cost volume processing circuit to perform a flow-semi-global matching (Flow-SGM) on the 4D cost volume to estimate an optical flow vector for pixels of the first image and to generate an estimated optical flow field comprising the estimated optical flow vector.

11. The system of claim 10 , wherein the cost volume construction circuit is further to normalize the extracted feature vectors to unity length, and calculate the distance metric as a Euclidean distance using a vector dot product operation applied to the normalized extracted feature vectors.

12. The system of claim 10 , wherein the cost volume construction circuit is further to rescale and bin the distance metrics of the 4D cost volume to a selected integer range.

13. The system of claim 10 , wherein the feature extraction circuit further comprises a trained convolutional neural network (CNN) to extract the feature vectors, the training based on training data comprising pairs of training images and associated ground truth optical flow vectors.

14. The system of claim 13 , further comprising a training system to train the CNN based on application of a stochastic gradient descent to the training data.

15. The system of claim 10 , further comprising:
an image down-sampling circuit to down-sample the first image and the second image, from an original resolution to a selected lower resolution; and
an up-sampling circuit to up-sample the estimated optical flow field to the original resolution using interpolation.

16. The system of claim 15 , further comprising a post-processing circuit to in-fill occluded regions of the up-sampled estimated optical flow field, the in-filling based on extrapolation performed within homography fitted segments of the up-sampled estimated optical flow field.

17. The system of claim 10 , wherein the post-processing circuit is further to provide the estimated optical flow vector to at least one of a video segmentation application, a motion detection application, an object tracking application, an action recognition application, an autonomous driving system, a computer navigation application, and a computer vision application.

18. At least one non-transitory computer readable storage medium having instructions encoded thereon that, when executed by one or more processors, result in the following operations for optical flow estimation, the operations comprising:
extracting a first set of feature vectors from a first image, each feature vector of the first set associated with a pixel of the first image;
extracting a second set of feature vectors from a second image, each feature vector of the second set associated with a pixel of the second image;
constructing a 4-dimensional (4D) cost volume to store a distance metric between each feature vector of the first set of feature vectors and a selected subset of feature vectors of the second set of feature vectors; and
performing a flow-semi-global matching (Flow-SGM) on the 4D cost volume to estimate an optical flow vector for pixels of the first image or the second image or both of the first image and second image.

19. The computer readable storage medium of claim 18 , further comprising the operations of normalizing the extracted feature vectors to unity length, and calculating the distance metric as a Euclidean distance using a vector dot product operation applied to the normalized extracted feature vectors.

20. The computer readable storage medium of claim 18 , further comprising the operations of rescaling and binning the distance metrics of the 4D cost volume to a selected integer range.

21. The computer readable storage medium of claim 18 , wherein the feature vector extraction is performed by a trained convolutional neural network (CNN), the training based on training data comprising pairs of training images and associated ground truth optical flow vectors.

22. The computer readable storage medium of claim 21 , wherein the training further comprises the operati

Assignees

• Intel Corp

Inventors

• Xu Jia
• Ranftl René
• Koltun Vladlen

Classifications

• G06V10/82 Primary

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06T7/60 Primary

  Analysis of geometric attributes · CPC title

• G06F18/2413

  based on distances to training or reference patterns · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title