System and method for optical flow estimation

Patent metadata

Field	Value
Publication number	US-10424069-B2
Application number	US-201815942213-A
Country	US
Kind code	B2
Filing date	Mar 30, 2018
Priority date	Apr 7, 2017
Publication date	Sep 24, 2019
Grant date	Sep 24, 2019

Abstract

A method, computer readable medium, and system are disclosed for estimating optical flow between two images. A first pyramidal set of features is generated for a first image and a partial cost volume for a level of the first pyramidal set of features is computed, by a neural network, using features at the level of the first pyramidal set of features and warped features extracted from a second image, where the partial cost volume is computed across a limited range of pixels that is less than a full resolution of the first image, in pixels, at the level. The neural network processes the features and the partial cost volume to produce a refined optical flow estimate for the first image and the second image.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
generating a first pyramidal set of features for a first image;
computing, by a neural network, a partial cost volume for a level of the first pyramidal set of features using features at the level of the first pyramidal set of features and warped features extracted from a second image, wherein the partial cost volume is computed across a limited range of pixels that is less than a full resolution of the first image, in pixels, at the level; and
processing, by the neural network, the features and the partial cost volume to produce an optical flow estimate for the first image and the second image.

2. The computer-implemented method of claim 1 , wherein the features extracted from the second image are included in a second pyramidal set of features extracted from the second image.

3. The computer-implemented method of claim 2 , wherein the features extracted from the second image for the level of the second pyramidal set of images are warped toward the features extracted from the first image using an optical flow estimate for a second level of the first pyramidal set of images to produce the warped features.

4. The computer-implemented method of claim 1 , further comprising:
convolving the first image with overlapping filters to extract the features for a first level of the first pyramidal set of features;
convolving the first level with additional filters to extract the features for a subsequent level of the first pyramidal set of features; and
the convolving the subsequent level with additional filters to extract the features for another subsequent level of the first pyramidal set of features until a last level of the first pyramidal set of features is generated.

5. The computer-implemented method of claim 1 , wherein a single layer of the neural network generates the warped features.

6. The computer-implemented method of claim 1 , wherein the neural network is a convolutional neural network.

7. The computer-implemented method of claim 1 , wherein a single layer of the neural network computes the partial cost function.

8. The computer-implemented method of claim 1 , wherein the optical flow estimate is computed based on a previous optical flow estimate produced using a previous level of the first pyramidal set of features.

9. The computer-implemented method of claim 8 , further comprising upscaling the previous optical flow estimate before computing the optical flow estimate.

10. The computer-implemented method of claim 9 , wherein an initial optical flow estimate of zero is used to compute the previous optical flow estimate.

11. The computer-implemented method of claim 8 , further comprising repeating the computing and processing for each level in the first pyramidal set of features.

12. The computer-implemented method of claim 1 , wherein the second image is after the first image in a video sequence.

13. The computer-implemented method of claim 1 , further comprising processing the optical flow estimate by a context network to produce a refined optical flow estimate.

14. A system, comprising:
a parallel processing unit configured to implement a neural network and a pyramidal image feature structure generator, wherein
the pyramidal image feature structure generator is configured to generate a first pyramidal set of features for a first image and
the neural network is configured to generate an optical flow estimate for the first image and a second image by:
computing a partial cost volume for a level of the first pyramidal set of features using features at the level of the first pyramidal set of features and warped features extracted from the second image, wherein the partial cost volume is computed across a limited range of pixels that is less than a full resolution of the first image, in pixels, at the level; and
processing the features and the partial cost volume to produce an optical flow estimate for the first image and the second image.

15. The system of claim 14 , wherein the pyramidal image feature structure generator is further configured to generate a second pyramidal set of features for the second image that includes the features extracted from the second image.

16. The system of claim 15 , further comprising a warping layer that is configured to warp the features extracted from the second image for the level of the second pyramidal set of images toward the features extracted from the first image using an optical flow estimate for a second level of the first pyramidal set of images to produce the warped features.

17. The system of claim 14 , wherein the pyramidal image feature structure generator generates the first pyramidal set of features for a first image by:
convolving the first image with overlapping filters to extract the features for a first level of the first pyramidal set of features;
convolving the first level with additional filters to extract the features for a subsequent level of the first pyramidal set of features; and
the convolving the subsequent level with additional filters to extract the features for another subsequent level of the first pyramidal set of features until a last level of the first pyramidal set of features is generated.

18. The system of claim 14 , wherein a single layer of the neural network generates the warped features.

19. The system of claim 14 , wherein the neural network is a convolutional neural network.

20. A non-transitory computer-readable media storing computer instructions for estimating optical flow that, when executed by a processor, cause the processor to perform the steps of:
generating a first pyramidal set of features for a first image;
computing, by a neural network, a partial cost volume for a level of the first pyramidal set of features using features at the level of the first pyramidal set of features and warped features extracted from a second image, wherein the partial cost volume is computed across a limited range of pixels that is less than a full resolution of the first image, in pixels, at the level; and
processing, by the neural network, the features and the partial cost volume to produce an optical flow estimate for the first image and the second image.

Assignees

• Nvidia Corp

Inventors

• Sun Deqing
• Yang Xiaodong
• Liu Ming-Yu
• Kautz Jan

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06T7/269

  using gradient-based methods · CPC title

• G06T7/246

  using feature-based methods, e.g. the tracking of corners or segments · CPC title

• G06T2200/28

  involving image processing hardware · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title