Method and system for feature tracking using image pyramids

Patent metadata

Field	Value
Publication number	US-11100652-B2
Application number	US-201716474457-A
Country	US
Kind code	B2
Filing date	Jan 24, 2017
Priority date	Jan 24, 2017
Publication date	Aug 24, 2021
Grant date	Aug 24, 2021

Abstract

Described herein are systems, methods, storage media, and computer programs for tracking a plurality of feature points in a first image and a second image. In one embodiment, a first layer of a first image pyramid for the first image and a first layer of a second image pyramid for the second image are obtained. Based on expected pixel locations of the plurality of feature points in the first layer of the second image pyramid, a feature point is selected from the plurality of feature points. For the feature point, a first pixel in the first layer of the first image pyramid is determined, and a second pixel in the first layer of the second image pyramid is determined, where the first pixel substantially matches the second pixel.

First claim

What is claimed is:

1. A method for tracking a plurality of feature points in a first image and a second image, comprising:
obtaining a first layer of a first image pyramid for the first image and a first layer of a second image pyramid for the second image, wherein both the first and second image pyramids have a plurality of layers and the first layer of the first image pyramid for the first image corresponds to the first layer of the second image pyramid for the second image;
selecting a feature point from the plurality of feature points, based on expected pixel locations of the plurality of feature points in the first layer of the second image pyramid for the second image; and
determining, for the feature point, a first pixel in the first layer of the first image pyramid for the first image and a second pixel in the first layer of the second image pyramid for the second image, wherein the second pixel is a matching point of the first pixel.

2. The method of claim 1 , further comprising:
forming the first and second image pyramids for the first and second images respectively.

3. The method of claim 1 , wherein the first and second image pyramids have a same number of layers.

4. The method of claim 1 , wherein a layer immediately above the first layer of the first image pyramid is a topmost layer of the first image pyramid.

5. The method of claim 1 , further comprising:
obtaining, for the feature point, an expected pixel location in a second layer of the second image pyramid for the second image.

6. The method of claim 5 , wherein the first layer of the first image pyramid is a layer immediately above a second layer of the first image pyramid, wherein the second layer of the first image pyramid corresponds to the second layer in the second image pyramid.

7. The method of claim 5 , wherein the expected pixel location for the feature point in the second layer of the second image pyramid for the second image is determined based on an offset obtained at the first layer of the second image pyramid for the feature point.

8. The method of claim 5 , wherein the expected pixel location for the feature point in the second layer of the second image pyramid is determined based on gradient information of the feature point at the first layer of the first image pyramid.

9. The method of claim 5 , wherein the expected pixel location for the feature point in the second layer of the second image pyramid for the second image is determined based on optical flow information obtained for the feature point at the first layer of the second image pyramid.

10. The method of claim 1 , further comprising:
obtaining expected pixel locations, for the plurality of feature points, in a second layer of the second image pyramid for the second image, in an order determined based on the expected pixel locations of the plurality of feature points in the first layer of the second image pyramid for the second image.

11. The method of claim 10 , wherein the order is a raster scanning order or a zig-zag order.

12. The method of claim 1 , wherein the first pixel and the second pixel are matched based on pixel values.

13. The method of claim 12 , further comprising:
obtaining pixel values for pixels at consecutive pixel locations in the first layer of the second image pyramid for the second image.

14. The method of claim 1 , wherein the first layer of the first image pyramid for the first image and the first layer of the second image pyramid for the second image are obtained from a storage.

15. The method of claim 14 , wherein the first layer of the first image pyramid for the first image and the first layer of the second image pyramid for the second image are obtained from a double data rate synchronous dynamic random-access memory (DDR SDRAM) of the storage.

16. An electronic device to track a plurality of feature points in a first image and a second image, comprising:
a processor and a non-transitory machine-readable storage medium that is coupled to the processor, the non-transitory machine-readable storage medium containing instructions, which when executed by the processor, cause the electronic device to:
obtain data for a first layer of a first image pyramid for the first image and a first layer of a second image pyramid for the second image, wherein both the first and second image pyramids have a plurality of layers and the first layer of the first image pyramid for the first image corresponds to the first layer of the second image pyramid for the second image,
select a feature point from the plurality of feature points, based on expected pixel locations of the plurality of feature points in a first layer of a second image pyramid for the second image; and
determine, for the feature point, a first pixel in the first layer of the first image pyramid for the first image and a second pixel in the first layer of the second image pyramid for the second image based on the obtained data, wherein the second pixel is a matching point of the first pixel.

17. The electronic device of claim 16 , further comprising a storage configured to store the first and second image pyramids formed for the first and second image pyramids respectively, wherein the electronic device is configured to store, in the storage, the data for the first layer of the first image pyramid for the first image and the first layer of the second image pyramid for the second image.

18. The electronic device of claim 17 , wherein the storage comprises a double data rate synchronous dynamic random-access memory (DDR SDRAM) to store the first and second image pyramids.

19. The electronic device of claim 16 , wherein a layer immediately above the first layer of the first image pyramid is a topmost layer of the first image pyramid.

20. A vision accelerator to track a plurality of feature points in a first image and a second image, the vision accelerator comprising:
a cache configured to store data for at least a first layer of a first image pyramid for the first image and a first layer of a second image pyramid for the second image, wherein both the first and second image pyramids have a plurality of layers and the first layer of the first image pyramid for the first image corresponds to the first layer of the second image pyramid for the second image; and
a circuit coupled to the cache, the circuit configured to
select a feature point from the plurality of feature points, based on expected pixel locations of the plurality of feature points in a first layer of a second image pyramid for the second image, and

determine, for the feature point, a first pixel in the first layer of the first image pyramid for the first image and a second pixel in the first layer of the second image pyramid for the second image, wherein the second pixel is a matching point of the first pixel.

Assignees

• Sz Dji Technology Co Ltd

Inventors

• Gao Mingming
• Yang Kang

Classifications

• G06T7/246 Primary

  using feature-based methods, e.g. the tracking of corners or segments · CPC title

• G06T7/248 Primary

  involving reference images or patches · CPC title

• G06T7/269

  using gradient-based methods · CPC title

• G06T2207/20016

  Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform · CPC title

• G06K9/6202

  Physics · mapped topic