Sliding window for image keypoint detection and descriptor generation

Patent metadata

Field	Value
Publication number	US-12526381-B2
Application number	US-202418444524-A
Country	US
Kind code	B2
Filing date	Feb 16, 2024
Priority date	Mar 8, 2021
Publication date	Jan 13, 2026
Grant date	Jan 13, 2026

Abstract

Embodiments relate to extracting features from images, such as by identifying keypoints and generating keypoint descriptors of the keypoints. An apparatus includes a pyramid image generator circuit, a keypoint descriptor generator circuit, and a pyramid image buffer. The pyramid image generator circuit generates an image pyramid from an input image. The keypoint descriptor generator circuit processes the pyramid images for keypoint descriptor generation. The pyramid image buffer stores different portions of the pyramid images generated by the pyramid image generator circuit at different times and provides the stored portions of the pyramid images to the keypoint descriptor generator circuit for keypoint descriptor generation. When first portions of the pyramid images are no longer needed for the keypoint descriptor generation, the first portions are removed from the pyramid image buffer to provide space for second portions of the pyramid images that are needed for the keypoint descriptor generation.

First claim

What is claimed is:

1 . An apparatus, comprising:
a response map (RM) generator circuit configured to generate RM images based on an image pyramid, the RM images comprising a first subset of RM images of a first octave and a second subset of RM images of a second octave; an intra octave keypoint candidate selector circuit configured to determine keypoint candidates of keypoints of an input image based on a comparison between at least two RM images in the first subset of RM images of the first octave; and an inter octave keypoint candidate selector circuit configured to determine keypoints from the keypoint candidates based on a comparison between the first subset of RM images and the second subset of RM images.

2 . The apparatus of claim 1 , further comprising:
a pyramid image buffer circuit configured to generate the image pyramid from the input image and store different portions of pyramid images of the image pyramid at different times.

3 . The apparatus of claim 2 , wherein the pyramid image buffer circuit has a storage size smaller than a data size of the image pyramid.

4 . The apparatus of claim 1 , further comprising:
a pyramid image buffer circuit configured to store different portions of pyramid images of the image pyramid at different times.

5 . The apparatus of claim 1 , further comprising:
an RM buffer circuit configured to store different portions of the RM images generated at different times.

6 . The apparatus of claim 5 , wherein the RM buffer circuit is configured to store partial RM images.

7 . The apparatus of claim 5 , wherein the RM buffer circuit is a static random-access memory (SRAM).

8 . The apparatus of claim 1 , further comprising an RM buffer circuit configured to:
provide first portions of the RM images stored in memory locations of the RM buffer circuit to the inter octave keypoint candidate selector circuit for keypoint determination; remove the first portions of the RM images from the memory locations responsive to the first portions being no longer needed for the keypoint determination by the inter octave keypoint candidate selector circuit; and store second portions of the RM images that are needed for the keypoint determination by the inter octave keypoint candidate selector circuit in the memory locations that previously stored the first portions of the RM images.

9 . A method, comprising:
generating, by a response map (RM) generator circuit, RM images based on an image pyramid, the RM images comprising a first subset of RM images of a first octave and a second subset of RM images of a second octave; determining, by an intra octave keypoint candidate selector circuit, keypoint candidates of keypoints of an input image based on a comparison between at least two RM images in the first subset of RM images of the first octave; and determining, by an inter octave keypoint candidate selector circuit, keypoints from the keypoint candidates based on a comparison between the first subset of RM images and the second subset of RM images.

10 . The method of claim 9 , further comprising:
generating, by a pyramid image buffer circuit, the image pyramid from the input image.

11 . The method of claim 10 , wherein the pyramid image buffer circuit has a storage size smaller than a data size of the image pyramid.

12 . The method of claim 9 , further comprising:
storing, by a pyramid image buffer circuit, different portions of pyramid images of the image pyramid at different times.

13 . The method of claim 9 , further comprising:
storing, by an RM buffer circuit, different portions of the RM images generated at different times.

14 . The method of claim 13 , wherein the RM buffer circuit is configured to store partial RM images.

15 . The method of claim 13 , wherein the RM buffer circuit is a static random-access memory (SRAM).

16 . The method of claim 9 , further comprising:
providing, by an RM buffer circuit, first portions of the RM images stored in memory locations of the RM buffer circuit to the inter octave keypoint candidate selector circuit for keypoint determination; removing, by the RM buffer circuit, the first portions of the RM images from the memory locations responsive to the first portions being no longer needed for the keypoint determination by the inter octave keypoint candidate selector circuit; and storing, by the RM buffer circuit, second portions of the RM images that are needed for the keypoint determination by the inter octave keypoint candidate selector circuit in the memory locations that previously stored the first portions of the RM images.

17 . A system, comprising:
an image sensor configured to obtain an input image; and an image signal processor connected to the image sensor, the image signal processor including:
a response map (RM) generator circuit configured to generate RM images based on an image pyramid, the RM images comprising a first subset of RM images of a first octave and a second subset of RM images of a second octave;
an intra octave keypoint candidate selector circuit configured to determine keypoint candidates of keypoints of the input image based on a comparison between at least two RM images in the first subset of RM images of the first octave; and
an inter octave keypoint candidate selector circuit configured to determine keypoints from the keypoint candidates based on a comparison between the first subset of RM images and the second subset of RM images.

18 . The system of claim 17 , wherein the image signal processor further comprises:
a pyramid image buffer circuit configured to generate the image pyramid from the input image.

19 . The system of claim 18 , wherein the pyramid image buffer circuit has a storage size smaller than a data size of the image pyramid.

20 . The system of claim 17 , further comprising an RM buffer circuit configured to:
provide first portions of the RM images stored in memory locations of the RM buffer circuit to the inter octave keypoint candidate selector circuit for keypoint determination; remove the first portions of the RM images from the memory locations responsive to the first portions being no longer needed for the keypoint determination by the inter octave keypoint candidate selector circuit; and store second portions of the RM images that are needed for the keypoint determination by the inter octave keypoint candidate selector circuit in the memory locations that previously stored the first portions of the RM images.

Assignees

• Apple Inc

Inventors

• Pope David R
• Fishel Liran
• Metuki Assaf
• Wang Muge

Classifications

• G06T2207/20016

  Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform · CPC title

• G06T7/251

  involving models · CPC title

• G06T1/60

  Memory management · CPC title

• H04N1/2141 Primary

  in a multi-frame buffer · CPC title

• G06T3/40

  Scaling of whole images or parts thereof, e.g. expanding or contracting · CPC title