Dense feature scale detection for image matching

Patent metadata

Field	Value
Publication number	US-2023419512-A1
Application number	US-202318367034-A
Country	US
Kind code	A1
Filing date	Sep 12, 2023
Priority date	Sep 23, 2016
Publication date	Dec 28, 2023
Grant date	—

Abstract

Dense feature scale detection can be implemented using multiple convolutional neural networks trained on scale data to more accurately and efficiently match pixels between images. An input image can be used to generate multiple scaled images. The multiple scaled images are input into a feature net, which outputs feature data for the multiple scaled images. An attention net is used to generate an attention map from the input image. The attention map assigns emphasis as a soft distribution to different scales based on texture analysis. The feature data and the attention data can be combined through a multiplication process and then summed to generate dense features for comparison.

First claim

What is claimed is:

1 . A method comprising:
identifying a source image having a source dense feature; identifying a target image having a target dense feature; training, using one or more processors of a machine, one or more convolutional neural networks by maximizing a product of the source dense feature and the target dense feature; generating dense features by combining image features that are generated with the one or more convolutional neural networks with attention values of an attention map that is generated with the one or more convolutional neural networks; and identifying, using the dense features, a location of an object depicted in a sequence of images.

2 . The method of claim 1 further comprising:
generating the image features for a plurality of scaled images of an image using the one or more convolutional neural networks; and
generating the attention map based on texture data of pixels of the image using the one or more convolutional neural networks.

3 . The method of claim 1 , further comprising:
identifying an image; generating a plurality of scaled images from the image; and generating the image features for the plurality of scaled images of the image using the one or more convolutional neural networks.

4 . The method of claim 1 , further comprising:
generating the attention values of the attention map, the attention values being one or more numerical values that modify values of the dense features based at least in part on a scale of a plurality of scaled images of an image.

5 . The method of claim 4 , wherein the attention values are a range of numerical values in a distribution, and wherein the image features are combined using a multiplication operation.

6 . The method of claim 4 , wherein the plurality of scaled images comprises a first scaled image and a second scaled image, wherein the first scaled image is used to generate a first set of attention values and a first image feature dataset, wherein the second scaled image is used to generate a second set of attention values and a second image feature dataset.

7 . The method of claim 6 , wherein the first set of attention values and the first image feature dataset are multiplied together to produce a first multiplication output,
wherein the second set of attention values and the second image feature dataset are multiplied together to produce a second multiplication output, wherein the method further comprises: summing the first multiplication output and the second multiplication output to generate a dense feature dataset, wherein the dense feature dataset comprises a plurality of vectors for a plurality of pixels of the image.

8 . The method of claim 1 , further comprising:
generating a sequence of modified images from the sequence of images using the location of the object in the sequence of images.

9 . The method of claim 8 , further comprising:
storing, in a memory of the machine, the location of the object within each image of the sequence of images.

10 . The method of claim 8 , further comprising:
publishing the sequence of modified images on a social network site as an electronic message.

11 . A system comprising:
one or more processors of a machine; and a memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising: identifying a source image having a source dense feature; identifying a target image having a target dense feature; training, using one or more processors of a machine, one or more convolutional neural networks by maximizing a product of the source dense feature and the target dense feature; generating dense features by combining image features that are generated with the one or more convolutional neural networks with attention values of an attention map that is generated with the one or more convolutional neural networks; and identifying, using the dense features, a location of an object depicted in a sequence of images.

12 . The system of claim 11 , wherein the operations further comprise:
generating the image features for a plurality of scaled images of an image using the one or more convolutional neural networks; and generating the attention map based on texture data of pixels of the image using the one or more convolutional neural networks.

13 . The system of claim 11 , wherein the operations further comprise:
identifying an image; generating a plurality of scaled images from the image; and generating the image features for the plurality of scaled images of the image using the one or more convolutional neural networks.

14 . The system of claim 11 , wherein the operations further comprise:
generating the attention values of the attention map, the attention values being one or more numerical values that modify values of the dense features based at least in part on a scale of a plurality of scaled images of an image.

15 . The system of claim 14 , wherein the attention values are a range of numerical values in a distribution, and wherein the image features are combined using a multiplication operation.

16 . The system of claim 14 , wherein the plurality of scaled images comprises a first scaled image and a second scaled image, wherein the first scaled image is used to generate a first set of attention values and a first image feature dataset, wherein the second scaled image is used to generate a second set of attention values and a second image feature dataset.

17 . The system of claim 16 , wherein the first set of attention values and the first image feature dataset are multiplied together to produce a first multiplication output,
wherein the second set of attention values and the second image feature dataset are multiplied together to produce a second multiplication output, wherein the operations further comprise: summing the first multiplication output and the second multiplication output to generate a dense feature dataset, wherein the dense feature dataset comprises a plurality of vectors for a plurality of pixels of the image.

18 . The system of claim 11 , wherein the operations further comprise:
generating a sequence of modified images from the sequence of images using the location of the object in the sequence of images.

19 . The system of claim 18 , wherein the operations further comprise:
storing, in a memory of the machine, the location of the object within each image of the sequence of images; and publishing the sequence of modified images on a social network site as an electronic message.

20 . A non-transitory machine-readable storage device embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:
identifying a source image having a source dense feature; identifying a target image having a target dense feature; training, using one or more processors of a machine, one or more convolutional neural networks by maximizing a product of the source dense feature and the target dense feature; generating dense features by combining image features that are generated with the one or more convolutional neural networks with attention values of an attention map that is generated with the one or more convolutional neural networks; and identifying, using the dense features, a location of an object depicted in a sequence of images.

Assignees

• Snap Inc

Inventors

• Wang Shenlong
• Luo Linjie
• Zhang Ning
• Li Jia

Classifications

• G06T7/248 Primary

  involving reference images or patches · CPC title

• G06T7/33

  using feature-based methods · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title