Method and electronic device for recognizing product

Patent metadata

Field	Value
Publication number	US-12430676-B2
Application number	US-202218148629-A
Country	US
Kind code	B2
Filing date	Dec 30, 2022
Priority date	Dec 29, 2021
Publication date	Sep 30, 2025
Grant date	Sep 30, 2025

Abstract

A method and electronic device for recognizing a product are provided. The method includes obtaining first feature information and second feature information from an image related to a product, obtaining fusion feature information based on the first feature information and the second feature information by using a main encoder model that reflects a correlation between feature information of different modalities, matching the fusion feature information against a database of the product, and providing information about the product, based on a result of the matching.

First claim

What is claimed is:

1. A method of performed by an electronic device for recognizing a product, the method comprising:
obtaining first feature information and second feature information from an image related to a product;
obtaining fusion feature information based on the first feature information and the second feature information by using a main encoder model that reflects a correlation between feature information of different modalities;
comparing the fusion feature information against a product database for a match to the fusion feature information in the product database; and
providing information about the product, based on a result of the comparing of the fusion feature information against the product database being that there is the match to the fusion feature information in the product database;
comparing the first feature information and the second feature information against the product database for a match to the first feature information or the second feature information in the product database, based on the result of the comparing of the fusion feature information against the product database being that there is not the match to the fusion feature information in the product database; and
updating the main encoder model in such a manner that there is a match to the non-matching fusion feature information in the product database, based on a result of the comparing of the first feature information and the second feature information against the product database being that there is the match to the first feature information or the second feature information in the product database.

2. The method of claim 1 , wherein the main encoder model:
receives at least one piece of the first feature information and at least one piece of the second feature information as an input;
obtains an attention value of any one piece of feature information in the image by identifying a correlation between the any one piece of feature information and all pieces of input feature information, based on self-attention; and
outputs the fusion feature information by summing up attention values of the all pieces of input feature information.

3. The method of claim 2 , wherein the main encoder model:
identifies the correlation between the any one piece of feature information and the all pieces of input feature information through matrix multiplication between a query vector extracted from the any one piece of feature information and key vectors extracted from the all pieces of input feature information; and
obtains the attention value by calculating a weighted sum in which the identified correlation is reflected.

4. The method of claim 1 ,
wherein the first feature information is image feature information, and
wherein the second feature information is text feature information.

5. The method of claim 4 , wherein the image feature information includes information about at least one shape in an area of the image corresponding to a label of the product.

6. The method of claim 1 , wherein the obtaining of the first feature information and the second feature information comprises:
dividing the image into a first element and a second element;
extracting a first feature from the first element divided from the image and encoding the extracted first feature as the first feature information, by using a first sub-encoder model; and
extracting a second feature from the second element divided from the image and encoding the extracted second feature as the second feature information, by using a second sub-encoder model.

7. The method of claim 1 , further comprising:
registering the product in the product database as a new product, based on the result of the comparing of the first feature information and the second feature information against the product database being that there is not the match to the first feature information or the second feature information in the product database.

8. The method of claim 1 , further comprising:
receiving the main encoder model and the product database from at least one server individually or collectively configured to train the main encoder model and manage the product database.

9. The method of claim 1 , further comprising:
executing a product recognition application, based on an input of a user, and
obtaining, through a camera, the image related to the product.

10. An electronic device for recognizing a product, the electronic device comprising:
a memory storing instructions; and
at least one processor communicatively coupled to the memory,
wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic device to:
obtain first feature information and second feature information from an image related to a product,
obtain fusion feature information based on the first feature information and the second feature information by using a main encoder model that reflects a correlation between feature information of different modalities,
compare the fusion feature information against a product database for a match to the fusion feature information in the product database,
provide information about the product, based on a result of the comparing of the fusion feature information against the product database being that there is the match to the fusion feature information in the product database,
compare the first feature information and the second feature information against the product database for a match to the first feature information or the second feature information in the product database, based on the result of the comparing of the fusion feature information against the product database being that there is not the match to the fusion feature information in the product database, and
update the main encoder model in such a manner that there is a match to the non-matching fusion feature information in the product database, based on a result of the comparing of the first feature information and the second feature information against the product database being that there is the match to the first feature information or the second feature information in the product database.

11. The electronic device of claim 10 , wherein the instructions, when executed by the at least one processor individually or collectively, further cause the electronic device to perform at least one function of the main encoder model to:
receive at least one piece of the first feature information and at least one piece of the second feature information as an input,
obtain an attention value of any one piece of feature information in the image by identifying a correlation between the any one piece of feature information and all pieces of input feature information, based on self-attention, and
output the fusion feature information by summing up attention values of the all pieces of input feature information.

12. The electronic device of claim 11 , wherein the instructions, when executed by the at least one processor individually or collectively, further cause the electronic device to perform a at least one function of the main encoder model to:
identify the correlation between the any one piece of feature information and the all pieces of input feature information through matrix multiplication between a query vector extracted from the any one piece of feature information and key vectors extracted from the all pieces of input feature information, and
obtain the attention value by calculating a weighted sum in which the identified correlation is reflected.

13. The electronic device of claim 10 , wherein the instructions, when executed by the at least one processor individually or collectively, f

Assignees

• Samsung Electronics Co Ltd

Inventors

• Kim Kyungsu
• Kwon Ohjoon
• Kim Younguk
• Choi Hyunsoo
• Kim Yehoon
• Kim Hyunhan
• Kim Hyosang
• Lee Hyungmin

Classifications

• G06V10/806

  of extracted features · CPC title

• G06V30/1444

  Selective acquisition, locating or processing of specific regions, e.g. highlighted text, fiducial marks or predetermined fields · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V30/18

  Extraction of features or characteristics of the image · CPC title

• G06V20/62

  Text, e.g. of license plates, overlay texts or captions on TV images · CPC title