Backward compatible and backfill-free image search system

Patent metadata

Field	Value
Publication number	US-11216697-B1
Application number	US-202016815787-A
Country	US
Kind code	B1
Filing date	Mar 11, 2020
Priority date	Mar 11, 2020
Publication date	Jan 4, 2022
Grant date	Jan 4, 2022

Abstract

Techniques for building a backward compatible and backfill-free image search system are described. According to some embodiments, a backwards compatible training system trains a new embedding model to be backward compatible with the face embeddings (e.g., floating-point vectors) generated by a previous embedding model. In one embodiment, backwards compatible training uses a classifier of the previous embedding model as a form of constraint in the training of the new embedding model.

First claim

What is claimed is:

1. A computer-implemented method comprising:
generating a first set of face embeddings from a dataset of images using a first embedding model;
training a second, different embedding model to generate a second set of face embeddings generated by the second, different embedding model that are compatible with the first set of face embeddings generated from the dataset of images by the first embedding model;
generating a third set of face embeddings from a query image from a user using the second, different embedding model;
performing a search of the third set of face embeddings from the query image in the first set of face embeddings of the dataset of images using a classifier model to generate a result; and
returning the result to the user.

2. The computer-implemented method of claim 1 , wherein the training comprises using a classifier model of the first embedding model as a training constraint of the training of the second, different embedding model.

3. The computer-implemented method of claim 1 , wherein the performing the search occurs before generating a set of face embeddings for the dataset of images using the second, different embedding model.

4. A computer-implemented method comprising:
generating a first set of features from a dataset of images using a first embedding model;
training a second, different embedding model to generate a second set of features generated by the second, different embedding model that are compatible with the first set of features generated from the dataset of images by the first embedding model;
generating a third set of features from a query image from a user using the second, different embedding model;
performing a search of the third set of features from the query image in the first set of features of the dataset of images using a classifier model to generate a result; and
returning the result to the user.

5. The computer-implemented method of claim 4 , wherein the training comprises using a classifier model of the first embedding model as a training constraint of the training of the second, different embedding model.

6. The computer-implemented method of claim 4 , further comprising generating a fourth set of features from the dataset of images using the second, different embedding model, wherein the performing the search comprises performing the search of the third set of features from the query image in the first set of features of the dataset of images and in the fourth set of features of the dataset of images to generate the result.

7. The computer-implemented method of claim 4 , wherein the performing the search occurs before generating a set of features for the dataset of images using the second, different embedding model.

8. The computer-implemented method of claim 4 , further comprising training the first embedding model on a training dataset, and the second, different embedding model on the training dataset.

9. The computer-implemented method of claim 4 , further comprising training the first embedding model on a first training dataset, and the second, different embedding model on a second, different training dataset.

10. The computer-implemented method of claim 4 , wherein an embedding dimension of the first embedding model is different than an embedding dimension of the second, different embedding model.

11. The computer-implemented method of claim 4 , wherein an architecture of the first embedding model is different than an architecture of the second, different embedding model.

12. The computer-implemented method of claim 4 , wherein the result comprises an indication of a most probable object depicted in the query image.

13. The computer-implemented method of claim 9 , wherein the training of the second, different embedding model on the second, different training dataset comprises generating classifier weights for one or more classes of images in the second, different training dataset that are not in a set of classes of images in the first training dataset.

14. The computer-implemented method of claim 9 , wherein the training of the second, different embedding model on the second, different training dataset comprises not generating classifier weights for one or more classes of images in the second, different training dataset that are not in a set of classes of images in the first training dataset.

15. A system comprising:
a first one or more electronic devices to implement a storage service in a multi-tenant provider network to store a dataset of images; and
a second one or more electronic devices to implement a search service in the multi-tenant provider network, the search service including instructions that upon execution cause the search service to perform a method comprising:
generating a first set of features from the dataset of images using a first embedding model,
training a second, different embedding model to generate a second set of features generated by the second, different embedding model that are compatible with the first set of features generated from the dataset of images by the first embedding model,
generating a third set of features from a query image from a user using the second, different embedding model,
performing a search of the third set of features from the query image in the first set of features of the dataset of images using a classifier model to generate a result, and
returning the result to the user.

16. The system of claim 15 , wherein the training comprises using a classifier model of the first embedding model as a training constraint of the training of the second, different embedding model.

17. The system of claim 15 , wherein the performing the search occurs before generating a set of features for the dataset of images using the second, different embedding model.

18. The system of claim 15 , wherein an embedding dimension of the first embedding model is different than an embedding dimension of the second, different embedding model.

19. The system of claim 15 , wherein an architecture of the first embedding model is different than an architecture of the second, different embedding model.

20. The system of claim 15 , wherein the result comprises an indication of a most probable person depicted in the query image.

Assignees

• Amazon Tech Inc

Inventors

• Shen Yantao
• Xiong Yuanjun
• Deng Siqi
• Xia Wei
• Yang Shuo
• Xing Yifan
• Li Wei
• Soatto Stefano

Classifications

• G06F16/55 Primary

  Clustering; Classification · CPC title

• G06V10/774

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V10/82

  using neural networks · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06N3/045

  Combinations of networks · CPC title