Generating numeric embeddings of images

Patent metadata

Field	Value
Publication number	US-9836641-B2
Application number	US-201514972670-A
Country	US
Kind code	B2
Filing date	Dec 17, 2015
Priority date	Dec 17, 2014
Publication date	Dec 5, 2017
Grant date	Dec 5, 2017

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating numeric embeddings of images. One of the methods includes obtaining training images; generating a plurality of triplets of training images; and training a neural network on each of the triplets to determine trained values of a plurality of parameters of the neural network, wherein training the neural network comprises, for each of the triplets: processing the anchor image in the triplet using the neural network to generate a numeric embedding of the anchor image; processing the positive image in the triplet using the neural network to generate a numeric embedding of the positive image; processing the negative image in the triplet using the neural network to generate a numeric embedding of the negative image; computing a triplet loss; and adjusting the current values of the parameters of the neural network using the triplet loss.

First claim

What is claimed is:

1. A method comprising:
obtaining a plurality of training images, wherein the training images have been classified as images of objects of a particular object type;
generating a plurality of triplets of training images, wherein each of the triplets comprises a respective anchor image, a respective positive image, and a respective negative image, and wherein, for each triplet, the anchor image and the positive image have both been classified as images of the same object of the particular object type and the negative image has been classified as an image of a different object of the particular object type; and
training a neural network on each of the triplets to determine trained values of a plurality of parameters of the neural network, wherein the neural network is configured to receive an input image of an object of the particular object type and to process the input image to generate a numeric embedding of the input image, wherein training the neural network comprises, for each of the triplets:
processing the anchor image in the triplet using the neural network in accordance with current values of the parameters of the neural network to generate a numeric embedding of the anchor image;
processing the positive image in the triplet using the neural network in accordance with the current values of the parameters of the neural network to generate a numeric embedding of the positive image;
processing the negative image in the triplet using the neural network in accordance with the current values of the parameters of the neural network to generate a numeric embedding of the negative image;
computing a triplet loss from the numeric embedding of the anchor image, the positive image, and the negative image; and
adjusting the current values of the parameters of the neural network to minimize the triplet loss such that distances between the numeric embedding of the input image of the object and respective numeric embeddings of other images of the same object is less than distances between the numeric embedding of the input image of the object and numeric embeddings of other images of other objects,

wherein generating the plurality of triplets of training images, includes, for each of the triplets, generating the triplet such that i) the numeric embedding of the positive image in the triplet is farther from the numeric embedding of the anchor image in the triplet than any other numeric embedding of any other image of the same object as in the positive image and the anchor image, and ii) the numeric embedding of the negative image in the triplet is closer to the numeric embedding of the anchor image than any other number embedding of any other image of a different object as in the positive image and the anchor image.

2. The method of claim 1 , wherein the particular object type is faces of people.

3. The method of claim 1 , wherein the neural network is a deep convolutional neural network.

4. The method of claim 1 , wherein the neural network is configured to generate a vector of floating point values for the input image.

5. The method of claim 4 , further comprising:
using the vector of floating point values as the numeric embedding of the input image.

6. The method of claim 4 , further comprising:
normalizing the vector of floating point values to generate a normalized vector; and
using the normalized vector as the numeric embedding of the input image.

7. The method of claim 4 , further comprising:
normalizing the vector of floating point values to generate a normalized vector;
quantizing the normalized vector to generate a quantized vector; and
using the quantized vector as the numeric embedding of the input image.

8. The method of claim 1 , wherein the triplet loss satisfies, for each of the triplets:
L =max(0,∥ f ( x a )− f ( x p )∥ 2 2 −∥f ( x a )− f ( x n )∥ 2 2 +α),

wherein f(x a ) is the numeric embedding of the anchor image in the triplet, f(x p ) is the numeric embedding of the positive image in the triplet, f(x n ) is the numeric embedding of the negative image in the triplet, and α is a predetermined value.

9. The method of claim 1 , further comprising:
receiving a first image and a second image;
processing the first image using the neural network in accordance with the trained values of the parameters of the neural network to determine a numeric embedding of the first image;
processing the second image using the neural network in accordance with the trained values of the parameters of the neural network to determine a numeric embedding of the second image; and
determining whether the first image and the second image are images of the same object from a distance between the numeric embedding of the first image and the numeric embedding of the second image.

10. The method of claim 1 , further comprising:
processing each of a plurality of images using the neural network in accordance with the trained values of the parameters of the neural network to determine a respective numeric embedding of each of the plurality of images;
receiving a new image;
processing the new image using the neural network in accordance with the trained values of the parameters of the neural network to determine a numeric embedding of the new image; and
classifying the new images as being an image of the same object as one or more of the plurality of images from distances between the numeric embedding of the new image and numeric embeddings of images from the plurality of images.

11. The method of claim 1 , further comprising:
processing each of a plurality of images using the neural network in accordance with the trained values of the parameters of the neural network to determine a respective numeric embedding of each of the plurality of images;
clustering the numeric embedding of the plurality of images into a plurality of clusters; and
for each cluster, classifying the images having numeric embeddings that are in the cluster as being images of the same object.

12. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
obtaining a plurality of training images, wherein the training images have been classified as images of objects of a particular object type;
generating a plurality of triplets of training images, wherein each of the triplets comprises a respective anchor image, a respective positive image, and a respective negative image, and wherein, for each triplet, the anchor image and the positive image have both been classified as images of the same object of the particular object type and the negative image has been classified as an image of a different object of the particular object type; and
training a neural network on each of the triplets to determine trained values of a plurality of parameters of the neural network, wherein the neural network is configured to receive an input image of an object of the particular object type and to process the input image to generate a numeric embedding of the input image, wherein training the neural network comprises, for each of the triplets:
processing the anchor image in the triplet using the neural network in accordance with current values of the parameters of the neural network to generate a numeric embedding of the anchor image;
processing the positive image in the triplet using the neural network in accordance with the current values of the parameters of the neural network to generate a numeric embedding of the positive image;
processing the negative image in the triplet using the neural network in accordance wi

Assignees

• Google Inc

Inventors

• Philbin James William
• Schroff Gerhard Florian
• Kalenichenko Dmitry

Classifications

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06V30/19147

  Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V40/172

  Classification, e.g. identification · CPC title

• G06N3/084 Primary

  Backpropagation, e.g. using gradient descent · CPC title