Robust content fingerprinting for image attribution

Patent metadata

Field	Value
Publication number	US-12147495-B2
Application number	US-202117142030-A
Country	US
Kind code	B2
Filing date	Jan 5, 2021
Priority date	Jan 5, 2021
Publication date	Nov 19, 2024
Grant date	Nov 19, 2024

Abstract

A visual search system facilitates retrieval of provenance information using a machine learning model to generate content fingerprints that are invariant to benign transformations while being sensitive to manipulations. The machine learning model is trained on a training image dataset that includes original images, benign transformed variants of the original images, and manipulated variants of the original images. A loss function is used to train the machine learning model to minimize distances in an embedding space between benign transformed variants and their corresponding original images and increase distances between the manipulated variants and their corresponding original images.

First claim

What is claimed is:

1. One or more computer storage media storing computer-useable instructions that, when used by a computing device, cause the computing device to perform operations, the operations comprising:
accessing a training image dataset comprising a plurality of original images, one or more positive images for each original image, and one or more negative images for each original image, the one or more positive images for each original image consisting of one or more benign transformed images that each comprises one or more benign transformations to a corresponding original image reformatting the corresponding original image and no manipulation changing a salient detail impacting a meaning of the corresponding original image, the one or more negative images for each original image comprising one or more manipulated images that each comprises one or more manipulations to a corresponding original image changing a salient detail impacting a meaning of the corresponding original image; and
training a machine learning model to learn an embedding using the images from the training image dataset and a loss function that minimizes a distance in an embedding space between a representation for each original image and a representation for the one or more benign transformed images corresponding to each original image while increasing a distance in the embedding space between the representation for each original image and a representation for the one or more manipulated images corresponding to each original image.

2. The one or more computer storage media of claim 1 , the operations further comprising:
generating a scene graph representation of each image from the training image dataset; and
wherein the machine learning model is trained using the scene graph representation of each image from the training image dataset.

3. The one or more computer storage media of claim 2 , wherein generating the scene graph representation for a first image from the training image dataset comprises:
identifying a plurality of objects in the first image;
determining one or more object features for each object in the first image and one or more relationship features for a relationship between each pair of objects; and
generating the scene graph representation for the first image based on the one or more object features and the one or more relationship features.

4. The one or more computer storage media of claim 3 , wherein the one or more object features for each object in the first image comprise a visual appearance feature, a shape feature, and a geometric feature.

5. The one or more computer storage media of claim 1 , the operations further comprising:
generating a content fingerprint for each of a plurality of source images using the machine learning model; and
storing each content fingerprint in association with a corresponding source image in an image repository.

6. The one or more computer storage media of claim 5 , wherein the image repository stores provenance information in association with each source image.

7. The one or more computer storage media of claim 5 , the operations further comprising:
receiving a search request comprising a query image;
generating a content fingerprint of the query image using the machine learning model; and
searching the image repository by determining a distance in the embedding space between the content fingerprint for the query image and the content fingerprint for one or more source images.

8. A computerized method comprising:
receiving, by a user interface module, a search request comprising a query image;
generating, via a machine learning model, a content fingerprint for the query image, the machine learning model trained using a training image dataset and a loss function, the training image dataset comprising a plurality of original images, one or more positive images for each original image, and one or more negative images for each original image, the one or more positive images for each original image consisting of one or more benign transformed images that each comprises one or more image benign transformations to a corresponding original image reformatting the corresponding original image and no manipulation changing a salient detail impacting a meaning of the corresponding original image, the one or more negative images comprising one or more manipulated images that each comprises one or more manipulations to a corresponding original image changing a salient detail impacting a meaning of the corresponding original image, the loss function minimizing a distance in an embedding space between a representation for each original image and a representation for the one or more benign transformed images corresponding to each original image while increasing a distance in the embedding space between the representation for each original image and a representation for the one or more manipulated images corresponding to each original image;
searching, by a search module, an image repository using the content fingerprint, the image repository comprising a plurality of source images associated with a content fingerprint generated using the machine learning model; and
providing, by the user interface module, a response to the search query based on the searching.

9. The computerized method of claim 8 , wherein generating the content fingerprint for the query image comprises:
identifying a plurality of objects in the query image;
determining one or more object features for each object in the query image and one or more relationship features for a relationship between each pair of objects;
generating a scene graph representation for the query image based on the one or more object features and the one or more relationship features; and
feeding the scene graph representation to the machine learning model.

10. The computerized method of claim 8 , wherein the searching identifies a matching source image from the image repository, and the response identifies the matching source image.

11. The computerized method of claim 10 , wherein the image repository stores provenance information in association with each source image, and the response includes the provenance information for the matching source image.

12. The computerized method of claim 8 , wherein the searching does not identify a matching source image from the image repository, and the response includes an indication that a matching source image was not found.

13. The computerized method of claim 8 , wherein the searching does not identify a matching source image from the image repository, and the method further comprises:
identifying a first source image from the image repository by performing a second search using a second visual search approach; and
wherein the response identifies the first source image.

14. The computerized method of claim 13 , wherein the response includes an indication that the query image may be a manipulated version of the first source image.

15. The computerized method of claim 13 , wherein the response includes a user interface element for comparing visual aspects of the query image and the first source image.

16. A computer system comprising:
a processor; and
a computer storage medium storing computer-useable instructions that, when used by the processor, causes the computer system to perform operations comprising:
training, by a training module, a machine learning model using a training image dataset and a loss function, the training image dataset including a plurality of original images, one or more positive images for each original image, and one or more negative images for each

Assignees

• Adobe Inc

Inventors

• Swaminathan Viswanathan
• Collomosse John Philip
• Nguyen Eric

Classifications

• G06N3/0495

  Quantised networks; Sparse networks; Compressed networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06V20/00

  Scenes; Scene-specific elements (control of digital cameras H04N23/60) · CPC title

• G06V10/40

  Extraction of image or video features · CPC title