Style classification for authentic content search

Patent metadata

Field	Value
Publication number	US-11017019-B1
Application number	US-201615236357-A
Country	US
Kind code	B1
Filing date	Aug 12, 2016
Priority date	Aug 14, 2015
Publication date	May 25, 2021
Grant date	May 25, 2021

Abstract

Various aspects of the subject technology relate to systems, methods, and machine-readable media for authentic content search using style classifications. A system may be a search engine that uses a set of style classifiers to detect one or more styles associated with an image and a logistic regression model to determine a level of authenticity for the image based on the associated styles. Training images are fed to train a series of neural networks that output a set of style classifiers. An image is processed through the style classifiers to determine respective probabilities for each style classification. The results from the set of style classifiers are then input to the logistic regression model to determine an authenticity score for the image. For example, the authenticity score shows how authentic is an image (e.g., a score of 1.0 refers to 100% authenticity, whereas a score of 0.0 represents a non-authentic image).

First claim

What is claimed is:

1. A computer-implemented method, comprising:
providing a plurality of sets of training images to a computer-operated convolutional neural network, wherein the computer-operated convolutional neural network processes the plurality of sets of training images to learn to identify features relating to at least one of a plurality of style classes, wherein each of the plurality of sets of training images is associated with one style class of the plurality of style classes;
generating feature vectors for each training image in the plurality of sets of training images using the computer-operated convolutional neural network;
generating processed pixel data including the feature vectors from the plurality of sets of training images;
mapping the processed pixel data to a style classifier of a set of style classifiers;
determining a style class probability using the style classifier for a style class of a set of style classes, the style class probability indicating a likelihood that a subject image corresponds to the style class;
receiving user input identifying a search query from a client device, the search query indicating a request for authentic content;
identifying one or more image identifiers corresponding to the search query;
determining images from a collection of images that contain authentic content based on an authenticity score associated with each of the images, the authenticity score indicating for an associated image from the images a probability that the associated image contains content that visually appears to be authentic, and the authenticity score corresponding to how closely the associated image resembles a naturally-occurring situation,
wherein the authenticity score for the associated image is based at least in part on a set of authenticity probabilities associated with style classes detected for the associated image, and
wherein the style classes that are detected for the associated image and the set of authenticity probabilities associated with the style classes for the associated image are determined via machine learning using the computer-operated convolutional neural network;
wherein training the computer-operated convolutional neural network comprises mapping, by the computer-operated convolutional neural network, feature descriptor vectors extracted from training images to a corresponding weighted classifier model for each style class, the weighted classifier model comprising a set of weighted data points where each weighted data point corresponds to a different style, each weighted data point representing a likelihood that the corresponding style is an authentic style;
providing a prioritized listing of the images determined to contain authentic content to the client device; and
tagging each of the images with metadata identifying the authenticity score for the image, the metadata comprising one or more rows of data including an image identifier, an image Uniform Resource Locator (URL), and a style class.

2. The computer-implemented method of claim 1 , wherein the prioritized listing of the images comprises a plurality of image groupings, wherein the plurality of image groupings comprises images grouped by their relative levels of authenticity.

3. The computer-implemented method of claim 2 , further comprising:
providing a user interface control to allow a user to select at least one grouping from the plurality of image groupings;
receiving a user selection of at least one grouping of the plurality of image groupings via the user interface control;
modifying the prioritized listing of images based on the received user selection; and
providing the modified listing of images for display on the client device.

4. The computer-implemented method of claim 2 , further comprising:
providing the prioritized listing of images for display on the client device, the prioritized listing of images including the plurality of image groupings;
providing a user interface control to allow a user to exclude at least one grouping from the plurality of image groupings;
receiving a user command to exclude at least one grouping of the plurality of image groupings via the user interface control;
modifying the prioritized listing of images based on the received user command; and
providing the modified listing of images for display on the client device.

5. The computer-implemented method of claim 2 , wherein;
providing a user interface control to allow a user to dial in a relative amount of authenticity within the prioritized listing of images;
receiving a user selection that corresponds to a proportionate amount of authenticity within the prioritized listing of images, via the user interface control;
modifying the prioritized listing of images based on the received user selection; and
providing the modified listing of images for display on the client device.

6. The computer-implemented method of claim 1 , further comprising:
clustering the feature vectors into a plurality of clusters,
wherein at least one of the plurality of clusters is associated with one of the one or more image identifiers.

7. The computer-implemented method of claim 1 , further comprising:
providing an aggregate of style class probabilities to a logistic regression model, the aggregate of style class probabilities including a style class probability for each style class in the set of style classes.

8. The computer-implemented method of claim 7 , further comprising:
combining a plurality of style class probabilities from the set of style classifiers to form the aggregate of style class probabilities,
wherein the authenticity score is determined based on the aggregate of the style class probabilities via the logistic regression model.

9. The computer-implemented method of claim 6 , further comprising:
obtaining historical user interaction data, the historical user interaction data indicating a number of prior images that a user interacted with in prior image search queries over a given period of time;
parsing the historical user interaction data to obtain an authenticity probability for each image in the number of prior images;
providing authenticity probabilities of the last N images from the number of prior images to the computer-operated convolutional neural network, where N is a positive integer; and
generating a search probability from the provided authenticity probabilities of the last N images using the computer-operated convolutional neural network, the search probability indicating a probability that a user is searching for authentic content,
wherein the prioritized listing of images includes a number of images that contain authentic content at a proportion that corresponds to the search probability.

10. The computer-implemented method of claim 6 , wherein the set of training images comprises a plurality of annotated training images, the plurality of annotated training images including training images annotated to indicate a level of authenticity for the training image.

11. The computer-implemented method of claim 6 , further comprising:
determining a subset of images in the collection of images that respectively include an authenticity score within a predetermined range of authenticity scores;
providing the subset of images to a third-party crowd-sourcing service for annotation of the subset of images;
receiving the subset of images with annotations set by users of the third-party crowd-sourcing service, the annotations respectively indicating a level of authenticity for an image in the subset of images; and
providing the subset of images with the annotations as part of the set of training images to the computer-operated convolutional neural network

Assignees

• Shutterstock Inc

Inventors

• Hohwald Heath
• Chester David

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N3/08

  Learning methods · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06F16/51 Primary

  Indexing; Data structures therefor; Storage structures · CPC title