Query matching to media collections in a messaging system

Patent metadata

Field	Value
Publication number	US-11328008-B2
Application number	US-202016844300-A
Country	US
Kind code	B2
Filing date	Apr 9, 2020
Priority date	Feb 13, 2018
Publication date	May 10, 2022
Grant date	May 10, 2022

Abstract

Systems and methods are provided for generating training data from queries and user interactions associated with media collections related to the queries, and training a machine learning model using the generated training data to generate a trained machine learning model. The systems and methods further provide for receiving a prediction request comprising a query for relevant media collections, analyzing the query to determine query features, determining a plurality of media collections for the query, analyzing the plurality of media collections to determine media collection features for each media collection of the plurality of media collections, and generating, using the trained machine learning model, a semantic matching score for each media collection of the plurality of media collections based on matching the query features to the media collection features for each media collection of the plurality of media collections.

First claim

What is claimed is:

1. A method comprising:
generating, by a computing system comprising a hardware processor, a trained machine learning model using training data generated from extracted queries and user interactions associated with media collections related to the queries;
receiving, by the computing system, a query for relevant media collections in a messaging system;
determining, by the computing system, a subset of media collections of a plurality of media collections for the query based on query features for the query;
analyzing, by the computing system, the subset of the plurality of media collections to extract visual concepts from individual images and video contained in each of the subset media of collections of the plurality of media collections; and
generating, by the computing system using the trained machine learning model, a semantic matching score for each media collection of the subset of the plurality of media collections based on matching the query features to the visual concepts for each media collection of the subset of the plurality of media collections.

2. The method of claim 1 , wherein the queries and user interactions are extracted from user data generated by a plurality of computing device associated with a plurality of users in the messaging system.

3. The method of claim 1 , wherein the semantic matching score is used to rank the plurality of media collections.

4. The method of claim 1 , wherein the query is received from a computing device, and the method further comprises:
returning a rank of the plurality of media collections to the computing device, wherein the computing device displays a list of the media collections in the order of the rank.

5. The method of claim 1 , wherein the query features comprise terms in the query, terms associated with terms in the query, and location information associated with a computing device that sent the query.

6. The method of claim 1 , further comprising:
extracting location and caption features associated with individual images and video and wherein the semantic matching score is further generated using the extracted location and caption features.

7. The method of claim 1 , wherein the visual concepts are extracted from the individual images and video using an object recognition model to determine visual concepts associated with media content items within each media collection.

8. A system comprising:
one or more hardware processors; and
a computer-readable medium storing instructions that are executable by the one or more hardware processors to cause the system to perform operations comprising:
generating a trained machine learning model using training data generated from extracted queries and user interactions associated with media collections related to the queries;
receiving a query for relevant media collections in a messaging system;
determining a subset of media collections of a plurality of media collections for the query based on query features for the query;
analyzing the subset of the plurality of media collections to extract visual concepts from individual images and video contained in each of the subset media of collections of the plurality of media collections; and
generating, using the trained machine learning model, a semantic matching score for each media collection of the subset of the plurality of media collections based on matching the query features to the visual concepts for each media collection of the subset of the plurality of media collections.

9. The system of claim 8 , wherein the queries and user interactions are extracted from user data generated by a plurality of computing device associated with a plurality of users in the messaging system.

10. The system of claim 8 , wherein the semantic matching score is used to rank the plurality of media collections.

11. The system of claim 8 , wherein the query is received from a computing device, and the operations further comprise:
returning a rank of the subset of media collections of the plurality of media collections to the computing device, wherein the computing device displays a list of the subset of media collections in the order of the rank.

12. The system of claim 8 , wherein the query features comprise terms in the query, terms associated with terms in the query, and location information associated with a computing device that sent the query.

13. The system of claim 8 , the operations further comprising:
extracting location and caption features associated with individual images and video and wherein the semantic matching score is further generated using the extracted location and caption features.

14. The system of claim 8 , wherein the visual concepts are extracted from the individual images and video using an object recognition model to determine visual concepts associated with media content items within each media collection.

15. A non-transitory computer-readable medium comprising instructions stored thereon that are executable by at least one processor to cause a computing device to perform operations comprising:
generating a trained machine learning model using training data generated from extracted queries and user interactions associated with media collections related to the queries;

receiving a query for relevant media collections in a messaging system;
determining a subset of media collections of a plurality of media collections for the query based on query features for the query;
analyzing the subset of the plurality of media collections to extract visual concepts from individual images and video contained in each of the subset media of collections of the plurality of media collections; and
generating, using the trained machine learning model, a semantic matching score for each media collection of the subset of the plurality of media collections based on matching the query features to the visual concepts for each media collection of the subset of the plurality of media collections.

16. The non-transitory computer-readable medium of claim 15 , wherein the visual concepts are extracted from the individual images and video using an object recognition model to determine visual concepts associated with media content items within each media collection.

17. The method of claim 1 , further comprising:
generating an aggregated vector of visual concepts for each media collection of the subset of media collections.

18. The method of claim 1 , further comprising:
projecting the query features and the aggregated vector of visual concepts of the subset of media collections within a semantic vector space; and
wherein matching the query features to the visual concepts of the subset of media collections is performed my measuring a similarity of the query features and aggregated vector of visual concepts for each media collection within the semantic vector space.

19. The method of claim 1 , wherein the subset of media collections is determined based on at least a partial or fuzzy match of the query and the title of a media collection.

20. The method of claim 1 , wherein the subset of media collections is determined based on at least a location associated with each media collection, a location associated with the individual images or videos contained in each media collections, or a location of an event associated with each media collection.

Assignees

• Snap Inc

Inventors

• He Xinran
• Luo Jie
• Nayak Sushobhan
• Ren Zhou
• Van Gysel Christophe Jacky Henri

Classifications

• G06Q10/40 Primary

  Business processes related to social networking or social networking services · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title