Video Understanding Platform

Patent metadata

Field	Value
Publication number	US-2018189570-A1
Application number	US-201615395511-A
Country	US
Kind code	A1
Filing date	Dec 30, 2016
Priority date	Dec 30, 2016
Publication date	Jul 5, 2018
Grant date	—

Abstract

In one embodiment, a method includes accessing a first feature vector representing a video-content object corresponding to a node in a social graph, wherein the video-content object comprises frames and audio and is associated with text, the first feature vector is based on one or more of the frames; accessing a second feature vector representing the video-content object, wherein the second feature vector is based on at least some of the text; accessing a third feature vector representing the video-content object, wherein the third feature vector is based on one or more portions of the audio; determining a fourth feature vector representing the video-content object, wherein the fourth feature vector is based on a combination of the first, second, and third feature vectors; and determining a context of the video-content object based on the fourth feature vector and social-graph information.

First claim

What is claimed is:

1 . A method comprising:
by one or more computing devices, accessing a first feature vector representing a video-content object corresponding to a node in a social graph of a social-networking system, wherein:
the video-content object comprises frames and audio and is associated with text;
the first feature vector is based on one or more of the frames of the video-content object; and
the social graph comprises a plurality of nodes and edges connecting the nodes;
by one or more computing devices, accessing a second feature vector representing the video-content object, wherein the second feature vector is based on at least some of the text; by one or more computing devices, accessing a third feature vector representing the video-content object, wherein the third feature vector is based on one or more portions of the audio; by one or more computing devices, determining a fourth feature vector representing the video-content object, wherein the fourth feature vector is based on a combination of the first, second, and third feature vectors; and by one or more computing devices, determining a context of the video-content object based on the fourth feature vector and social-graph information based at least in part on one or more nodes or edges connected to the node corresponding to the video-content object.

2 . The method of claim 1 , wherein the text associated with the video-content object comprises one or more of:
a transcript of one or more portions of the audio; metadata associated with the video-content object; or a post by a user of the social-networking system associated with the video-content object.

3 . The method of claim 1 , further comprising:
by one or more computing devices, receiving a request to access the video-content object from a client device of a user of the social-networking system;
by one or more computing devices, generating a recommendation for a second video-content object based on the feature vector of the video-content object and a user profile for the user; and
by one or more computing devices, sending, to the client device, the recommendation.

4 . The method of claim 1 , wherein determining a context of the video-content object comprises one or more of:
recognizing a physical object; identifying a particular physical object; detecting a physical object; tracking a physical object; recognizing a pose; recognizing a face; determining a topic; recognizing a scene; recognizing an activity; or recognizing behavior.

5 . The method of claim 1 , further comprising, by one or more computing devices, removing a second video-content object based on determining that the video-content object and the second video-content object are similar based on the feature vector for the video-content object and a feature vector for the second video-content object, wherein determining the context of the video-content object comprises determining that the video-content object is inappropriate.

6 . The method of claim 1 , further comprising:
by one or more computing devices, receiving, a query associated with the video-content object from a client device of a user of the social-networking system; by one or more computing devices, identifying one or more objects matching the query; by one or more computing devices, for each identified object, accessing a feature vector representing the identified object; by one or more computing devices, ranking each identified object based on a similarity metric between the feature vector representing the video-content object and the feature vector representing the identified object; and by one or more computing devices, sending, to the client system in response to the query, one or more search results corresponding to one or more of the identified objects, respectively, each identified object corresponding to a search result having a rank greater than a threshold rank.

7 . The method of claim 1 , wherein the text associated with the first video-content object comprises text output of an audio-recognition module.

8 . One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
access a first feature vector representing a video-content object corresponding to a node in a social graph of a social-networking system, wherein:
the video-content object comprises frames and audio and is associated with text;
the first feature vector is based on one or more of the frames of the video-content object; and
the social graph comprises a plurality of nodes and edges connecting the nodes;
access a second feature vector representing the video-content object, wherein the second feature vector is based on at least some of the text; access a third feature vector representing the video-content object, wherein the third feature vector is based on one or more portions of the audio; determine a fourth feature vector representing the video-content object, wherein the fourth feature vector is based on a combination of the first, second, and third feature vectors; and determine a context of the video-content object based on the fourth feature vector and social-graph information based at least in part on one or more nodes or edges connected to the node corresponding to the video-content object.

9 . The media of claim 8 , wherein the text associated with the video-content object comprises one or more of:
a transcript of one or more portions of the audio; metadata associated with the video-content object; or a post by a user of the social-networking system associated with the video-content object.

10 . The media of claim 8 , wherein the software is further operable when executed to:
receive a request to access the video-content object from a client device of a user of the social-networking system; generate a recommendation for a second video-content object based on the feature vector of the video-content object and a user profile for the user; and send, to the client device, the recommendation.

11 . The media of claim 8 , wherein determining a context of the video-content object comprises one or more of:
recognizing a physical object; identifying a particular physical object; detecting a physical object; tracking a physical object; recognizing a pose; recognizing a face; determining a topic; recognizing a scene; recognizing an activity; or recognizing behavior.

12 . The media of claim 8 , wherein the software is further operable when executed to remove a second video-content object based on determining that the video-content object and the second video-content object are similar based on the feature vector for the video-content object and a feature vector for the second video-content object, wherein determining the context of the video-content object comprises determining that the video-content object is inappropriate.

13 . The media of claim 8 , wherein the software is further operable when executed to:
receive, a query associated with the video-content object from a client device of a user of the social-networking system; identify one or more objects matching the query; for each identified object, access a feature vector representing the identified object; rank each identified object based on a similarity metric between the feature vector representing the video-content object and the feature vector representing the identified object; and send, to the client system in response to the query, one or more search results corresponding to one or more of the identified objects, respectively, each identified object corresponding to a search result having a rank greater than

Assignees

• Facebook Inc

Inventors

• Paluri Balmanohar
• Dumoulin Benoit F
• Deng Merlyn
• Philip Reena
• Garcia Dario Garcia

Classifications

• G06V10/771

  Feature selection, e.g. selecting representative features from a multi-dimensional feature space · CPC title

• G06V10/761

  Proximity, similarity or dissimilarity measures · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V20/46

  Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames · CPC title

• H04N21/44008 Primary

  involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream (arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59) · CPC title