Automatic recognition of entities in media-captured events

Patent metadata

Field	Value
Publication number	US-9704020-B2
Application number	US-201514741015-A
Country	US
Kind code	B2
Filing date	Jun 16, 2015
Priority date	Jun 16, 2015
Publication date	Jul 11, 2017
Grant date	Jul 11, 2017

Abstract

Architecture that enables the identification of entities such as people and content in live broadcasts (e.g., streaming content (e.g., video) of live events) and non-live presentations (e.g., movies), in realtime, using recognition processes. This can be accomplished by extracting live data related to a live event. With respect to people entities, filtering can be performed to identify the named (people) entities from the extracted live data, and trending topics discovered as relate to the named entities, as associated with the live event. Multiple images of the named entities that capture the named entities under different conditions are captured for the named entities. The images are then processed to extract and learn facial features (train one or more models), and facial recognition is then performed on faces in the video using the trained model(s).

First claim

What is claimed is:

1. A system, comprising a hardware processor and a memory device, the memory device configured to store computer-executable instructions that when executed by the hardware processor cause the hardware processor to implement one or more computer-executable components, comprising:
an access component configured to access information related to at least one of an event or people participating in the event, the people and event captured in a video, the information accessed as part of presenting the video, the access component further configured to access the information according to a predetermined time window, the time window determined based on at least one of kind of the event or duration of the event;
a collection component configured to collect training data from the information for identification of at least one or more persons in the video;
a training component configured to train a model using the training data, the training data collected and model trained as part of presenting the video;
a recognition component configured to perform facial recognition processing of the people in the video to identify the one or more persons, the facial recognition processing performed using the model; and
a content component configured to obtain content relevant to the one or more persons identified, the content for presentation with the video and while the video is being viewed.

2. The system of claim 1 , wherein the event is a live event captured via the video and the video is streamed for viewing on first device while the event is occurring, the relevant content presented on the first device while the video is being viewed on the first device.

3. The system of claim 1 , wherein the event is a live event captured via the video and the video is streamed for viewing on a first device while the event is occurring, the relevant content presented on one or more other devices while the video is being viewed on the first device.

4. The system of claim 1 , wherein the training component is configured to train the model using training data that comprises a set of tagged images of the one or more persons to learn facial features of the one or more persons.

5. The system of claim 1 , further comprising an entity recognition component configured to process social media messages, and output named entities, entity popularity parameters, and entity type information, where the entities include an event and people associated with the event.

6. The system of claim 1 , further comprising a trend component configured to access trend information that indicates people that are trending in correlation to the event, and from which context of the event is inferred.

7. A computer-implemented method, comprising computer-executable instructions that when executed by a hardware processor cause the hardware processor to perform acts of:
accessing information related to people in a video and related to an event in the video, the information accessed as part of presenting the video, the information accessed according to a predetermined time window, the time window determined based on at least one of kind of the event or duration of the event;
collecting training data from the information for identification of specific people in the video;
training a model using the training data, the training occurs as part of viewing the video;
performing recognition processing of the people in the video to identify one or more persons while the video is being presented, the recognition processing performed using the trained model;
performing a search to retrieve content relevant to the one or more persons identified; and
presenting the content relevant to the one or more persons identified.

8. The method of claim 7 , further comprising presenting the relevant content while the video is being viewed.

9. The method of claim 7 , further comprising presenting the relevant content on a device from which the video is being viewed or on a device different than the device from which the video is being viewed.

10. The method of claim 7 , further comprising accessing the information, collecting the training data, performing the recognition processing, and presenting the content while the video is being viewed.

11. The method of claim 7 , further comprising accessing the information, collecting the training data, performing the recognition processing, and presenting the content while the video captures a live event.

12. The method of claim 7 , further comprising accessing the information from social media networks and according to a predetermined time window relative to the event.

13. The method of claim 7 , further comprising performing name recognition processing to identify names in the information as obtained from social media.

14. The method of claim 7 , further comprising accessing trending information as to named people to infer context of the event.

15. The method of claim 7 , further comprising presenting relevant content in association with recognized faces of the people.

16. A computer-implemented method, comprising computer-executable instructions that when executed by a hardware processor cause the hardware processor to perform acts of:
accessing information about a live event and about people associated with the live event, the live event and the people captured in video as part of a live broadcast of the live event, the information accessed according to a predetermined time window based on at least one of kind of the event or duration of the event;
performing named entity recognition on the information to obtain a list of named people and associated attributes related to the live event;
identifying trends in social media messages associated with the live event to compute a ranked list of the named people;
obtaining and processing images of the named people in the ranked list to output training data that comprises features of faces of people associated with the live event;
training a model using the training data, the training occurs as part of viewing the live event; and
performing face recognition processing of the people in the video based on the training model, to identify the named people and return content relevant to the named people for presentation in association with viewing the video.

17. The method of claim 16 , further comprising accessing of the information, performing the named entity recognition, identifying the trends, obtaining and processing the images, training of the model, performing the face recognition processing, and presenting the relevant content while the live event is occurring.

18. The method of claim 16 , further comprising presenting the relevant content on a device from which the video is being viewed or on a device different than the device from which the video is being viewed.

19. The method of claim 16 , further comprising presenting the relevant content in association with a named person when the named person is shown in the video.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Koul Anirudh
• Tremblay Serge-Eric

Classifications

• H04N21/2187 Primary

  Live feed · CPC title

• G06V40/172

  Classification, e.g. identification · CPC title

• H04N21/23424

  involving splicing one content stream with another content stream, e.g. for inserting or substituting an advertisement · CPC title

• G06F18/2411

  based on the proximity to a decision surface, e.g. support vector machines · CPC title

• G06V20/62

  Text, e.g. of license plates, overlay texts or captions on TV images · CPC title