Real-time annotation and enrichment of captured video

Patent metadata

Field	Value
Publication number	US-9652444-B2
Application number	US-201414547941-A
Country	US
Kind code	B2
Filing date	Nov 19, 2014
Priority date	May 28, 2010
Publication date	May 16, 2017
Grant date	May 16, 2017

Abstract

An annotation suggestion platform may comprise a client and a server, where the client captures a media object and sends the captured object to the server, and the server provides a list of suggested annotations for a user to associate with the captured media object. The user may then select which of the suggested metadata is to be associated or stored with the captured media. In this way, a user may more easily associate metadata with a media object, facilitating the media object's search and retrieval. The server may also provide web page links related to the captured media object. Suggested metadata may be viewed, selected, and associated through a user interface. The user interface may additionally provide space for web pages to be navigated and viewed.

First claim

We claim:

1. A system comprising:
a processor;
a display; and
a computer readable memory containing computer executable instructions, the computer-executable instructions configured to be executed by the processor to perform operations including:
transmitting a captured media object to a remote server;
receiving from the remote server a plurality of suggested descriptions for the captured media object and a first plurality of suggested links to web pages;
displaying the plurality of suggested descriptions for the captured media object in a user metadata selection form, the user metadata selection form including a toggle to allow toggling between the plurality of suggested descriptions and the first plurality of suggested links to web pages;
receiving a first input to select at least one suggested description of the plurality of suggested descriptions from the user metadata selection form, the first input indicating that the at least one suggested description is relevant to the captured media object;
receiving a second input to de-select at least one suggested description of the plurality of suggested descriptions from the user metadata selection form, the second input indicating that the at least one suggested description is irrelevant to the captured media object;
transmitting the first input to the remote server;
transmitting the second input to the remote server;
receiving, from the remote server, a second plurality of links to web pages that are determined to be relevant to the captured media object, the second plurality of links being determined based at least in part on the captured media object, the first input, and the second input; and
displaying the second plurality of links.

2. The system of claim 1 , wherein the system further comprises one or more input controls, and the computer executable instructions are further configured for:
displaying a user preference form on a display;
receiving user preference information in response to the user preference form using at least one input control of the one or more input controls; and
transmitting over a cellular subsystem the user preference information to the remote server.

3. The system of claim 1 , wherein receiving the first input to select the at least one suggested description of the plurality of suggested descriptions from the user metadata selection form comprises:
receiving user metadata selection input in response to the user metadata selection form; and
transmitting over a cellular subsystem the user metadata selection input.

4. The system of claim 3 , wherein the second plurality of links to web pages are further determined, at least in part, on the user metadata selection input.

5. The system of claim 3 , wherein the computer executable instructions are further configured for:
identifying selected ones of the plurality of suggested descriptions from the user metadata selection input; and
storing locally on the system the selected ones of the plurality of suggested descriptions and storing associations of the selected ones of the plurality of suggested descriptions with the captured media object.

6. The system of claim 1 , wherein the computer executable instructions are further configured for displaying thumbnails of the web pages corresponding to the second plurality of links on the display.

7. The system of claim 6 , wherein the computer executable instructions are further configured for:
receiving a user web page selection using at least one input control;
transmitting over the cellular subsystem the user web page selection;
receiving web page content, based on the user web page selection; and
displaying the web page content on the display.

8. The system of claim 1 , wherein the plurality of suggested descriptions of the captured media object have a confidence score that satisfied a predetermined confidence criterion.

9. A method comprising:
transmitting, by a computing device, a captured media object to a remote server; receiving, by the computing device and from the remote server, a plurality of suggested descriptions for the captured media object and a first plurality of suggested links to web pages;
receiving, by the computing device, a first input to select at least one suggested description of the plurality of suggested descriptions from the user metadata selection form, the first input indicating that the at least one suggested description is relevant to the captured media object, the user metadata selection form including a toggle to allow toggling between the plurality of suggested descriptions and the first plurality of suggested links to web pages;
receiving, by the computing device, a second input to de-select at least one suggested description of the plurality of suggested descriptions from the user metadata selection form, the second input indicating that the at least one suggested description is irrelevant to the captured media object;
transmitting, by the computing device, the first input to the remote server;
transmitting, by the computing device, the second input to the remote server;
receiving, by the computing device, a second plurality of links to web pages that are determined to be relevant to the captured media object from the remote server, the second plurality of links to the web pages being determined based at least in part on the captured media object, the first input, and the second input; and
displaying the second plurality of links to the web pages.

10. The method of claim 9 , further comprising:
displaying a user preference form on a display;
receiving user preference information in response to the displaying of the user preference form using at least one input control; and
transmitting over a cellular subsystem the user preference information to the remote server.

11. The method of claim 9 , further comprising:
displaying the plurality of suggested descriptions in a user metadata selection form on a display;
receiving user metadata selection input in response to the user metadata selection form; and
transmitting over a cellular subsystem the user metadata selection input.

12. The method of claim 11 , wherein the second plurality of links to the web pages are further determined based at least on the user metadata selection input.

13. The method of claim 12 , further comprising displaying thumbnails of the web pages corresponding to the second plurality of links on the display.

14. The method of claim 13 , further comprising:
receiving a user web page selection using at least one input control;
transmitting over the cellular subsystem the user web page selection;
receiving web page content, based on the user web page selection; and
displaying the web page content on the display.

15. The method of claim 11 , further comprising:
identifying selected ones of the plurality of suggested descriptions from the user metadata selection input; and
storing locally on the computing device selected ones of the plurality of suggested descriptions corresponding to the selected user metadata input and storing associations of the selected ones of the plurality of suggested descriptions with the captured media object.

16. The method of claim 9 , wherein the plurality of suggested descriptions of the captured media object have confidence scores that satisfy a predetermined confidence criterion.

17. An apparatus comprising:
a processor;
a display; and
a computer readable memory having, computer-executable instructions stored thereon which, when executed by a computing device, cause the computing device to perform operations comp

Assignees

• Microsoft Technology Licensing Llc
• Microsoft Technology Licensing Llc

Inventors

• El-Saban Motaz Ahmed
• Wang Xin-Jing
• Sayed May Abdelreheem

Classifications

• G06F3/04817

  using icons (graphical or visual programming using iconic symbols G06F8/34) · CPC title

• H04L67/02

  based on web technology, e.g. hypertext transfer protocol [HTTP] · CPC title

• G06F16/9535

  Search customisation based on user profiles and personalisation · CPC title

• G06F3/0482

  Interaction with lists of selectable items, e.g. menus · CPC title

• G06F16/7867

  using information manually generated, e.g. tags, keywords, comments, title and artist information, manually generated time, location and usage information, user ratings · CPC title