Actionable content displayed on a touch screen

Patent metadata

Field	Value
Publication number	US-9329692-B2
Application number	US-201314040443-A
Country	US
Kind code	B2
Filing date	Sep 27, 2013
Priority date	Sep 27, 2013
Publication date	May 3, 2016
Grant date	May 3, 2016

Abstract

Some implementations may present a media file that includes video on a touchscreen display. A user gesture performed on the touchscreen display may be detected. The user gesture may include one of a tap gesture, a swipe gesture, or a tap and hold and drag while holding gesture. Text selected by the user gesture may be determined. One or more follow-up actions may be performed automatically based at least partly on the text selected by the user gesture.

First claim

What is claimed is:

1. A method comprising:
under control of one or more processors configured with instructions that are executable by the one or more processors to perform acts comprising:
initiating presentation of a media file on a touchscreen display, the media file including video;
detecting a user gesture performed on the touchscreen display;
determining text selected by the user gesture;
determining a user intent based at least partly on the text selected by the user gesture;
determining a context associated with the text selected by the user gesture based on the user intent, the context including additional text captured in the video, wherein the additional text is associated with the text selected by the user gesture; and
automatically performing one or more follow-up actions based at least partly on the text selected by the user gesture and based at least partly on the context.

2. The method of claim 1 , wherein determining the text selected by the user gesture comprises:
creating one or more screen captures based on at least a portion of the media file;
extracting text from the one or more screen captures to create extracted text;
determining positional data associated with the user gesture; and
determining the text selected by the user gesture based on correlating the extracted text with the positional data.

3. The method of claim 1 , further comprising:
determining one or more user preferences; and
selecting the one or more follow-up actions based on the one or more user preferences.

4. The method of claim 1 , further comprising:
determining one or more default actions; and
selecting the one or more follow-up actions based on the one or more default actions.

5. The method of claim 1 , further comprising:
determining that the user gesture includes a tap and hold gesture followed by a drag while holding gesture;
pausing the presentation of the video on the touchscreen display based at least partly on the tap and hold gesture; and
determining the text selected by the user gesture based at least partly on the drag while holding gesture.

6. The method of claim 1 , wherein the one or more follow-up actions include:
submitting, to a search engine, a search query that includes the text or translated text, wherein the translated text is created by translating the text selected by the user gesture from a source language to a target language.

7. The method of claim 1 , further comprising:
determining that the user gesture comprises a tap gesture; and
determining that the text selected by the user gesture comprises a word.

8. The method of claim 1 , further comprising:
determining that the user gesture comprises a swipe gesture; and
determining that the text selected by the user gesture comprises a plurality of words.

9. A computer implemented method comprising:
displaying one or more portions of a video file on a touchscreen display;
receiving, by the touchscreen display, input comprising a user gesture;
identifying selected text in the video file based on the user gesture;
determining that the user gesture comprises:
a tap and hold gesture; and
a drag while holding gesture;

determining that the selected text comprises a plurality of words that span more than one frame of the video file; and
automatically performing at least one follow-up action based at least partly on the selected text.

10. The computer implemented method of claim 9 , further comprising:
receiving, by the touchscreen display, second input comprising a tap gesture; and
determining that the selected text in the video file comprises a word.

11. The computer implemented method of claim 9 , further comprising:
receiving, by the touchscreen display, third input comprising a swipe gesture; and
determining that the selected text comprises two or more words.

12. The computer implemented method of claim 9 , wherein:
the tap and hold gesture pauses presentation of the video file; and
the drag while holding gesture causes selection of text in one or more frames of the video file to create the selected text.

13. The computer implemented method of claim 9 , wherein the at least one follow-up action comprises one or more of:
translating the selected text from a source language to a target language to create translated text;
submitting a first search query that includes the selected text to a search engine; or
submitting a second search query that includes the translated text to the search engine.

14. The computer implemented method of claim 9 , further comprising:
displaying results from automatically performing the at least one follow-up action in a pop-up window that at least partially overlays the one or more portions of the video file being presented on the touchscreen display.

15. A computing device comprising,
a touchscreen display;
one or more processors; and
one or more computer readable storage media storing instructions that are executable by the one or more processors to perform acts comprising:
playing a media file that includes video;
detecting a user gesture performed on the touchscreen display while the video is playing;
identifying, in a frame of the video, selected text based on the user gesture;
determining a context associated with the selected text based on additional text that is within a predetermined distance from the selected text;
modifying the selected text to create modified text based at least partly on the additional text; and
automatically performing a follow-up action based on the modified text.

16. The computing device of claim 15 , wherein the user gesture comprises one of a tap gesture, a swipe gesture, or a tap and hold and drag while holding gesture.

17. The computing device of claim 15 , further comprising receiving the media file as a stream over a network from a server.

18. The computing device of claim 15 , wherein identifying, in the frame of the video, selected text based on the user gesture comprises:
determining positional data associated with the user gesture, the positional data identifying a position of the user gesture relative to the touchscreen display;
extracting text from the frame of the video using optical character recognition to create extracted text; and
correlating the extracted text with the positional data to identify the selected text.

19. The computing device of claim 15 , the acts further comprising:
displaying results caused by automatically performing the follow-up action based on the selected text;
receiving an additional user gesture selecting a portion of the results;
performing an additional follow-up action based on the portion of the results; and
displaying additional results caused by performing the additional follow-up action.

20. The computing device of claim 15 , the acts further comprising:
determining that the user gesture is being performed on the touchscreen display while the video is playing;
determining a time code associated with a current portion of the video that is being played; and
storing the user gesture and the time code in a history file associated with a user.

Assignees

• Microsoft Technology Licensing Inc
• Microsoft Technology Licensing Llc

Inventors

• Bai Peng
• Huo Qiang
• Du Jun
• Sun Lei

Classifications

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F3/04883 Primary

  for inputting data by handwriting, e.g. gesture or text · CPC title

• H04N21/84

  Generation or processing of descriptive data, e.g. content descriptors {(systems specially adapted for using meta-information in broadcast systems H04H60/73)} · CPC title

• H04N21/4622

  Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet (web site content organization and management for information retrieval from the Internet G06F16/958; transmission by internet of broadcast information H04H60/82; stock exchange data over packet-switching network H04L12/1804; push services including data channel over packet-switching network H04L12/1859) · CPC title

• H04N21/44008

  involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream (arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59) · CPC title