Actionable content displayed on a touch screen

Patent metadata

Field	Value
Publication number	US-2016210040-A1
Application number	US-201615085435-A
Country	US
Kind code	A1
Filing date	Mar 30, 2016
Priority date	Sep 27, 2013
Publication date	Jul 21, 2016
Grant date	—

Abstract

Some implementations may present a media file that includes video on a touchscreen display. A user gesture performed on the touchscreen display may be detected. The user gesture may include one of a tap gesture, a swipe gesture, or a tap and hold and drag while holding gesture. Text selected by the user gesture may be determined. One or more follow-up actions may be performed automatically based at least partly on the text selected by the user gesture.

First claim

1 . (canceled)

2 . A computer-implemented method comprising:
displaying media on a touchscreen display, the media including a photograph; detecting a user gesture performed on the touchscreen display; determining text selected by the user gesture; determining a user intent based at least partly on the text selected by the user gesture; determining a context associated with the text selected by the user gesture based on the user intent, the context including additional text captured in the media, wherein the additional text is associated with the text selected by the user gesture; and automatically performing one or more follow-up actions based at least partly on the text selected by the user gesture and based at least partly on the context.

3 . The computer-implemented method of claim 2 , further comprising:
extracting additional text from the media to create extracted text; determining positional data associated with the user gesture; and determining the text selected by the user gesture based on correlating the extracted text with the positional data.

4 . The computer-implemented method of claim 2 , further comprising:
determining one or more user preferences; and selecting the one or more follow-up actions based on the one or more user preferences.

5 . The computer-implemented method of claim 2 , further comprising:
determining one or more default actions; and selecting the one or more follow-up actions based on the one or more default actions.

6 . The computer-implemented method of claim 2 , further comprising:
determining that the user gesture comprises a tap gesture; and determining that the text selected by the user gesture comprises a word.

7 . The computer-implemented method of claim 2 , further comprising:
determining that the user gesture comprises a swipe gesture; and determining that the text selected by the user gesture comprises one or more words.

8 . The computer-implemented method of claim 2 , further comprising:
determining that the user gesture comprises a tap and hold gesture including a drag while holding gesture; and determining that the text selected by the user gesture comprises a plurality of words.

9 . A computer implemented method comprising:
displaying one or more portions of a first image on a touchscreen display; receiving, by the touchscreen display, input comprising a user gesture; identifying selected text in the first image based on the user gesture; determining that the user gesture comprises:
a tap and hold gesture; and
a drag while holding gesture;
determining that the selected text comprises a plurality of words included in the first image and a second image; and automatically performing at least one follow-up action based at least partly on the selected text.

10 . The computer implemented method of claim 9 , further comprising:
receiving, by the touchscreen display, a second input comprising a tap gesture; and determining that the selected text in the first image comprises a word.

11 . The computer implemented method of claim 9 , further comprising:
receiving, by the touchscreen display, a third input comprising a swipe gesture; and determining that the selected text comprises two or more words included in the first image.

12 . The computer implemented method of claim 9 , wherein:
the tap and hold gesture causes the first image to be selected; and the drag while holding gesture causes selection of text in the first image and the second image to create the selected text.

13 . The computer implemented method of claim 9 , further comprising:
translating the selected text from a source language to a target language to create translated text.

14 . The computer implemented method of claim 9 , further comprising:
submitting a search query that includes the selected text to a search engine.

15 . The computer implemented method of claim 13 , further comprising:
submitting a search query that includes translated text to a search engine.

16 . The computer implemented method of claim 9 , further comprising:
displaying results from automatically performing the at least one follow-up action in a pop-up window that at least partially overlays the first image on the touchscreen display.

17 . A computing device comprising:
a touchscreen display; one or more processors; and a memory storing instructions that are executable by the one or more processors to perform operations comprising:
displaying an image on the touchscreen display;
detecting a user gesture performed on the touchscreen display;
determining selected text comprising text selected by the user gesture;
determining a context associated with the selected text based on additional text that is within a predetermined distance from the selected text;
modifying the selected text to create modified text based at least partly on the additional text; and
automatically performing a follow-up action based on the modified text.

18 . The computing device of claim 17 , the operations further comprising:
receiving the image from a server.

19 . The computing device of claim 17 , the operations further comprising:
determining positional data identifying a position of the user gesture relative to the touchscreen display; extracting additional text from the image using optical character recognition to create extracted text; and correlating the extracted text with the positional data to identify the selected text.

20 . The computing device of claim 17 , the operations further comprising:
displaying results caused by automatically performing the follow-up action based on the selected text; receiving an additional user gesture selecting a portion of the results; performing an additional follow-up action based on the portion of the results; and displaying additional results based on performing the additional follow-up action.

21 . The computing device of claim 17 , the operations further comprising:
translating the selected text from a source language to a target language to create translated text; and submitting, to a search engine, a search query that includes at least one of the text or the translated text.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Bai Peng
• Du Jun
• Sun Lei
• Huo Qiang

Classifications

• G06F3/04883 Primary

  for inputting data by handwriting, e.g. gesture or text · CPC title

• G06F17/30796

  Physics · mapped topic

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F17/30253

  Physics · mapped topic

• G06F3/04842

  Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) · CPC title