Assist layer with automated extraction

Patent metadata

Field	Value
Publication number	US-2016350136-A1
Application number	US-201514804861-A
Country	US
Kind code	A1
Filing date	Jul 21, 2015
Priority date	May 27, 2015
Publication date	Dec 1, 2016
Grant date	—

Abstract

In general, this disclosure is directed to techniques for outputting, by a computing device and for display at a display component that is operatively coupled to the computing device, a graphical user interface of an application. The computing device determines context data from content included in the graphical user interface. The computing device further determines, based at least in part on the context data, an indication of assistance that is contextually related to the content included in the graphical user interface. The computing device outputs the indication of the assistance.

First claim

1 . A method comprising:
outputting, by a computing device and for display at a display component that is operatively coupled to the computing device, a graphical user interface of an application; determining context data from content included in the graphical user interface that is output for display; determining, based at least in part on the context data, an indication of assistance that is contextually related to the content included in the graphical user interface; and outputting, by the computing device, the indication of the assistance.

2 . The method of claim 1 , wherein determining the context data comprises:
accessing a view hierarchy at the computing device; and identifying a view object of the view hierarchy that is represented in the graphical user interface.

3 . The method of claim 2 , further comprising:
extracting at least one property of the view object; and determining, based at least in part on the at least one property, at least a portion of the context data.

4 . The method of claim 1 , wherein the context data from content included in the graphical user interface is primary context data, and wherein determining the indication of assistance comprises:
determining the indication of assistance based at least in part on the primary context data of the graphical user interface and secondary context data, wherein the secondary context data comprises at least one of a location, a time, a user identity, cloud computing data, calendar information, contact information, a previous application, and a temporal relation between a current time and a time that the previous application was run.

5 . The method of claim 1 , wherein determining the context data comprises:
capturing a screenshot of the graphical user interface of the application; performing an image analysis on the screenshot; and determining, based at least in part on the image analysis, the context data.

6 . The method of claim 5 , wherein performing the image analysis comprises:
performing an optical character recognition process on the screenshot to determine one or more characters in the screenshot; and determining, based at least in part on the one or more characters, the context data.

7 . The method of claim 1 , wherein determining the context data comprises:
analyzing a view hierarchy of the application; for a view object included in the view hierarchy:
identifying the view object;
determining information associated with the identified view object; and
determining, based at least in part on the determined information, at least a portion of the context data.

8 . The method of claim 1 , wherein determining the indication of assistance further comprises:
sending, to a remote computing device, the context data, wherein the remote computing device determines information for the indication of assistance from the context data; receiving, from the remote computing device, the information for the indication of assistance; and generating for annunciation, based at least in part on the information, the indication of assistance.

9 . A computing device comprising:
at least one processor; and at least one module, operable by the at least one processor to:
output, for display at a display component that is operatively coupled to the computing device, a graphical user interface of an application;
determine context data from content included in the graphical user interface;
determine, based at least in part on the context data, an indication of assistance that is contextually related to the content included in the graphical user interface; and
output the indication of the assistance.

10 . The computing device of claim 9 , wherein the at least one module is executing with operating system privileges on the computing device, and wherein the at least one module is operable by the at least one processor to:
extract at least one property of the view object; and determine, based at least in part on the at least one property, at least a portion of the context data.

11 . The computing device of claim 9 , wherein the at least one module operable by the at least one processor is further operable to:
send, to a remote computing device, the context data, wherein the remote computing device determines information for the indication of assistance from the context data; receive, from the remote computing device, the information for the indication of assistance; and generate for annunciation, based at least in part on the information, the indication of assistance.

12 . The computing device of claim 9 , wherein the context data from content included in the graphical user interface is primary context data, and wherein the at least one module is further operable to:
determine the indication of assistance based at least in part on the primary context data of the graphical user interface and secondary context data, wherein the secondary context data comprises at least one of a location, a time, a user identity, cloud computing data, calendar information, contact information, a previous application, and a temporal relation between a current time and a time that the previous application was run.

13 . The computing device of claim 9 , wherein the at least one module is operable by the at least one processor to:
capture a screenshot of the graphical user interface; perform an image analysis on the screenshot; and determine, based at least in part on the image analysis, the context data.

14 . The computing device of claim 13 , wherein the at least one module is operable by the at least one processor to:
perform an optical character recognition process on the screenshot to determine one or more characters in the screenshot; and determine, based at least in part on the one or more characters, the context data.

15 . The computing device of claim 9 , wherein the at least one module is operable by the at least one processor to determine context data from content included in the graphical user interface to:
analyze a view hierarchy of the application at the computing device; identify a view object included in the view hierarchy that is represented in a current screen image of the graphical user interface of the application; for the view object included in the view hierarchy:
determine information associated with the view object; and
determine, based at least in part on the determined information, the context data.

16 . A computer-readable storage medium encoded with instructions that, when executed, cause at least one processor of a computing device to:
output, for display at a display component that is operatively coupled to the computing device, a graphical user interface of an application; determine context data from content included in the graphical user interface; determine, based at least in part on the context data, an indication of assistance that is contextually related to the content included in the graphical user interface; and output the indication of the assistance.

17 . The computer-readable storage medium of claim 16 , wherein the instructions causing the at least one processor to determine the indication of assistance comprise instructions that, when executed, cause the at least one processor of the computing device to:
send, to a remote computing device, the context data, wherein the remote computing device determines information for the indication of assistance from the context data; receive, from the remote computing device, the information for the indication of assistance; and generate for annunciation, based at least in part

Assignees

• Google Inc

Inventors

• Karlo Thomas Ramon
• Chennapragada Aparna
• Cleron Michael Andrew
• Kirazci Ulas
• Shiplacoff Daniel Marc Gatan
• Hansson Othar
• Boortz Jeff
• Hackborn Dianne

Classifications

• G06F9/453 Primary

  Help systems · CPC title

• G06F3/04842

  Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) · CPC title

• G06F9/44 Primary

  Arrangements for executing specific programs · CPC title

• G06F16/00

  Information retrieval; Database structures therefor; File system structures therefor · CPC title

• G06F16/903

  Querying (for retrieval from the web G06F16/953) · CPC title