Annotating relevant content in a screen capture image

Patent metadata

Field	Value
Publication number	US-9798708-B1
Application number	US-201414451389-A
Country	US
Kind code	B1
Filing date	Aug 4, 2014
Priority date	Jul 11, 2014
Publication date	Oct 24, 2017
Grant date	Oct 24, 2017

Abstract

Systems and methods are provided for highlighting relevant mobile onscreen content. For example, a mobile device can include memory storing instructions that, when executed by at least one processor, cause the mobile device to perform operations including capturing an image of a screen on the mobile device, the screen being displayed on a display of the mobile device, and providing the image to a server. The operations may also include receiving annotation data from the server, the annotation data including a visual cue that corresponds to a portion of the image that includes an entry in a list, the entry being associated with an entity in a graph-based data store relevant to a user of the mobile device, and display the annotation data with a second screen being displayed on the displaying of the mobile device so that the visual cue aligns with the entry in the second screen.

First claim

What is claimed is:

1. A system comprising:
at least one processor; and
memory storing instructions that, when executed by the at least one processor, cause the system to:
receive an image of a screen captured from a display of a mobile device,
perform text recognition on the image,
determine that the image includes a body of text, wherein the image is determined to include a body of text:
when at least one paragraph of text is identified in the image, or
when a majority of the image is determined to be text, or
when a number of words identified in the image exceeds a threshold value,

identify, responsive to the determination that the image includes a body of text, content-relevant text that summarizes the body of text identified in the image,
generate annotation data, the annotation data including a visual cue that changes an appearance of the content-relevant text, and
provide the annotation data to the mobile device.

2. The system of claim 1 , wherein the mobile device displays the annotation data with a screen being displayed on the display of the mobile device so that the visual cue is aligned with the content-relevant text in the display.

3. The system of claim 1 , wherein the image is a first image and the annotation data is first annotation data, and the memory further stores instructions that, when executed by the at least one processor, cause the system to:
receive a second image of a second screen captured on the mobile device;
identify at least one entity in the second image relevant to a user of the mobile device by performing recognition on the image;
determine that the entity is associated with a particular structure element of a display structure that repeats, the display structure being identified in the second image;
generate second annotation data for the second image, the second annotation data including a second visual cue that corresponds to the particular structure element; and
provide the second annotation data to the mobile device.

4. The system of claim 3 , wherein identifying the at least one entity as relevant includes:
determining a collection the entity belongs to; and
determining that the collection is consistent with an interest of the user.

5. The system of claim 3 , wherein identifying the at least one entity as relevant includes:
determining that the entity is a highly ranked entity for the user based on analysis of prior screen captures.

6. The system of claim 3 , wherein the display structure is a list, and the structure element is an entry in the list.

7. The system of claim 1 , wherein the annotation data is transparent other than the visual cue.

8. A method comprising:
receiving an image of a screen captured from a display of a mobile device;
determining entities in the image by performing recognition and entity detection on the image;
identifying at least one entity of the entities relevant to a user of the mobile device;
determining that the identified entity is associated with a particular structure element of a display structure that repeats, the display structure including structure elements other than those associated with entities identified as relevant to the user, wherein the display structure is identified in the image;
generating annotation data for the image, the annotation data including a visual cue that changes an appearance of the particular structure element when displayed on the mobile device; and
providing the annotation data to the mobile device.

9. The method of claim 8 , wherein the mobile device displays the annotation data with a screen being displayed on the display of the mobile device.

10. The method of claim 8 , wherein identifying the at least one entity as relevant includes:
determining a type for the entity; and
determining that the type is consistent with an interest of the user.

11. The method of claim 8 , wherein identifying the at least one entity as relevant includes:
determining that the entity is a highly ranked entity for the user based on analysis of prior screen captures.

12. The method of claim 8 , wherein determining entities in the image includes:
performing image recognition on the image; and
determining that a portion of the image corresponds to an entity.

13. The method of claim 8 , wherein the image is a first image and the annotation data is first annotation data, and the method further comprises:
receiving a second image of a second screen captured on the mobile device;
identifying at least one body of text in the second image by performing text recognition on the second image;
determining a portion of the body of text that summarizes the body of text;
generating second annotation data, the second annotation data including a second visual cue that corresponds to the portion of the body of text; and
providing the second annotation data to the mobile device.

14. The method of claim 8 , wherein the display structure is a list and the structure element is an entry in the list.

15. A mobile device comprising:
at least one processor; and
memory storing instructions that, when executed by the at least one processor, cause the mobile device to:
capture an image of a screen on the mobile device, the screen being displayed on a display of the mobile device,
provide the image to a server,
receive annotation data, the annotation data including a visual cue that corresponds to a portion of the image that includes all content of a structure element in a display structure that repeats, the structure element being associated with an entity in a graph-based data store relevant to a user of the mobile device and the display structure including structure elements other than the structure element associated with the entity identified as relevant to the user, wherein the visual cue changes an appearance of the structure element when displayed on the mobile device, and
display the annotation data with a second screen being displayed on the display of the mobile device so that the visual cue aligns with the display structure in the second screen.

16. The mobile device of claim 15 , the memory further storing instructions that, when executed by the at least one processor, causes the mobile device to:
determine whether the second screen matches the image; and
display the annotation data with the second screen when the second screen matches the image.

17. The mobile device of claim 15 , the memory further storing instructions that, when executed by the at least one processor, causes the mobile device to:
capture a second image of a third screen on the mobile device;
provide the second image to the server;
receive second annotation data, the second annotation data including a second visual cue that corresponds to a portion of the second image that represents text that summarizes a body of text in the second image; and
display the second annotation data with a fourth screen being displayed on the display of the mobile device so that the second visual cue aligns with the text that summarizes the body of text on the fourth screen.

18. The mobile device of claim 17 , the second annotation data further including a third visual cue that corresponds to a second portion of the second image, the second portion including content that represents an entity in the graph-based data store.

19. The mobile device of claim 15 , the annotation data further including a second visual cue that corresponds to a second portion of the image, the second portion representing text that summarizes a body of text in the image.

Assignees

• Google Inc

Inventors

• Sharifi Matthew
• Petrou David

Classifications

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06F40/295

  Named entity recognition · CPC title

• G06F40/169 Primary

  Annotation, e.g. comment data or footnotes · CPC title

• G06F40/134

  Hyperlinking · CPC title

• G06F16/24578

  using ranking · CPC title