Automatically identifying and displaying objects of interest in a graphic novel

Patent metadata

Field	Value
Publication number	US-2017365083-A1
Application number	US-201615186208-A
Country	US
Kind code	A1
Filing date	Jun 17, 2016
Priority date	Jun 17, 2016
Publication date	Dec 21, 2017
Grant date	—

Abstract

Locations and presentation orders of objects of interest (e.g., speech bubbles) in digital graphic novel content are identified such that expanded versions of the objects of interest can be presented to a reader. Specifically, digital graphic novel content is received and locations of interest regions (e.g., rectangular text regions of speech bubbles) in the content are identified by applying a machine-learned model to the content. Locations and presentation orders of objects of interest in the digital graphic novel content are identified based on the identified locations of the interest regions. The digital graphic novel content and presentation metadata including the locations and presentation orders of the objects of interest are provided to a reading device such that expanded versions of the objects of interest are presented to the user in accordance with the presentation metadata.

First claim

What is claimed is:

1 . A computer-implemented method of providing digital graphic novel content to a reading device, the method comprising:
receiving digital graphic novel content; identifying locations of a plurality of interest regions of the digital graphic novel content by applying a model to the digital graphic novel content; identifying locations and presentation orders of a plurality of objects of interest in the digital graphic novel content based on the identified locations of the plurality of interest regions; creating presentation metadata for the digital graphic novel content indicating the identified locations and presentation orders of the plurality of objects of interest; and providing the digital graphic novel content and the presentation metadata to the reading device for presentation of expanded versions of the plurality of objects of interest in accordance with the presentation metadata.

2 . The computer-implemented method of claim 1 , wherein the model is a machine-learned model, and further comprising building the machine-learned model, the building comprising:
selecting a set of images; tagging interest regions in the set of images to generate training data of tagged images; and building the machine-learned model based on the tagged images of the training data, the machine-learned model capable of receiving the digital graphic novel content and generating the locations of the plurality of interest regions in the digital graphic novel content.

3 . The computer-implemented method of claim 1 , wherein the plurality of objects comprise speech bubble objects in the digital graphic novel content that contain text associated with characters in the digital graphic novel content.

4 . The computer-implemented method of claim 3 , wherein the plurality of interest regions comprise text regions of the speech bubble objects in the digital graphic novel content that encompass the text of the speech bubble objects.

5 . The computer-implemented method of claim 1 , wherein identifying locations of the plurality of objects of interest comprises, for each identified interest region, identifying a set of points surrounding the interest region indicative of the location of the corresponding object of interest, the set of points identified based on a color gradient between a color associated with the interest region and colors of points surrounding the interest region.

6 . The computer-implemented method of claim 1 , wherein identifying presentation orders of the plurality of objects of interest comprises, for each object of interest:
identifying a reference point associated with the object of interest indicating coordinates of a distinctive feature of the object of interest; determining a panel containing the object of interest based on a spatial relationship between the reference point and location of the panel; and determining the presentation order of the object of interest within the panel based on spatial relationships between the reference point and reference points of other objects of interest contained within the panel.

7 . The computer-implemented method of claim 6 , wherein the object of interest is a speech bubble object in the digital graphic novel content and the distinctive feature is an anchor point of the speech bubble object.

8 . The computer-implemented method of claim 1 , further comprising:
obtaining feedback data on presentation of the digital graphic novel content; and updating the machine-learned model based on the obtained feedback data to improve presentation metadata associated with the digital graphic novel content.

9 . The computer-implemented method of claim 8 , wherein the feedback data includes portions of the digital graphic novel content that have been zoomed-in on the reader device.

10 . A non-transitory computer-readable storage medium storing executable computer program instructions for providing digital graphic novel content to a reading device, the computer program instructions comprising:
receiving digital graphic novel content; identifying locations of a plurality of interest regions of the digital graphic novel content by applying a model to the digital graphic novel content; identifying locations and presentation orders of a plurality of objects of interest in the digital graphic novel content based on the identified locations of the plurality of interest regions; creating presentation metadata for the digital graphic novel content indicating the identified locations and presentation orders of the plurality of objects of interest; and providing the digital graphic novel content and the presentation metadata to the reading device for presentation of expanded versions of the plurality of objects of interest in accordance with the presentation metadata.

11 . The computer-readable storage medium of claim 10 , wherein the model is a machine-learned model, and the computer program instructions further comprise building the machine-learned model, the building comprising:
selecting a set of images; tagging interest regions in the images to generate training data of tagged images; and building the machine-learned model based on the tagged images of the training data, the machine-learned model capable of receiving the digital graphic novel content and generating the locations of the plurality of interest regions in the digital graphic novel content.

12 . The computer-readable storage medium of claim 10 , wherein the plurality of objects comprise speech bubble objects in the digital graphic novel content that contain text associated with characters in the digital graphic novel content.

13 . The computer-readable storage medium of claim 10 , wherein identifying locations of the plurality of objects of interest comprises, for each identified interest region, identifying a set of points surrounding the interest region indicative of the location of the corresponding object of interest, the set of points identified based on a color gradient between a color associated with the interest region and colors of points surrounding the interest region.

14 . The computer-readable storage medium of claim 10 , wherein identifying presentation orders of the plurality of objects of interest comprises, for each object of interest:
identifying a reference point associated with the object of interest indicating coordinates of a distinctive feature of the object of interest; determining a panel containing the object of interest based on a spatial relationship between the reference point and location of the panel; and determining the presentation order of the object of interest within the panel based on spatial relationships between the reference point and reference points of other objects of interest contained within the panel.

15 . The computer-readable storage medium of claim 14 , wherein the object of interest is a speech bubble object in the digital graphic novel content and the distinctive feature is an anchor point of the speech bubble object.

16 . A server for providing digital graphic novel content to a reading device, comprising:
a processor for executing computer program instructions; and a non-transitory computer-readable storage medium storing computer program instructions executable to perform steps comprising:
receiving digital graphic novel content;
identifying locations of a plurality of interest regions of the digital graphic novel content by applying a model to the digital graphic novel content;
identifying locations and presentation orders of a plurality of objects of interest in the digital graphic novel

Assignees

• Google Inc

Inventors

• Hartrell Greg
• Conboy Garth
• Duga Brady
• Gu xinxing
• Beavers Adeline
• Ghosh Debajit
• Rivlin John
• Newberger Alan
• An Tae Wan
• Moon Jacob
• Salvarani Christopher
• Blass Karl

Classifications

• G06V30/19167

  Active pattern learning · CPC title

• G06V30/19147

  Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F18/2178

  based on feedback of a supervisor · CPC title

• G06F18/2413

  based on distances to training or reference patterns · CPC title