Automatic translation of digital graphic novels

Patent metadata

Field	Value
Publication number	US-9881003-B2
Application number	US-201514863394-A
Country	US
Kind code	B2
Filing date	Sep 23, 2015
Priority date	Sep 23, 2015
Publication date	Jan 30, 2018
Grant date	Jan 30, 2018

Abstract

Digital graphic novel content is received and features of the graphic novel content are identified. At least one of the identified features includes text. Contextual information corresponding to the feature or features that include text is generated based on the identified features. The contextual information is used to aid translation of the text included in the feature or features that include text.

First claim

What is claimed is:

1. A computer-implemented method comprising:
receiving, by at least one processor, digital graphic novel content;
producing, by at least one processor, a numerical map that represents an image extracted from the digital graphic novel content;
responsive to inputting the numerical map into a first artificial neural network of a machine learning model configured to determine regions of the digital graphic novel content that are likely to include speech bubbles, receiving, by the at least one processor, from the first artificial neural network, a plurality of candidate regions of the digital graphic novel content that are likely to include speech bubbles; and
responsive to inputting the plurality of candidate regions into a second artificial neural network of the machine learning model, receiving, by at least one processor, from the second artificial neural network, features of the digital graphic novel content that include a plurality of speech bubbles containing text;
generating, by at least one processor, based on the features of the digital graphic novel content, contextual information corresponding to the features, the contextual information including the text of the plurality of speech bubbles in an intended reading order of the plurality of speech bubbles; and
automatically translating, by at least one processor, based at least in part on the contextual information, from a first natural language to a second natural language, the text contained in the plurality of speech bubbles to produce translated text.

2. The computer-implemented method of claim 1 wherein automatically translating the text comprises:
after extracting the text contained in the plurality of speech bubbles, compiling the text into a single piece of text based on the intended reading order; and
translating the single piece of text from the first natural language to the second natural language to produce the translated text.

3. The computer-implemented method of claim 1 , wherein the contextual information further includes a depicted character or object contained in the digital graphic novel content, and automatically translating the text comprises applying a translation algorithm to the text contained in the plurality of speech bubbles that is tailored for the depicted character or object contained in the digital graphic novel content.

4. The computer-implemented method of claim 1 , further comprising:
creating a packaged digital graphic novel including the digital graphic novel content and presentation metadata, the presentation metadata including the translated text and an indication of the at least one feature of the features of the digital graphic novel to which the translated text corresponds; and
providing, by the at least one processor, the packaged digital graphic novel to a reader device that is configured to present the digital graphic novel content in a manner in accordance with the presentation metadata.

5. The computer-implemented method of claim 4 , wherein the reader device is configured to display the translated text within the plurality of speech bubbles in place of the text contained in the plurality of speech bubbles.

6. The computer-implemented method of claim 1 , wherein the contextual information further includes depicted weather contained in the digital graphic novel content, and automatically translating the text comprises applying a translation algorithm to the text contained in the plurality of speech bubbles that is tailored for the depicted weather contained in the digital graphic novel content.

7. The computer-implemented method of claim 1 , wherein each of the plurality of candidate regions is a panel of the digital graphic novel content.

8. The computer-implemented method of claim 1 , wherein generating the contextual information associated with the features of the digital graphic novel content includes limiting the contextual information to information contained in each panel of the digital graphic novel content that includes the plurality of speech bubbles.

9. The computer-implemented method of claim 1 , wherein generating the contextual information associated with the features of the digital graphic novel content includes limiting the contextual information to information depicted in the digital graphic novel content that is within a threshold distance the plurality of speech bubbles.

10. The computer-implemented method of claim 9 , wherein automatically translating the text contained in the plurality of speech bubbles to produce translated text comprises weighting portions of the information depicted in the digital graphic novel content by a proximity of that portion to the plurality of speech bubbles.

11. An electronic device comprising:
at least one processor; and a non-transitory computer-readable storage medium storing instructions executable by the at least one processor for:
receiving digital graphic novel content;
producing a numerical map that represents an image extracted from the digital graphic novel content;
responsive to inputting the numerical map into a first artificial neural network of a machine learning model configured to determine regions of the digital graphic novel content that are likely to include speech bubbles, receiving, from the first artificial neural network, a plurality of candidate regions of the digital graphic novel content that are likely to include speech bubbles; and
responsive to in putting the plurality of candidate regions into a second artificial neural network of the machine learning model, receiving, from the second artificial neural network, features of the digital graphic novel content that include a plurality of speech bubbles containing text;
generating, based on the features of the digital graphic novel content, contextual information associated with the features of the digital graphic novel content, the contextual information including the text of the plurality of speech bubbles in an intended reading order of the plurality of speech bubbles; and
automatically translating, based at least in part on the contextual information, from a first natural language to a second natural language, the text contained in the plurality of speech bubbles to produce translated text.

12. The electronic device of claim 11 , wherein automatically translating the text comprises:
after extracting the text contained in the plurality of speech bubbles, compiling the text into a single piece of text based on the intended reading order; and
translating the single piece of text from the first natural language to the second natural language to produce the translated text.

13. The electronic device of claim 11 , wherein the contextual information further includes a depicted character or object contained in the digital graphic novel content, and automatically translating the text comprises applying a translation algorithm to the text contained in the plurality of speech bubbles that is tailored for the depicted character or object contained in the digital graphic novel content.

14. The electronic device of claim 11 , wherein the executable computer program code further includes instructions for:
creating a packaged digital graphic novel including the digital graphic novel content and presentation metadata, the presentation metadata including the translated text and an indication of the at least one feature of the features of the digital graphic novel to which the translated text corresponds; and
providing, by the at least one processor, the packaged digital graphic novel to a reader device that is configured to present the digital graphic novel content in a manner in accordance wi

Assignees

• Google Inc
• Google Llc

Inventors

• Hartrell Greg Don
• Ghosh Debajit
• Vaughan-Vail Matthew William
• Rivlin John Michael
• Conboy Garth
• Gu xinxing
• Toshev Alexander Toshkov

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06F40/174

  Form filling; Merging · CPC title

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title

• G06F40/58

  Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation · CPC title

• G06F40/47 Primary

  Machine-assisted translation, e.g. using translation memory · CPC title