Translation and display of text in picture

Patent metadata

Field	Value
Publication number	US-9773197-B2
Application number	US-201213412827-A
Country	US
Kind code	B2
Filing date	Mar 6, 2012
Priority date	Mar 22, 2007
Publication date	Sep 26, 2017
Grant date	Sep 26, 2017

Abstract

A method performed by a mobile terminal may include displaying an image via a display of the mobile device. A first user selection of a portion of the image is received and text in the selected portion of the image is identified. The identified text is displayed via the display. A second user selection of at least a portion of the identified text is received. The portion of the identified text is translated from a first language into a second language that differs from the first language. The translated text, in the second language, is displayed over the image via the display.

First claim

What is claimed is:

1. A method performed by a mobile device, comprising:
processing an image on a mobile device to automatically identify text in the image;
displaying the processed version of the image via a display of the mobile device, wherein the identified text is displayed as user selectable portions of the image from the displayed processed image;
receiving a first user selection of at least one of the displayed user selectable portions of the image;
performing character recognition on each selected portion of the image to generate at least one recognized word for each selected portion;
displaying each recognized word via the display as a user selectable word;
receiving a second user selection of one or more of the selectable words;
translating the selected recognized words from a first language into a second language that differs from the first language, wherein the first language is identified based on an evaluation of the selected recognized words;
receiving the second language from a user input where a user selects the second language from a plurality of languages; and
displaying the translation as text in the second language via the display.

2. The method of claim 1 , wherein displaying the translation as text in the second language via the display further comprises:
overlaying the translation over a portion of the image.

3. The method of claim 2 , wherein, when overlaying the translation over the portion of the image, at least some of the portion of the image is viewable behind the translated text.

4. The method of claim 2 , wherein the translation is displayed in a more prominent manner that the image.

5. The method of claim 1 , wherein the performing character recognition includes performing optical character recognition (OCR) processing.

6. The method of claim 5 , further comprising:
receiving input from a user selecting a language for performing the OCR processing.

7. The method of claim 1 , wherein the image includes at least one of a digital image or a video stream.

8. The method of claim 1 , further comprising:
determining position information associated with the mobile device; and
further identifying the first language based on the position information.

9. The method of claim 1 , further comprising:
determining the second language based on identification information associated with the mobile device.

10. The method of claim 1 , wherein translating the selected recognized words into the second language further comprises:
transmitting, by the mobile device, the selected recognized words to a server via a network;
translating, by the server, the selected recognized words into the second language; and
receiving, by the mobile device, the translation as text from the server via the network.

11. The method of claim 10 , further comprising:
transmitting identification information associated with the mobile device to the server; and
determining, by the server, the second language based on the identification information associated with the mobile device.

12. A mobile communication device, comprising:
a camera configured to store an image;
a display; and
a processor configured to:
process the image to automatically identify text in the image;
control the display to display the processed version of the image, wherein the identified text is displayed as user selectable portions of the image from the displayed processed image;
receive a first user selection of at least one of the displayed user selectable portions of the image;
perform character recognition on each selected portion of the image to generate at least one recognized word for each selected portion;
display each recognized word via the display as a user selectable word;
receive a second user selection of one or more of the selectable words;
receive a translation language from a user input where a user of the mobile communication device selects the translation language from a plurality of languages;
transmit the selected recognized words and the translation language to a server via a network, wherein the server is configured to:
identify a first language based on an evaluation of the selected recognized words, and
translate the selected recognized words from the first language into the translation language, the first language being different from the translation language;

receive the translation from the server via the network; and

display the translation as text, in the translation language, via the display.

13. The mobile communication device of claim 12 , wherein, when displaying the translation as text, at least a portion of the image is viewable behind the translation.

14. The mobile communication device of claim 12 , wherein the translation is displayed in a more prominent manner than the image.

15. The mobile communication device of claim 12 , wherein the character recognition comprises optical character recognition (OCR) processing.

16. A method performed by a mobile communications device, comprising:
capturing an image via a camera of the mobile communications device, wherein the image comprises a still image or a video stream;
processing the image on the mobile device to automatically identify text captured in the image;
displaying the processed version of the image via a display of the mobile communications device, wherein the identified text is displayed as user selectable portions of the image from the displayed processed image;
receiving a first user selection of at least one of the words from the displayed processed image;
performing character recognition on each selected portion of the image to generate at least one recognized word for each selected portion;
displaying each recognized word via the display as a user selectable word;
receiving a second user selection of one or more of the selectable words;
receiving a translation language from a user input where a user selects the translation language from a plurality of languages;
transmitting the selected recognized words and the translation language to a server via a network, wherein the server is configured to identify a first language based on an evaluation of the selected recognized words, and to translate the selected recognized words from the first language into the translation language, wherein the translation language differs from the first language;
receiving, from the server via the network, the translation as text based on the translating; and
displaying the translation as text in the translation language via the display.

17. The method of claim 16 , wherein the performing character recognition includes performing optical character recognition (OCR) processing.

18. The method of claim 17 , further comprising:
transmitting, to the server, identification information associated with the mobile communications device, wherein the server is configured to identify the translation language based on the identification information associated with the mobile communications device.

19. The method of claim 16 , wherein displaying the translation in the translation language via the display further comprises:
overlaying the translation over a portion of the image such that the portion of the image is viewable behind the translated text.

20. The method of claim 16 , wherein the performing character recognition includes character recognition processing carried out by the server.

Assignees

• Englund Hakan Mats
• Lessing Simon
• Sony Corp
• Sony Mobile Communications Inc

Inventors

• Englund Hakan Mats
• Lessing Simon

Classifications

• G06V10/10

  Image acquisition (document image scanning and transmission H04N1/00; control of digital cameras H04N23/60) · CPC title

• G06F40/58 Primary

  Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation · CPC title

• G06V30/142

  using hand-held instruments; Constructional details of the instruments · CPC title

• G06V30/10

  Character recognition · CPC title

• G06K9/228

  Physics · mapped topic