Multi-script handwriting recognition using a universal recognizer

Patent metadata

Field	Value
Publication number	US-9495620-B2
Application number	US-201414291865-A
Country	US
Kind code	B2
Filing date	May 30, 2014
Priority date	Jun 9, 2013
Publication date	Nov 15, 2016
Grant date	Nov 15, 2016

Abstract

Methods, systems, and computer-readable media related to a technique for providing handwriting input functionality on a user device. A handwriting recognition module is trained to have a repertoire comprising multiple non-overlapping scripts and capable of recognizing tens of thousands of characters using a single handwriting recognition model. The handwriting input module provides real-time, stroke-order and stroke-direction independent handwriting recognition. User interfaces for providing the handwriting input functionality are also disclosed.

First claim

What is claimed is:

1. A non-transitory computer-readable media having instructions stored thereon, the instructions, when executed by one or more processors, cause the processors to perform operations comprising:
training a multi-script handwriting recognition model based on spatially-derived features of a multi-script training corpus, the multi-script training corpus including respective handwriting samples corresponding to characters of at least three non-overlapping scripts; and
providing real-time handwriting recognition for a user's handwriting input using the multi-script handwriting recognition model that has been trained on the spatially-derived features of the multi-script training corpus.

2. The media of claim 1 , wherein the spatially-derived features of the multi-script training corpus are stroke-order independent and stroke-direction independent.

3. The media of claim 1 , wherein the training of the multi-script handwriting recognition model is independent of temporal information associated with respective strokes in the handwriting samples.

4. The media of claim 1 , wherein the at least three non-overlapping scripts include Chinese characters, emoji characters, and Latin script.

5. The media of claim 1 , wherein the at least three non-overlapping scripts include Chinese characters, Arabic script, and Latin script.

6. The media of claim 1 , wherein the at least three non-overlapping scripts include non-overlapping scripts defined by the Unicode standard.

7. The media of claim 1 , wherein training the multi-script handwriting recognition model further comprises:
providing the handwriting samples of the multi-script training corpus to a single convolutional neural network having a single input plane and a single output plane; and
determining, using the convolutional neural network, the spatially-derived features of the handwriting samples and respective weights for the spatially-derived features for differentiating characters of the at least three non-overlapping scripts represented in the multi-script training corpus.

8. The media of claim 1 , wherein the multi-script handwriting recognition model has at least thirty thousand output classes, representing at least thirty thousand characters spanning the at least three non-overlapping scripts.

9. The media of claim 1 , wherein providing real-time handwriting recognition for a user's handwriting input further comprises:
providing the multi-script handwriting recognition model to a user device, wherein the user device receives a plurality of handwritten strokes from the user, and locally performs handwriting recognition on one or more recognition units identified from the plurality of handwritten strokes based on the received multi-script handwriting recognition model.

10. The media of claim 1 , wherein providing real-time handwriting recognition for a user's handwriting input further comprises:
continuously revising one or more recognition results for the user's handwriting input in response to continued additions to or revisions of the handwriting input by the user; and
in response to each revision of the one or more recognition results, displaying the respective revised one or more recognition results to the user in a candidate display area of the handwriting input user interface.

11. The media of claim 1 , comprising instructions, which when executed by the one or more processors, cause the processors to perform operations comprising:
providing the multi-script handwriting recognition model to a plurality of devices having no existing overlap in input languages, wherein the multi-script handwriting recognition model is used on each of the plurality of devices for handwriting recognition of a different input language associated with said each user device.

12. A method of providing multi-script handwriting recognition, comprising:
at a device having one or more processors and memory:
training a multi-script handwriting recognition model based on spatially-derived features of a multi-script training corpus, the multi-script training corpus including respective handwriting samples corresponding to characters of at least three non-overlapping scripts; and
providing real-time handwriting recognition for a user's handwriting input using the multi-script handwriting recognition model that has been trained on the spatially-derived features of the multi-script training corpus.

13. A system, comprising:
one or more processors; and
memory having instructions stored thereon, the instructions, when executed by the one or more processors, cause the processors to perform operations comprising:
training a multi-script handwriting recognition model based on spatially-derived features of a multi-script training corpus, the multi-script training corpus including respective handwriting samples corresponding to characters of at least three non-overlapping scripts; and
providing real-time handwriting recognition for a user's handwriting input using the multi-script handwriting recognition model that has been trained on the spatially-derived features of the multi-script training corpus.

14. A non-transitory computer-readable media having instructions stored thereon, the instructions, when executed by one or more processors, cause the processors to perform operations comprising:
receiving a multi-script handwriting recognition model, the multi-script recognition model having been trained on spatially-derived features of a multi-script training corpus, the multi-script training corpus including respective handwriting samples corresponding to characters of at least three non-overlapping scripts;
receiving a handwriting input from a user, the handwriting input comprising one or more handwritten strokes provided on a touch-sensitive surface coupled to the user device; and
in response to receiving the handwriting input, providing in real-time one or more handwriting recognition results to the user based on the multi-script handwriting recognition model that has been trained on the spatially-derived features of the multi-script training corpus.

15. The media of claim 14 , wherein providing real-time handwriting recognition results to the user further comprises:
segmenting the user's handwriting input into one or more recognition units, each recognition unit comprising one or more of the handwritten strokes provided by the user;
providing a respective image of each of the one or more recognition units as an input to the multi-script handwriting recognition model; and
for at least one of the one or more recognition units, obtaining from the multi-script handwriting recognition model, at least a first output character from a first script, and at least a second output character from a second script different from the first script.

16. The media of claim 15 , wherein providing real-time handwriting recognition results to the user further comprises:
displaying both the first output character and the second output character in a candidate display area of a handwriting input user interface of the user device.

17. The media of claim 15 , wherein providing real-time handwriting recognition results to the user further comprises:
selectively displaying one of the first output character and the second output character based on which one of the first or second scripts is a respective script used in a soft keyboard currently installed on the user device.

18. The media of claim 14 , wherein providing real-time handwriting recognition for a user's handwriting input further comprises:
continuously revising one or mo

Assignees

• Apple Inc

Inventors

• Dolfing Jannes G A
• Groethe Karl M
• Dixon Ryan S
• Bellegarda Jerome R

Classifications

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title

• G06F40/129

  Handling non-Latin characters, e.g. kana-to-kanji conversion · CPC title

• G06F3/04883

  for inputting data by handwriting, e.g. gesture or text · CPC title

• G06N20/00

  Machine learning · CPC title

• G06N3/088

  Non-supervised learning, e.g. competitive learning · CPC title