Multi-task recurrent neural network architecture for efficient morphology handling in neural language modeling

Patent metadata

Field	Value
Publication number	US-10657328-B2
Application number	US-201715851487-A
Country	US
Kind code	B2
Filing date	Dec 21, 2017
Priority date	Jun 2, 2017
Publication date	May 19, 2020
Grant date	May 19, 2020

Abstract

The present disclosure generally relates to systems and processes for morpheme-based word prediction. An example method includes receiving a current word; determining a context of the current word based on the current word and a context of a previous word; determining, using a morpheme-based language model, a likelihood of a prefix based on the context of the current word; determining, using the morpheme-based language model, a likelihood of a stem based on the context of the current word; determining, using the morpheme-based language model, a likelihood of a suffix based on the context of the current word; determining a next word based on the likelihood of the prefix, the likelihood of the stem, and the likelihood of the suffix; and providing an output including the next word.

First claim

What is claimed is:

1. An electronic device, comprising:
one or more processors;
a memory; and
one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for:
receiving a current word;
determining a context of the current word based on the current word and a context of a previous word;
determining, using a morpheme-based language model, a first representation indicating a likelihood of each prefix of a predetermined set of prefixes, wherein the likelihood of each prefix is determined based on the context of the current word;
determining, using the morpheme-based language model, a second representation indicating a likelihood of each stem of a predetermined set of stems, wherein the likelihood of each stem is determined based on the context of the current word;
determining, using the morpheme-based language model, a third representation indicating a likelihood of each suffix of a predetermined set of suffixes, wherein the likelihood of each suffix is determined based on the context of the current word;
determining a next word based on the first representation, the second representation, and the third representation; and
providing an output including the next word.

2. The electronic device of claim 1 , wherein at least one likelihood of a prefix, at least one likelihood of a stem, and at least one likelihood of a suffix are determined, at least in part, concurrently.

3. The electronic device of claim 1 , wherein determining a next word based on the first representation, the second representation, and the third representation comprises:
concatenating at least one prefix, at least one stem, and at least one suffix.

4. The electronic device of claim 1 , wherein the one or more programs further include instructions for:
determining whether the next word is valid.

5. The electronic device of claim 4 , wherein determining whether the next word is valid comprises:
determining whether the next word is included in a lexicon.

6. The electronic device of claim 1 , wherein determining a first representation indicating a likelihood of each prefix of a predetermined set of prefixes comprises:
determining, using the morpheme-based language model, a likelihood of a first prefix;
determining, using the morpheme-based language model, a likelihood of a second prefix, and wherein determining a next word based on the first representation, the second representation, and the third representation comprises:
determining the next word based on the likelihood of the first prefix and the second prefix.

7. The electronic device of claim 1 , wherein determining a third representation indicating a likelihood of each suffix of a predetermined set of suffixes comprises:
determining, using the morpheme-based language model, a likelihood of a first suffix;
determining, using the morpheme-based language model, a likelihood of a second suffix, and wherein determining a next word based on the first representation, the second representation, and the third representation comprises:
determining the next word based on the likelihood of the first suffix and the second suffix.

8. The electronic device of claim 1 , wherein at least one prefix of the predetermined set of prefixes is an empty prefix.

9. The electronic device of claim 1 , wherein at least one suffix of the determined set of suffixes is an empty suffix.

10. The electronic device of claim 1 , wherein providing an output including the next word comprises:
displaying the next word on a display of the electronic device.

11. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of an electronic device, cause the electronic device to:
receive a current word;
determine a context of the current word based on the current word and a context of a previous word;
determine, using a morpheme-based language model, a first representation indicating a likelihood of each prefix of a predetermined set of prefixes, wherein the likelihood of each prefix is determined based on the context of the current word;
determine, using the morpheme-based language model, a second representation indicating a likelihood of each stem of a predetermined set of stems, wherein the likelihood of each stem is determined based on the context of the current word;
determine, using the morpheme-based language model, a third representation indicating a likelihood of each suffix of a predetermined set of suffixes, wherein the likelihood of each suffix is determined based on the context of the current word;
determine a next word based on the first representation, the second representation, and the third representation; and
providing an output including the next word.

12. The non-transitory computer-readable storage medium of claim 11 , wherein at least one likelihood of a prefix, at least one likelihood of a stem, and at least one likelihood of a suffix are determined, at least in part, concurrently.

13. The non-transitory computer-readable storage medium of claim 11 , wherein determining a next word based on the first representation, the second representation, and the third representation comprises:
concatenating at least one prefix, at least one stem, and at least one suffix.

14. The non-transitory computer-readable storage medium of claim 11 , wherein the instructions further cause the electronic device to:
determine whether the next word is valid.

15. The non-transitory computer-readable storage medium of claim 14 , wherein determining whether the next word is valid comprises:
determining whether the next word is included in a lexicon.

16. A method, comprising:
at an electronic device having one or more processors:
receiving a current word;
determining a context of the current word based on the current word and a context of a previous word;
determining, using a morpheme-based language model, a first representation indicating a likelihood of each prefix of a predetermined set of prefixes, wherein the likelihood of each prefix is determined based on the context of the current word;
determining, using the morpheme-based language model, a second representation indicating a likelihood of each stem of a predetermined set of stems, wherein the likelihood of each stem is determined based on the context of the current word;
determining, using the morpheme-based language model, a third representation indicating a likelihood of each suffix of a predetermined set of suffixes, wherein the likelihood of each suffix is determined based on the context of the current word;
determining a next word based on the first representation, the second representation, and the third representation; and
providing an output including the next word.

17. The method of claim 16 , wherein at least one likelihood of a prefix, at least one likelihood of a stem, and at least one likelihood of a suffix are determined, at least in part, concurrently.

18. The method of claim 16 , wherein determining a next word based on the first representation, the second representation, and the third representation comprises:
concatenating at least one prefix, at least one stem, and at least one suffix.

19. The method of claim 16 , further comprising:
determining whether the next word is valid.

20. The method of claim 19 , wherein determining whether the next word is valid comprises:
d

Assignees

• Apple Inc

Inventors

• Bellegarda Jerome R
• Dolfing Jannes G

Classifications

• G06F40/268 Primary

  Morphological analysis · CPC title

• G10L25/30

  using neural networks · CPC title

• G06F40/274 Primary

  Converting codes to words; Guess-ahead of partial word inputs · CPC title

• G06F40/253

  Grammatical analysis; Style critique · CPC title

• G06F40/30

  Semantic analysis · CPC title