System and method for building diverse language models

Patent metadata

Field	Value
Publication number	US-9081760-B2
Application number	US-201113042890-A
Country	US
Kind code	B2
Filing date	Mar 8, 2011
Priority date	Mar 8, 2011
Publication date	Jul 14, 2015
Grant date	Jul 14, 2015

Abstract

Disclosed herein are systems, methods, and non-transitory computer-readable storage media for collecting web data in order to create diverse language models. A system configured to practice the method first crawls, such as via a crawler operating on a computing device, a set of documents in a network of interconnected devices according to a visitation policy, wherein the visitation policy is configured to focus on novelty regions for a current language model built from previous crawling cycles by crawling documents whose vocabulary considered likely to fill gaps in the current language model. A language model from a previous cycle can be used to guide the creation of a language model in the following cycle. The novelty regions can include documents with high perplexity values over the current language model.

First claim

We claim:

1. A method comprising:
identifying vocabulary gaps in a current language model;
establishing a visitation policy based on a previous crawling cycle and the vocabulary gaps, wherein the visitation policy identifies web pages likely to have information capable of filling the vocabulary gaps in the current language model, and wherein the visitation policy comprises a crawling schedule based on predicted perplexity of the web pages with respect to the current language model;
crawling, via a crawler operating on a computing device, the web-pages according to the crawling schedule, to yield new vocabulary words; and
generating a diverse language model based on the current language model and the new vocabulary words.

2. The method of claim 1 , further comprising recognizing received speech with the diverse language model.

3. The method of claim 1 , wherein the diverse language model is generated by modifying the current language model.

4. The method of claim 1 , wherein the web pages are identified using an information theoretic measure.

5. The method of claim 4 , wherein the web pages have high perplexity values over the current language model from a previous cycle.

6. The method of claim 1 , further comprising updating the visitation policy for the crawler once a specified number of pages is crawled.

7. The method of claim 6 , wherein updating the visitation policy is based on an expected perplexity value of the novelty regions.

8. The method of claim 7 , wherein the expected perplexity value of a web page is determined by evaluating links to the web page.

9. The method of claim 1 , further comprising merging a set of language models.

10. A system comprising:
a processor; and
a computer-readable storage medium having instructions stored which, when executed by the processor, cause the processor to perform operations comprising:
identifying vocabulary gaps in a current language model;
establishing a visitation policy based on a previous crawling cycle and the vocabulary gaps, wherein the visitation policy identifies web pages likely to have information capable of filling the vocabulary gaps in the current language model, and wherein the visitation policy comprises a crawling schedule based on predicted perplexity of the web pages with respect to the current language model;
crawling, via a crawler operating on a computing device, web-pages according to the crawling schedule, to yield new vocabulary words; and

generating a diverse language model based on the current language model and the new vocabulary words.

11. The system of claim 10 , wherein the web pages are identified using an information theoretic measure.

12. The system of claim 10 , wherein the web pages have high perplexity values over the current language model from a previous cycle.

13. The system of claim 10 , wherein the language model is further generated by updating the visitation policy for the crawler once a specified number of web pages is crawled.

14. The system of claim 13 , wherein updating the visitation policy is based on an expected perplexity value of the web pages.

15. A computer-readable storage device having instructions stored which, when executed by a computing device, cause the computing device to perform operations comprising:
identifying vocabulary gaps in a current language model;
establishing a visitation policy based on a previous crawling cycle and the vocabulary gaps, wherein the visitation policy identifies web pages likely to have information capable of filling the vocabulary gaps in the current language model, and wherein the visitation policy comprises a crawling schedule based on predicted perplexity of the web pages with respect to the current language model;
crawling, via a crawler operating on a computing device, web-pages according to the crawling schedule, to yield new vocabulary words; and
generating a diverse language model based on the current language model and the new vocabulary words.

16. The computer-readable storage device of claim 15 , the computer-readable storage device having additional instructions stored which result in operations comprising recognizing received speech with the diverse language model.

17. The computer-readable storage device of claim 15 , the computer-readable storage device having additional instructions stored which result in operations comprising updating the visitation policy for the crawler once a crawling threshold is reached, wherein updating the visitation policy is based on an expected perplexity value of the web pages, and wherein the expected perplexity value of a web page is determined by evaluating links to the page.

18. The computer-readable storage device of claim 15 , the computer-readable storage device having additional instructions stored which result in operations comprising merging a set of language models.

19. The computer-readable storage device of claim 15 , wherein the new language model comprises a trigram model built using a language modeling toolkit.

Assignees

• Barbosa Luciano De Andrade
• Bangalore Srinivas
• At & T Ip I Lp

Inventors

• Barbosa Luciano De Andrade
• Bangalore Srinivas

Classifications

• G06F40/205

  Parsing · CPC title

• G06F40/242

  Dictionaries · CPC title

• G06F40/279

  Recognition of textual entities · CPC title

• G06F40/40

  Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) · CPC title

• G10L2015/0633

  using lexical or orthographic knowledge sources · CPC title