Automatic NER dictionary generation from structured business data

Patent metadata

Field	Value
Publication number	US-9959304-B2
Application number	US-201514697855-A
Country	US
Kind code	B2
Filing date	Apr 28, 2015
Priority date	Mar 11, 2015
Publication date	May 1, 2018
Grant date	May 1, 2018

Abstract

The present disclosure involves systems, software, and computer implemented methods for generating NER dictionaries from structured business data. In one example, a method may include identifying a set of structured data for use in generating a NER dictionary, identifying a particular collection within the identified structured data to use in the dictionary generation, the particular identified collection including a plurality of values, identifying at least one variant of the value for each value in the particular identified collection, each of the at least one variants of the value including at least a portion of the underlying value, determining a set of unique variants from the identified at least one variants across each of the values in the particular identified subset, adding the determined set of unique variants to the generated dictionary, each unique variant added to the generated dictionary being associated with the value from which the unique variant was derived.

First claim

What is claimed is:

1. A non-transitory computer storage medium encoded with a computer program, the program comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
identifying a set of structured data for use in generating a dictionary for a named entity extraction process;
identifying a particular collection within the identified structured data to use in the dictionary generation, wherein the particular identified collection includes a plurality of values;
for each value in the particular identified collection, identifying at least one variant of the value, wherein each of the at least one variant of the value includes at least a portion of the underlying value, wherein the value includes at least one delimiter, and wherein identifying the at least one variant of the value includes:
identifying a first variant based on the full value;
identifying a second variant by removing a first suffix from the full value to the first delimiter; and
identifying additional variants by removing additional suffixes for each additional delimiter;

determining a set of unique variants from the identified at least one variant across each of the values in the particular identified subset; and
adding the determined set of unique variants to the generated dictionary, wherein each unique variant added to the generated dictionary is associated with the value from which the unique variant was derived.

2. The medium of claim 1 , wherein the particular collection is associated with an object type, and wherein each unique variant added to the generated dictionary is associated with the object type of the particular collection.

3. The medium of claim 1 , wherein each value within the collection is associated with a unique identifier, and wherein each unique variant added to the generated dictionary is associated with the unique identifier of the value from which the unique identifier was derived.

4. The medium of claim 1 , wherein identifying at least one variant of the particular value includes identifying an acronym of the full value as a variant.

5. The medium of claim 1 , wherein determining the set of unique variants from the identified at least one variant across each of the values in the particular identified subset includes:
for each identified at least one variant:
determining whether a prior entry in a list of identified at least one variant exists;
in response to determining that a prior entry of the at least one variant does not exist, adding the at least one variant to the list of identified at least one variant, wherein the added at least one variant is associated with a counter set to a default value;
in response to determining that a prior entry of the at least one variant exists, incrementing a counter associated with the prior entry; and

removing any of the at least one variant from the list of identified at least one variant with an associated counter at any value other than the default value, wherein the list of identified at least one variant with associated counters at the default value represent the set of unique variants.

6. The medium of claim 1 , the operations further comprising removing common words and phrases from the determined set of unique variants prior to adding the determined set of unique variants to the generated dictionary.

7. The medium of claim 1 , wherein identifying the particular collection within the identified structured data to use in the dictionary generation includes identifying at least one additional collection within the structured data as a set of information related to the particular collection, the operations further comprising:
for each value in the identified at least one collection:
identifying at least one variant of the value, wherein each of the at least one variant of the value includes at least a portion of the underlying value; and
identifying, for the value, an association to a particular value from the particular collection in which a relationship is defined within the structured data;

determining at least one additional set of unique variants from the identified at least one variant across each of the values in the identified at least one additional collection;
adding the determined at least one additional set of unique variants to the generated dictionary, wherein each unique variant added to the generated dictionary from the at least one additional set of unique variants is associated with the particular value from the particular collection in which the relationship is defined within the structured data.

8. A computerized method performed by one or more processors, the method comprising:
identifying a set of structured data for use in generating a dictionary for a named entity extraction process;
identifying a particular collection within the identified structured data to use in the dictionary generation, wherein the particular identified collection includes a plurality of values;
for each value in the particular identified collection, identifying at least one variant of the value, wherein each of the at least one variant of the value includes at least a portion of the underlying value, wherein the value includes at least one delimiter, and wherein identifying the at least one variant of the value includes:
identifying a first variant based on the full value;
identifying a second variant by removing a first suffix from the full value to the first delimiter; and
identifying additional variants by removing additional suffixes for each additional delimiter;

determining a set of unique variants from the identified at least one variant across each of the values in the particular identified subset; and
adding the determined set of unique variants to the generated dictionary, wherein each unique variant added to the generated dictionary is associated with the value from which the unique variant was derived.

9. The method of claim 8 , wherein the particular collection is associated with an object type, and wherein each unique variant added to the generated dictionary is associated with the object type of the particular collection.

10. The method of claim 8 , wherein each value within the collection is associated with a unique identifier, and wherein each unique variant added to the generated dictionary is associated with the unique identifier of the value from which the unique identifier was derived.

11. The method of claim 8 , wherein identifying at least one variant of the particular value includes identifying an acronym of the full value as a variant.

12. The method of claim 8 , wherein determining the set of unique variants from the identified at least one variant across each of the values in the particular identified subset includes:
for each identified at least one variant:
determining whether a prior entry in a list of identified at least one variant exists;
in response to determining that a prior entry of the at least one variant does not exist, adding the at least one variant to the list of identified at least one variant, wherein the added at least one variant is associated with a counter set to a default value;
in response to determining that a prior entry of the at least one variant exists, incrementing a counter associated with the prior entry; and

removing any of the at least one variant from the list of identified at least one variant with an associated counter at any value other than the default value, wherein the list of identified at least one variant with associated counters at the default value represent the set of unique variants.

13. The method of claim 8 , further comprising removing c

Assignees

• Nathan Eyal
• Sap Portals Israel Ltd

Inventors

• Nathan Eyal

Classifications

• G06F2221/2113

  Multi-level security, e.g. mandatory access control · CPC title

• G06F16/2228 Primary

  Indexing structures · CPC title

• G06F16/637

  Administration of user profiles, e.g. generation, initialization, adaptation or distribution · CPC title

• G06F16/3344

  using natural language analysis · CPC title

• G06F16/20

  of structured data, e.g. relational data · CPC title