Method and system for linking heterogeneous data sources

Patent metadata

Field	Value
Publication number	US-2016180245-A1
Application number	US-201414577220-A
Country	US
Kind code	A1
Filing date	Dec 19, 2014
Priority date	Dec 19, 2014
Publication date	Jun 23, 2016
Grant date	—

Abstract

A method for linking records (related to an entity) from separate databases may include extracting a first record from a first database as a first vector, extracting a second record from a second database as a second vector, generating first and second sub-vectors for the first and second vectors, where each sub-vector includes quality features from the respective vector, pre-processing the first and second sub-vectors using domain knowledge, calculating a distance assessment classifier based on the first and second sub-vectors, and determining whether the distance represented by the distance assessment classifier is greater than a threshold. If the distance is greater than the threshold, the records may be linked; if not, the method extracts additional records and repeats after generating first and second sub-vectors until the distance is greater than the threshold. A system for linking records is also disclosed.

First claim

1 . A method for linking records from separate databases, the records related to an entity, the method comprising:
extracting a first record from a first database as a first vector; extracting a second record from a second database as a second vector; generating first and second sub-vectors for the first and second vectors, each sub-vector comprising quality features from the respective vector, pre-processing the first and second sub-vectors using domain knowledge; calculating a distance assessment classifier based on the first and second sub-vectors; determining whether the distance represented by the distance assessment classifier is greater than a threshold; if the distance is greater than the threshold, linking the records; and if the distance is not greater than the threshold, extracting additional records and returning to the generating first and second sub-vector operation until the distance is greater than the threshold.

2 . The method of claim 1 , wherein the pre-processing comprises cleaning or normalizing extracted values of the sub-vectors.

3 . The method of claim 2 , wherein the cleaning comprises at least one of making all letters uppercase, removing trailing spaces, removing hyphens or dashes, converting literals to numbers, and converting numbers to literals.

4 . The method of claim 1 , wherein the quality features comprise a social security number.

5 . The method of claim 1 , wherein the quality features comprise a compound used in a clinical setting.

6 . The method of claim 1 , wherein the quality features comprise an email address.

7 . The method of claim 1 , wherein the distance represented by the distance assessment classifier may be calculated as a sum of atomic probabilistic distance metrics.

8 . The method of claim 7 , wherein each atomic probabilistic distance metric is an algorithmic weighted quality feature.

9 . The method of claim 1 , further comprising eliminating unnecessary comparisons of distance assessment classifiers.

10 . The method of claim 1 , further comprising de-duplicating the distance assessment classifiers wherein those classifiers with close probabilities are not recorded as separate output entities.

11 . A system for linking records from separate clinical trial databases, comprising:
an enterprise information integration subsystem for integrating several database schemas into one federated schema; a data cleaner for parsing and cleaning up the data; a data normalizer and labeler for normalizing and labeling the cleansed data by standardizing lexical variations and ontological concepts; a feature vector builder for building features that map data in a finite dimensional space for comparison and separation; an entity classifier configured to resolve entities in the finite dimensional space using probabilistic matching; an entity clusterer for grouping data based on similarity in the finite dimensional space built by the feature vector builder; an application programming interface to interact with the data produced by the entity classifier and entity clusterer; and a linked database having a harmonized schema for presenting the resolved records.

12 . The system of claim 11 , wherein the enterprise information integration subsystem provides a database connectivity programming interface to one or more databases.

13 . The system of claim 11 , wherein the enterprise information integration subsystem supports automatic optimization of SQL queries.

14 . The system of claim 11 , wherein the application programming interface (API) is a REST-ful API.

15 . The system of claim 11 , wherein the probabilistic matching comprises calculating a distance between two records.

16 . A method for automatically selecting parameters for use in linking database records, comprising:
calculating a cutoff threshold by executing an entity resolution algorithm at least two times and observing the number of matches produced; determining, using domain knowledge, components of each record to be included in a sub-vector; determining atomic comparators; calculating a field quality value for each component of the sub-vector; selecting vectors from input data to generate training sets based on distances between vectors calculated as a sum of distances between components of sub-vectors multiplied by the field quality values; sampling the selected vectors against the training sets; and classifying the vectors binarily using the training sets.

17 . The method of claim 16 , wherein the entity resolution algorithm comprises the Jaro-Winkler distance metric.

18 . The method of claim 16 , wherein the entity resolution algorithm is selected from a genetic, harmony, or machine-learning algorithm.

19 . The method of claim 16 , wherein the cutoff threshold may be calculated when the number of matches in an algorithmic curve is stable in relation to changes in the cutoff threshold value.

20 . The method of claim 16 , wherein determining atomic comparators and calculating field quality values comprises finding a multi-parametric area in an algorithmic curve in which the number of matches does not substantially change in relation to changes in the field quality values and atomic comparators.

Assignees

• Medidata Solutions Inc

Inventors

• Tereshkov Vladimir
• Haider Syed
• Aimale Valerio
• Hartman Joshua
• Bound Christopher
• Katriel Ron

Classifications

• G06F17/30324

  Physics · mapped topic

• G06N99/005 Primary

  Physics · mapped topic

• G06F17/30687

  Physics · mapped topic

• G06F17/30598

  Physics · mapped topic

• G06F17/30563

  Physics · mapped topic