System and method for generating a synthetic dataset from an original dataset

Patent metadata

Field	Value
Publication number	US-11640446-B2
Application number	US-202117407181-A
Country	US
Kind code	B2
Filing date	Aug 19, 2021
Priority date	Aug 19, 2021
Publication date	May 2, 2023
Grant date	May 2, 2023

Abstract

A method for generating a synthetic dataset from an original dataset includes encoding categorical features of the original dataset, embedding the encoded dataset in a low-dimensional space, selecting a seed record from the embedded dataset, identifying a plurality of nearest neighbor records to the seed record, generating a new record by randomly selecting features from the plurality of nearest neighbor records, and concatenating the new record into the synthetic dataset. For a synthetic dataset that contains N records, which may be the same as or different from the number of records in the original dataset, the selecting, identifying, generating, and concatenating operations operate a total of N times on the records in the embedded dataset.

First claim

The invention claimed is:

1. A method for generating a synthetic dataset comprising N records from an original dataset, wherein each record comprises categorical features, the method comprising:
encoding the categorical features of the records from the original dataset;
embedding the encoded records in a low-dimensional space;
selecting a seed record from the encoded and embedded records;
identifying a plurality of nearest neighbor records to the selected seed record;
generating a new record by randomly selecting features from the plurality of nearest neighbor records;
concatenating the generated new record into the synthetic dataset;
selecting a new seed record; and
repeating the identifying, generating, concatenating, and selecting operations N-1 times for each newly selected seed record in the encoded and embedded records.

2. The method of claim 1 , wherein encoding categorical features of the original dataset comprises converting categorical values to numeric values.

3. The method of claim 1 , wherein the embedding comprises using t-stochastic neighbor embedding.

4. The method of claim 1 , wherein the embedding comprises using uniform manifold approximation and projection.

5. The method of claim 1 , wherein the embedding comprises using principal component analysis.

6. The method of claim 1 , wherein the original dataset comprises a mixture of categorical and numerical features.

7. The method of claim 1 , further comprising adding noise to the numeric features to generate different feature values.

8. The method of claim 1 , wherein the original dataset contains n records and N=n.

9. The method of claim 1 , wherein the original dataset contains n records and N≠n.

10. The method of claim 1 , wherein highly correlated features are co-segregated in the new record.

11. A system for generating a synthetic dataset comprising N records from an original dataset, wherein each record comprises categorical features, the system comprising:
an encoder for encoding the categorical features of the records from the original dataset;
an embedder for embedding the encoded records in a low-dimensional space;
a clusterer for selecting a seed record from the encoded and embedded records and identifying a plurality of nearest neighbor records to the selected seed record; and
a synthetic record generator that generates a new record by randomly selecting features from the plurality of nearest neighbor records and concatenates the generated new record into the synthetic dataset,
wherein the selecting, identifying, generating, and concatenating operations are carried out N times for each newly selected seed record in the encoded and embedded records.

12. The system of claim 11 , wherein encoding categorical features of the original dataset comprises converting categorical values to numeric values.

13. The system of claim 11 , wherein the embedding comprises using t-stochastic neighbor embedding.

14. The system of claim 11 , wherein the embedding comprises using uniform manifold approximation and projection.

15. The system of claim 11 , wherein the embedding comprises using principal component analysis.

16. The system of claim 11 , wherein the original dataset comprises a mixture of categorical and numerical features.

17. The system of claim 11 , wherein noise is added to the numeric features to generate different feature values.

18. The system of claim 11 , wherein the original dataset contains n records and N=n.

19. The system of claim 11 , wherein the original dataset contains n records and N≠n.

20. The system of claim 11 , further comprising a feature pairing detector for co-segregating highly correlated features in the new record.

Assignees

• Medidata Solutions Inc

Inventors

• Beigi Mandis
• Aptekar Jacob
• Shafquat Afrah
• Mezey Jason

Classifications

• G06F21/6245 Primary

  Protecting personal data, e.g. for financial or medical purposes · CPC title

• G06F16/278

  Data partitioning, e.g. horizontal or vertical partitioning · CPC title

• G06F18/24147

  Distances to closest patterns, e.g. nearest neighbour classification · CPC title

• G06F18/2163

  Partitioning the feature space · CPC title

• G06F18/2148 Primary

  characterised by the process organisation or structure, e.g. boosting cascade · CPC title