Systems and methods for removal of attributes from multi-modality and multi-attribute data

Patent metadata

Field	Value
Publication number	US-12153554-B2
Application number	US-202217934935-A
Country	US
Kind code	B2
Filing date	Sep 23, 2022
Priority date	Sep 23, 2022
Publication date	Nov 26, 2024
Grant date	Nov 26, 2024

Abstract

A method for removing uninterested attributes from multi-modality data may include: receiving, by a multi-modality attribute removal computer program executed by an electronic device, multi-modality data comprising a plurality of modalities from a data source, wherein data in each modality are related; receiving, by the multi-modality attribute removal computer program, an uninterested attribute in the multi-modality data to remove; training, by the multi-modality attribute removal computer program, a modality-focused encoder for each modality of the multi-modality data to remove the uninterested attribute using a removal loss and a retention loss for the respective modality; receiving, by the multi-modality attribute removal computer program, a multi-modality data set for processing; and processing, by the multi-modality attribute removal computer program, the multi-modality data set using the modality-focused encoders, wherein the processing results in a processed multi-modality data set with the uninterested attribute removed and one or more interested attribute retained.

First claim

What is claimed is:

1. A method for removing uninterested attributes from multi-modality data, comprising:
receiving, by a multi-modality attribute removal computer program executed by an electronic device, multi-modality training data comprising audio training data for an audio modality and visual training data for a visual modality from a data source, wherein the audio training data and the visual training data are related, and a change in the audio training data causes a change in the visual training data;
receiving, by the multi-modality attribute removal computer program, an identification of an uninterested attribute to remove from the multi-modality training data and an identification of an interested attribute to retain in the multi-modality training data;
training, by the multi-modality attribute removal computer program, a modality-focused encoder for each of the audio modality and the visual modality of the multi-modality data to remove the identified uninterested attribute using a removal loss and a retention loss for the respective modality;
receiving, by the multi-modality attribute removal computer program, a processing multi-modality data set comprising audio data for the audio modality and visual data for the visual modality; and
processing, by the multi-modality attribute removal computer program, the processing multi-modality data set using the modality-focused encoders, wherein the processing results in a processed multi-modality data set with the uninterested attribute removed and the interested attribute retained.

2. The method of claim 1 , wherein the removal loss comprises a cosine distance, and the retention loss comprises a L2 norm.

3. The method of claim 1 , further comprising:
pre-training, by the multi-modality attribute removal computer program, a single modality reidentification classifier for each of the audio modality and the visual modality of the multi-modality data and a multi-modality reidentification classifier for the multi-modality data to remove the uninterested attribute resulting in removal losses and retention losses.

4. The method of claim 3 , wherein the removal losses represent a loss associated with reidentification of the uninterested attribute in outputs of the single modality reidentification classifiers and the multi-modality reidentification classifier, and the retention losses represent a loss of utility of retained data in the outputs of the single modality reidentification classifiers and the multi-modality reidentification classifier.

5. The method of claim 3 , wherein the single modality reidentification classifiers and the multi-modality reidentification classifier are pre-trained by with cross-entropy loss and back propagation with Stochastic Gradient Descent (SGD).

6. The method of claim 1 , wherein the step of training the modality-focused encoder for each of the audio modality and the visual modality of the multi-modality data using the removal loss and the retention loss for the respective modality comprises:
receiving, by the multi-modality attribute removal computer program, a plurality of additional multi-modality data sets;
wherein the modality-focused encoders are trained using the removal loss and the retention loss for the respective modality, the plurality of additional multi-modality data sets, the uninterested attribute, and the interested attribute.

7. A system, comprising:
a multi-modality data source comprising multi-modality training data comprising audio training data for an audio modality and visual training data for a visual modality, wherein the audio training data and the visual training data are related, and a change in the audio training data causes a change in the visual training data; and
an electronic device comprising at least one computer program and executing a multi-modality attribute removal computer program comprising a modality-focused encoder for each of the audio modality and the visual modality in the multi-modality data, a modality reidentification classifier for each of the audio modality and the visual modality in the multi-modality data, and a multi-modality reidentification classifier;
wherein:
the multi-modality attribute removal computer program receives the multi-modality training data from the multi-modality data source;
the multi-modality attribute removal computer program receives an identification of an uninterested attribute in the multi-modality data to remove and an identification of an interested attribute to retain;
the multi-modality attribute removal computer program trains the modality-focused encoders to remove the identified uninterested attribute using a removal loss and a retention loss for the respective modality;
the multi-modality attribute removal computer program receives a processing multi-modality data set; and
the multi-modality attribute removal computer program processes the processing multi-modality data set using the modality-focused encoders, wherein the processing results in a processed multi-modality data set with the uninterested attribute removed and the interested attribute retained.

8. The system of claim 7 , wherein the removal loss comprises a cosine distance, and the retention loss comprises a L2 norm.

9. The system of claim 7 , wherein the multi-modality attribute removal computer program pre-trains a single modality reidentification classifier for each of the audio modality and the visual modality of the multi-modality data and a multi-modality reidentification classifier for the multi-modality data to remove the uninterested attribute resulting in removal losses and retention losses.

10. The system of claim 9 , wherein the removal losses represent a loss associated with reidentification of the uninterested attribute in outputs of the single modality reidentification classifiers and the multi-modality reidentification classifier, and the retention losses represent a loss of utility of retained data in the outputs of the single modality reidentification classifiers and the multi-modality reidentification classifier.

11. The system of claim 9 , wherein the single modality reidentification classifiers and the multi-modality reidentification classifier are pre-trained by with cross-entropy loss and back propagation with Stochastic Gradient Descent (SGD).

12. The system of claim 7 , wherein the modality-focused encoder are trained by:
receiving a plurality of additional multi-modality data sets;
wherein the modality-focused encoders are trained using the removal loss and the retention loss for the respective modality, the plurality of additional multi-modality data sets, the uninterested attribute, and the interested attribute.

13. The system of claim 7 , wherein the uninterested attribute comprises identity.

14. A non-transitory computer readable storage medium, including instructions stored thereon, which when read and executed by one or more computer processors, cause the one or more computer processors to perform steps comprising:
receiving multi-modality training data comprising audio training data for an audio modality and visual training data for a visual modality from a data source, wherein the audio training data and the visual training data are related, and a change in the audio training data causes a change in the visual training data;
receiving an identification of an uninterested attribute to remove from the multi-modality training data and an identification of an interested attribute to retain in the multi-modality training data;
training a modality-focused encoder for each of the audio modality and the visual modality of the multi-modality data to remove the identified u

Assignees

• Jpmorgan Chase Bank Na

Inventors

• Shi Zhonghao
• Chen Richard
• Hu Shaohan
• Moriarty William
• Pistoia Marco

Classifications

• G06F16/285

  Clustering or classification · CPC title

• G06F16/215 Primary

  Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors · CPC title

• G06F16/254 Primary

  Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses · CPC title