Evaluation method, evaluation apparatus, and non-transitory computer-readable recording medium storing evaluation program

Patent metadata

Field	Value
Publication number	US-2023222385-A1
Application number	US-202318174973-A
Country	US
Kind code	A1
Filing date	Feb 27, 2023
Priority date	Oct 8, 2020
Publication date	Jul 13, 2023
Grant date	—

Abstract

An evaluation method executed by a computer, the evaluation method comprising processing of: generating, based on information that indicates a degree of reduction of inference accuracy of a machine learning model to a change in first training data, second training data that reduces the inference accuracy; training the machine learning model by using the second training data; and evaluating the trained machine learning model.

First claim

What is claimed is:

1 . An evaluation method executed by a computer, the evaluation method comprising processing of:
generating, based on information that indicates a degree of reduction of inference accuracy of a machine learning model to a change in first training data, second training data that reduces the inference accuracy; training the machine learning model by using the second training data; and evaluating the trained machine learning model.

2 . The evaluation method according to claim 1 , wherein
the generating of the second training data includes: randomly selecting data as an initial point from clusters of all labels of the first training data; adding, to the initial point, data obtained by assigning one or a plurality of labels different from an original label to each piece of the selected data; adding, to the initial point, data obtained by pairing data with different labels with each other; and generating the second training data based on the initial point.

3 . The evaluation method according to claim 2 , wherein
the generating of the second training data includes generating a plurality of pieces of the second training data based on a plurality of the initial points, the training of the machine learning model includes training the machine learning model by using each piece of the plurality of second training data, and the evaluating of the trained machine learning model includes evaluating each of a plurality of the trained machine learning models trained by using each piece of the plurality of second training data.

4 . The evaluation method according to claim 2 , wherein
the generating of the second training data based on the initial point includes: updating the initial point by a gradient ascent method; and generating the second training data based on the updated initial point.

5 . The evaluation method according to claim 4 , wherein
the generating of the second training data based on the initial point includes: updating a label assigned to the initial point by the gradient ascent method; and generating the second training data based on the updated initial point and label.

6 . The evaluation method according to claim 1 , wherein
the evaluating of the trained machine learning model includes: calculating, by using a function that calculates a change amount of a loss function, a first accuracy difference of the inference accuracy between the machine learning model trained by using the second training data and the machine learning model trained by using the first training data; and evaluating the trained machine learning models based on the first accuracy difference.

7 . The evaluation method executed by the computer according to claim 6 , the evaluation method further comprising:
calculating, by using the loss function, a second accuracy difference of the inference accuracy between the machine learning model trained by using the first training data and the machine learning model trained by using the second training data; replacing, in a case where a difference between the first accuracy difference and the second accuracy difference is a predetermined threshold or more, the first training data with the second training data to generate fourth training data that reduces the inference accuracy; training the machine learning model by using the fourth training data; and evaluating the machine learning model trained by using the fourth training data.

8 . An evaluation apparatus comprising:
a memory; and a processor coupled to the memory, the processor being configured to perform processing including: generating, based on information that indicates a degree of reduction of inference accuracy of a machine learning model to a change in first training data, second training data that reduces the inference accuracy; training the machine learning model by using the second training data; and evaluating the trained machine learning model.

9 . A non-transitory computer-readable recording medium storing an evaluation program for causing a computer to perform processing including:
generating, based on information that indicates a degree of reduction of inference accuracy of a machine learning model to a change in first training data, second training data that reduces the inference accuracy; training the machine learning model by using the second training data; and evaluating the trained machine learning model.

Assignees

• Fujitsu Ltd

Inventors

• Shimizu Toshiya
• Higuchi Yuji

Classifications

• G06N20/00 Primary

  Machine learning · CPC title