Systems and methods for providing an automated testing pipeline for neural network models

Patent metadata

Field	Value
Publication number	US-12197317-B2
Application number	US-202318156323-A
Country	US
Kind code	B2
Filing date	Jan 18, 2023
Priority date	Jan 18, 2023
Publication date	Jan 14, 2025
Grant date	Jan 14, 2025

Abstract

Embodiments described herein provide an automated testing pipeline for providing a testing dataset for testing a trained neural network model trained using a first training dataset. A first testing dataset for the trained neural network including a first plurality of user queries is received. A dependency parser is used to filter the first plurality of user queries based on one or more action verbs. A pretrained language model is used to rank the remaining user queries based on respective relationships with queries in the first training dataset. Further, user queries that are classified as keyword matches with the queries in the first training dataset using a bag of words classifier are removed. A second testing dataset is generated using the ranked remaining user queries. Testing outputs are generated, by the trained neural network model, using the second testing dataset.

First claim

What is claimed is:

1. A method of generating a testing dataset for a neural network model, the method comprising:
providing the neural network model trained using a first training dataset;
receiving, via a data interface, a first testing dataset for the trained neural network model including a first plurality of user queries;
generating, by a dependency parser, a second plurality of user queries by filtering the first plurality of user queries based on one or more action verbs;
generating, by a first pretrained language model (PLM), PLM classifications and rankings for the second plurality of user queries based on respective relationships with queries in the first training dataset;
generating, by a bag of words (BoW) classifier, BoW classifications for the second plurality of user queries using keyword matches with the user queries in the first training dataset;
generating a second testing dataset based on the PLM classifications and rankings and BoW classifications; and
generating, by the trained neural network model, testing outputs using the second testing dataset.

2. The method of claim 1 , wherein the generating the second testing dataset based on the PLM classifications and rankings and BoW classifications includes:
removing, from the second testing dataset, a third plurality of user queries, wherein the removed user queries have PLM classifications that are the same as the BoW classifications; and
generating the second testing dataset based on rankings of the remaining user queries.

3. The method of claim 2 , wherein the remaining user queries are ranked based on confidence scores of the classifications of the user queries provided by the first pretrained language model, and
wherein a first user query with a lower confidence score is ranked with a higher priority than a second user query with a higher confidence score.

4. The method of claim 1 , further comprising:
finetuning the first pretrained language model by training the first pretrained language model using the first training dataset.

5. The method of claim 1 , further comprising:
finetuning the bag of word classifier by training the bag of word classifier using the first training dataset.

6. The method of claim 1 , further comprising:
receiving an updated first training dataset generated by updating the first training dataset based on the testing outputs; and
training the neural network model using the updated first training dataset.

7. The method of claim 1 , further comprising:
generating, by a second pretrained language model, classifications and rankings for the second plurality of user queries; and
ranking the second plurality of user queries by combining the rankings generated by the first pretrained language model and the second pretrained language model.

8. A system for providing a testing dataset for a neural network model, the system comprising:
a memory that stores an automated testing pipeline model and a plurality of processor-executable instructions;
a communication interface that receives a first testing dataset including a first plurality of user queries for a trained neural network trained using a first training dataset; and
one or more hardware processors that read and execute the plurality of processor-executable instructions from the memory to perform operations comprising:
generating, by a dependency parser, a second plurality of user queries by filtering the first plurality of user queries based on one or more action verbs;
generating, by a first pretrained language model (PLM), PLM classifications and rankings for the second plurality of user queries based on respective relationships with queries in the first training dataset;
generating, by a bag of words (BoW) classifier, BoW classifications for the second plurality of user queries using keyword matches with the user queries in the first training dataset;
generating a second testing dataset based on the PLM classifications and rankings and BoW classifications; and
generating, by the trained neural network model, testing outputs using the second testing dataset.

9. The system of claim 8 , wherein the generating the second testing dataset based on the PLM classifications and rankings and BoW classifications includes:
removing, from the second testing dataset, a third plurality of user queries, wherein the removed user queries have PLM classifications that are the same as the BoW classifications; and
generating the second testing dataset based on rankings of the remaining user queries.

10. The system of claim 9 , wherein the remaining user queries are ranked based on confidence scores of the classifications of the user queries provided by the first pretrained language model, and
wherein a first user query with a lower confidence score is ranked with a higher priority than a second user query with a higher confidence score.

11. The system of claim 8 , wherein the operations further comprise:
finetuning the first pretrained language model by training the first pretrained language model using the first training dataset.

12. The system of claim 8 , wherein the operations further comprise:
finetuning the bag of word classifier by training the bag of word classifier using the first training dataset.

13. The system of claim 8 , wherein the operations further comprise:
receiving an updated first training dataset generated by updating the first training dataset based on the testing outputs; and
training the neural network model using the updated first training dataset.

14. The system of claim 8 , wherein the operations further comprise:
generating, by a second pretrained language model, classifications and rankings for the second plurality of user queries; and
ranking the second plurality of user queries by combining the rankings generated by the first pretrained language model and the second pretrained language model.

15. A non-transitory machine-readable medium comprising a plurality of machine-executable instructions which, when executed by one or more processors, are adapted to cause the one or more processors to perform operations comprising:
providing a neural network model trained using a first training dataset;
receiving, via a data interface, a first testing dataset for the trained neural network including a first plurality of user queries;
generating, by a dependency parser, a second plurality of user queries by filtering the first plurality of user queries based on one or more action verbs;
generating, by a first pretrained language model (PLM), PLM classifications and rankings for the second plurality of user queries based on respective relationships with queries in the first training dataset;
generating, by a bag of words (BoW) classifier, BoW classifications for the second plurality of user queries using keyword matches with the user queries in the first training dataset;
generating a second testing dataset based on the PLM classifications and rankings and BoW classifications; and
generating, by the trained neural network model, testing outputs using the second testing dataset.

16. The non-transitory machine-readable medium of claim 15 , wherein the generating the second testing dataset based on the PLM classifications and rankings and BoW classifications includes:
removing, from the second testing dataset, a third plurality of user queries, wherein the removed user queries have PLM classifications that are the same as the BoW classifications; and
generating the second testing dataset based on rankings of the remaining user queries.

17. The non-transitory machine-rea

Assignees

• Salesforce Inc

Inventors

• Pentyala Shiva Kumar
• Harinath Shashank
• Asur Sitaram
• Alexander Zachary

Classifications

• G06F11/3684

  for test design, e.g. generating new test cases · CPC title

• G06F11/3688 Primary

  for test execution, e.g. scheduling of test suites · CPC title