Automated evaluation of neural networks using trained classifier

Patent metadata

Field	Value
Publication number	US-10410111-B2
Application number	US-201715793866-A
Country	US
Kind code	B2
Filing date	Oct 25, 2017
Priority date	Oct 25, 2017
Publication date	Sep 10, 2019
Grant date	Sep 10, 2019

Abstract

A computer system includes a memory storing a data structure representing a neural network. The data structure includes a plurality of fields including values representing topology of the neural network. The computer system also includes one or more processors configured to perform neural network classification by operations including generating a vector representing at least a portion of the neural network based on the data structure. The operations also include providing the vector as input to a trained classifier to generate a classification result associated with at least the portion of the neural network, where the classification result is indicative of expected performance or reliability of the neural network. The operations also include generating an output indicative of the classification result.

First claim

What is claimed is:

1. A computer system comprising:
a memory storing a data structure representing a neural network, the data structure including a plurality of fields including values representing topology of the neural network, the plurality of fields descriptive of a first node of the neural network, a second node of the neural network, and a link between the first node and second node; and
one or more processors configured to perform neural network classification by operations including:
generating, based on the data structure, a vector representing at least a portion of the neural network;
providing the vector as input to a trained classifier to generate a classification result associated with at least the portion of the neural network, the classification result indicative of expected performance or expected accuracy of the neural network;
generating an output indicative of the classification result;
generating an updated trained classifier based on a training data entry associated with the classification result; and
providing a second vector as input to the updated trained classifier to generate a second classification result associated with at least a portion of a second neural network, the second classification result indicative of expected performance or expected accuracy of the second neural network.

2. The computer system of claim 1 , wherein the operations further include storing the classification result and the vector as the training data entry in the memory.

3. The computer system of claim 1 , wherein generating the vector comprises:
identifying whether the neural network includes each feature of a plurality of features;
generating a plurality of values, each value of the plurality of values identifying whether a particular feature of the plurality of features is included in the neural network; and
combining the plurality of values to generate the vector.

4. The computer system of claim 3 , wherein a data length of the vector is independent of whether the plurality of features are included in the neural network.

5. The computer system of claim 1 , wherein the updated trained classifier is further based on training data entries associated with one or more other neural networks include first training data associated with a first neural network that is configured to analyze a first type of data and second training data associated with a second neural network that is configured to analyze a second type of data, wherein the first type of data is different from the second type of data.

6. The computer system of claim 1 , wherein the operations further include initiating training of the neural network in response to the classification result classifying the neural network as inaccurate.

7. The computer system of claim 1 , wherein the operations further include evaluating the vector using one or more heuristic rules, wherein the vector is provided as input to the trained classifier based on a determination that the vector satisfies the one or more heuristic rules.

8. The computer system of claim 1 , wherein the classification result is further indicative of a particular type of data with which the neural network is associated.

9. The computer system of claim 1 , wherein the classification result is independent of a type of data that the neural network is configured to operate on and independent of a type of result the neural network is configured to produce.

10. A method comprising:
accessing, by a processor, a data structure representing a neural network, the data structure including a plurality of fields including values representing topology of the neural network, the plurality of fields descriptive of a first node of the neural network, a second node of the neural network, and a link between the first node and second node;
generating, by the processor based on the data structure, a vector representing at least a portion of the neural network;
providing, by the processor, the vector as input to a trained classifier to generate a classification result associated with at least the portion of the neural network, the classification result indicative of expected performance or expected accuracy of the neural network;
generating, by the processor, an output indicative of the classification result;
generating an updated trained classifier based on a training data entry associated with the classification result; and
providing a second vector as input to the updated trained classifier to generate a second classification result associated with at least a portion of a second neural network, the second classification result indicative of expected performance or expected accuracy of the second neural network.

11. The method of claim 10 , further comprising generating, by the processor based on the data structure, a vector representation of at least the portion of the neural network, wherein the vector is generated based on the vector representation of at least the portion of the neural network.

12. The method of claim 10 , wherein the vector is a normalized vector including a set of values arranged based on a sequence of input nodes of the trained classifier.

13. The method of claim 10 , wherein the vector includes one or more values indicating a topology of the neural network.

14. The method of claim 10 , wherein the classification result indicates whether the neural network has been trained.

15. The method of claim 10 , wherein the classification result includes an accuracy classification.

16. The method of claim 15 , further comprising sending a signal to cause the neural network to be trained in response to the accuracy classification classifying the neural network as inaccurate.

17. The method of claim 10 , further comprising, before providing the vector as input to the trained classifier, evaluating the vector using one or more heuristic rules, wherein the vector is provided as input to the trained classifier based on a determination that the vector satisfies the one or more heuristic rules.

18. The method of claim 17 , wherein the one or more heuristic rules specify patterns in vectors of neural networks that are not expected to provide accurate results.

19. The method of claim 18 , wherein a particular pattern of the patterns includes a sequence of repeating values.

20. The method of claim 10 , wherein the trained classifier includes a decision tree, a support vector machine, or a third neural network.

21. A computer-readable storage device storing instructions that are executable by a processor to cause a computer system to perform operations comprising:
accessing a data structure representing a neural network, the data structure including a plurality of fields including values representing topology of the neural network, the plurality of fields descriptive of a first node of the neural network, a second node of the neural network, and a link between the first node and second node;
generating, based on the data structure, a vector representing at least a portion of the neural network;
providing the vector as input to a trained classifier to generate a classification result associated with at least the portion of the neural network, the classification result indicative of expected performance or expected accuracy of the neural network;
generating an output indicative of the classification result; and
initiating training of the neural network in response to the classification result.

22. The computer-readable storage device of claim 21 , wherein the ope

Assignees

• Sparkcognition Inc

Inventors

• Husain Syed Mohammad Amir

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N3/048

  Activation functions · CPC title

• G06N5/01

  Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound · CPC title

• G06N20/10

  using kernel methods, e.g. support vector machines [SVM] · CPC title

• G06N3/084 Primary

  Backpropagation, e.g. using gradient descent · CPC title