Real-time visualization of machine learning models

Patent metadata

Field	Value
Publication number	US-10810491-B1
Application number	US-201615074203-A
Country	US
Kind code	B1
Filing date	Mar 18, 2016
Priority date	Mar 18, 2016
Publication date	Oct 20, 2020
Grant date	Oct 20, 2020

Abstract

A visualization tool for machine learning models obtains metadata from a first training node at which a multi-layer machine learning model is being trained. The metadata includes a parameter of an internal layer of the model. The tool determines a plurality of metrics from the metadata, including respective loss function values corresponding to several training iterations of the model. The tool indicates the loss function values and the internal layer parameter values via a graphical interface.

First claim

What is claimed is:

1. A system, comprising:
a plurality of training nodes of a machine learning service, including a first training node implemented at least in part at a first computing device and a second training node implemented at least in part at a second computing device; and
a visualization manager of the machine learning service, implemented at least in part at a third computing device;
wherein the visualization manager is configured to:
obtain, from the first training node prior to a termination of training of a first neural network model at the first training node, wherein said training comprises using a particular input data set, a first collection of one or more model log entries, wherein a particular model log entry of the first collection indicates a value of a first parameter of an internal layer of the first neural network model, wherein the value of the first parameter is dynamically updated prior to the termination of training of the first neural network;
obtain, from the second training node, a second collection of one or more model log entries pertaining to training of a second neural network model using the particular input data set at the second training node, wherein a particular model log entry of the second collection indicates a value of a second parameter of an internal layer of the second neural network model, and wherein the second neural network model differs from the first neural network model;
determine, from the first collection, a first plurality of metrics associated with the first neural network model, wherein the first plurality of metrics includes a first value of a loss function corresponding to a respective training iteration of the first neural network model, wherein the first value of the loss function is dynamically updated prior to the termination of training of the first neural network;
determine, from the second collection, a second plurality of metrics associated with the second neural network model, wherein the second plurality of metrics includes a second value of the loss function corresponding to a particular training iteration of the second neural network model; and
indicate, via a dynamically updated visualization interface to a client of the machine learning service prior to the termination of training of the first neural network model, (a) the first and second values of the loss function and (b) the values of the first and second parameters.

2. The system as recited in claim 1 , wherein the visualization manager is configured to:
indicate, to the client via the dynamically updated visualization interface, respective quality metrics pertaining to (a) a first prediction generated by the first neural network model with respect to a first test data set, and (b) a second prediction generated by the second neural network model with respect to the first test data set.

3. The system as recited in claim 1 , wherein the visualization manager is configured to:
indicate, to the client via the dynamically updated visualization interface, (a) a representation of a feature processing filter associated with a particular layer of a particular training iteration of the first neural network model and (b) a representation of an output of the feature processing filter.

4. The system as recited in claim 1 , wherein the visualization manager is configured to:
indicate, to the client via the dynamically updated visualization interface, one or more of (a) a first series of gradient values pertaining to a particular layer of the first neural network model, wherein individual gradient values of the first series correspond to respective training iterations, or (b) a second series of input weight values pertaining to the particular layer of the first neural network model, wherein individual input weight values of the second series correspond to respective training iterations.

5. The system as recited in claim 1 , wherein the visualization manager is configured to:
indicate, to the client via the dynamically updated visualization interface, a reduced-dimension mapping of a set of multi-dimensional classification results obtained from the first neural network model, wherein the reduced-dimension mapping is indicative of an extent of an overlap between a first predicted class of the first neural network model and a second predicted class of the first neural network model.

6. A method, comprising:
performing, by a machine learning visualization tool implemented at one or more computing devices:
obtaining, from a first training node prior to a termination of training of a first neural network model, wherein said training comprises using a first input data set, a first collection of one or more model log entries, wherein a particular model log entry of the first collection indicates a value of a first parameter of a hidden layer of the first neural network model;
obtaining, from a second training node, a second collection of one or more model log entries pertaining to a second neural network model, wherein the second neural network model is trained at the second training node using the first input data set, and wherein the second neural network model differs from the first neural network model;
determining, from the first collection, a first plurality of metrics associated with the first neural network model, wherein the first plurality of metrics includes a first value of a loss function corresponding to a particular training iteration of the first neural network model;
determining, from the second collection, a second value of a loss function corresponding to a particular training iteration of the second neural network model;
indicating, via a graphical programmatic interface, (a) the first and second values of the loss function and (b) the value of the first parameter; and
providing an alert or a recommendation as to the training of the first neural network model based on the first collection of one or more model log entries.

7. The method as recited in claim 6 , further comprising performing, by the machine learning visualization tool:
indicating, via the graphical programmatic interface, respective quality scores corresponding to (a) a first prediction generated by the first neural network model with respect to a first test data set, and (b) a second prediction generated by a second neural network model with respect to the first test data set.

8. The method as recited in claim 6 , further comprising performing, by the machine learning visualization tool:
indicating, via the graphical programmatic interface, (a) a representation of a feature processing filter associated with a particular layer of a particular training iteration of the first neural network model and (b) a representation of an output of the feature processing filter.

9. The method as recited in claim 6 , further comprising performing, by the machine learning visualization tool:
indicating, via the graphical programmatic interface, one or more of (a) a series of gradient values pertaining to a particular layer of the first neural network model, wherein individual gradient values correspond to respective training iterations, or (b) a series of input weight values pertaining to the particular layer of the first neural network model, wherein individual input weight values correspond to respective training iterations.

10. The method as recited in claim 6 , wherein the first neural network model comprises a convolutional neural network model.

11. The method as recited in claim 6 , wherein the first neural network model comprises an object recognition model.

12. The method as recited in claim 6 , further comprising performing, by the machine learning visualization tool:

Assignees

• Amazon Tech Inc

Inventors

• Xia Wei
• Wu Weixin
• Wang Meng
• Das Ranju

Classifications

• G06N3/08 Primary

  Learning methods · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title