User interface for operating artificial intelligence experiments

Patent metadata

Field	Value
Publication number	US-12017148-B2
Application number	US-202217804716-A
Country	US
Kind code	B2
Filing date	May 31, 2022
Priority date	May 31, 2022
Publication date	Jun 25, 2024
Grant date	Jun 25, 2024

Abstract

A user interface (UI), for analyzing model training runs, tracking and visualizing various aspects of machine learning experiments, can be used when training an artificial intelligent agent in, for example, a racing game environment. The UI can be web-based and can allow researchers to easily see the status of their experiments. The UI can include an experiment synchronized event viewer that can synchronizes visualizations, videos, and timeline/metrics graphs in the experiment. This viewer allows researchers to see how experiments unfold in great detail. The UI can further include experiment event annotations that can generate event annotations. These annotations can be displayed via the synchronized event viewer. The UI can be used to consider consolidated results across experiments and can further consider videos. For example, the UI can provide a reusable dashboard that can capture and compare metrics across multiple experiments.

First claim

The invention claimed is:

1. A method for providing a user interface for analyzing model training runs and tracking and visualizing aspects of machine learning experiments, the method comprising:
displaying a timeline of a selected metric of the machine learning experiments;
displaying resources used across multiple cloud environments for running the machine learning experiments;
displaying a video synced to a selected portion of the timeline;
displaying a visualizer that shows a global representation from which data is gathered during the machine learning experiments, wherein the visualizer includes one or more event annotations for one or more key events from the machine learning experiments; and
displaying an experiment dashboard page with one or more dashboards, wherein the one or more dashboards are user-created dashboards that are applicable to a single experiment or to multiple experiments, wherein the one or more dashboards capture and compare metrics across multiple experiments, wherein
the resources include one or more of a number of graphics processing units (GPUs), a number of central processing units (CPUs) on GPU machines, memory resources on GPU machines, a number of CPUs on non GPU machines, and memory resources on non GPU machines.

2. The method of claim 1 , wherein the video is generated by a simulated application being run by a trainer or a data gatherer.

3. The method of claim 1 , wherein the video is a video feed when communicating from a cloud-based game console.

4. The method of claim 1 , wherein the video includes information from multiple video sources weaved together into a single coherent video stream.

5. The method of claim 1 , wherein the machine learning experiment is in a racing game environment.

6. The method of claim 5 , wherein the video shows a vehicle video from a selected vehicle participating in the racing game.

7. The method of claim 5 , wherein the visualizer shows a representation of a racing course, each racing vehicle, and a trajectory of each racing vehicle.

8. The method of claim 5 , wherein the timeline shows vehicle positions in the racing game environment.

9. The method of claim 1 , wherein the one or more key events include racing game key events including one or more of position gained, position lost, or car is off course.

10. A user interface of a machine learning training system computing architecture for training a racing game artificial agent, the user interface comprising:
displaying a timeline of a selected metric of a machine learning experiment;
displaying resources used across multiple cloud environments for running the machine learning experiment;
displaying a video synced to a selected portion of the timeline;
displaying a visualizer shows a global representation from which data is gathered during the machine learning experiment;
displaying one or more event annotations for one or more key events from the machine learning experiment on the visualizer; and
displaying an experiment dashboard page with one or more dashboards, wherein the one or more dashboards are user-created dashboards that are applicable to a single experiment or to multiple experiments, wherein the one or more dashboards capture and compare metrics across multiple experiments, wherein
the resources include one or more of a number of graphics processing units (GPUs), a number of central processing units (CPUs) on GPU machines, memory resources on GPU machines, a number of CPUs on non GPU machines, and memory resources on non GPU machines.

Assignees

• Sony Group Corp
• Sony Corp America

Inventors

• Douglas Rory
• Whitehead Dion
• Barrett Leon
• Khandelwal Piyush
• Walsh Thomas
• Barrett Samuel
• Subramanian Kaushik
• Macglashan James
• Gilpin Leilani
• Wurman Peter

Classifications

• G06N20/00

  Machine learning · CPC title

• A63F13/355

  Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client · CPC title

• A63F13/803 Primary

  Driving vehicles or craft, e.g. cars, airplanes, ships, robots or tanks · CPC title

• A63F13/55

  Controlling game characters or game objects based on the game progress · CPC title

• A63F13/60 Primary

  Generating or modifying game content before or while executing the game program, e.g. authoring tools specially adapted for game development or game-integrated level editor · CPC title