Methods for controlling use of computing resources, such as virtual game consoles

Patent metadata

Field	Value
Publication number	US-11745109-B2
Application number	US-202217650287-A
Country	US
Kind code	B2
Filing date	Feb 8, 2022
Priority date	Feb 8, 2022
Publication date	Sep 5, 2023
Grant date	Sep 5, 2023

Abstract

An artificial intelligent agent can act as a player in a video game, such as a racing video game. The game can be completely external to the agent and can run in real time. In this way, the training system is much more like a real world system. The consoles on which the game runs for training the agent are provided in a cloud computing environment. The agents and the trainers can run on other computing devices in the cloud, where the system can choose the trainers and agent compute based on proximity to console, for example. Users can choose the game they want to run and submit code which can be built and deployed to the cloud system. A resource management service can monitor game console resources between human users and research usage and identify experiments for suspension to ensure enough game consoles for human users.

First claim

The invention claimed is:

1. A method for managing computing resources between human users and research users to ensure a sufficient amount of the computing resources for the human users, the method comprising:
providing a resource management service for the computing resources in each of one or more could-based environments, the resource management service including:
a first resource management module measuring load due to human activity in each of the one or more cloud-based environments;
a second resource management module predicting future load due to human activity in each of the one or more cloud-based environments; and
a third resource management module determining the computing resources given to the research users in each of the one or more cloud-based environments; and

providing a resource control service for the computing resources in each of the one or more cloud-based environments, the resource control service including:
a first resource control module reading a number of computing resources available; and
a second resource control module identifying one or more experiments to suspend;

moving the one or more experiments identified to suspend into a <suspending> state;
terminating, by the resource control service in each of the one or more cloud-based environments, processes under control of the resource control service; and
saving state information of a trainer running the terminated processes to a remote storage.

2. The method of claim 1 , wherein the computing resources are cloud-based game consoles.

3. The method of claim 1 , further comprising determining the one or more experiments to suspend based on at least one of a location of the computing resources in use by the one or more experiments, a priority level of the one or more experiments, an age of the one or more experiments, and/or a user ID of the one or more experiments.

4. The method of claim 1 , further comprising assigning the resource control service at each environment containing the computing resources, wherein each resource control service monitors for the one or more experiments in the <suspending> state.

5. The method of claim 1 , further comprising changing a status of each process under control of the resource control service to a <suspended> state when each process under control of the resource control service is terminated.

6. The method of claim 5 , further comprising changing the one or more experiments to a <suspended> state when each of the resource control service at each environment containing the computing resources has its respective processes for the one or more experiments in the <suspended> state.

7. The method of claim 1 , further comprising restarting a suspended experiment when the computing resources are determined to be available by the resource management service.

8. The method of claim 7 , further comprising selecting the suspended experiment to restart based on at least one of a location of the computing resources in use by the one or more experiments, a priority level of the one or more experiments, an age of the one or more experiments, and/or a user ID of the one or more experiments.

9. The method of claim 7 , further comprising smoothing a signal regarding an availability of computing resources to avoid thrashing.

10. The method of claim 1 , further comprising providing a user with an option to manually suspend and manually resume an experiment.

11. A method for training an artificial intelligent agent to play a video game on a cloud-based game console shared with human users, comprising:
providing the artificial intelligent agent to interact in the video game;
configuring trainers to review experiences from the artificial intelligent agents and improve policies for the artificial intelligent agents for interacting with the video game;
monitoring a state of the experiment with a resource control service and determining whether to run the experiment once the experiment is in a scheduling state;
starting the experiment on a predetermined number of the cloud-based game consoles, with a predetermined number of the data gatherers, at one or more predetermined environments;
providing a resource management service for monitoring an amount of the cloud-based game consoles available at each of the environments, the resource management service including:
a first resource management module measuring load due to activity of the human users on the cloud-based game consoles at each of the environments;
a second resource management module predicting future load due to activity of the human users; and
a third resource management module determining the cloud-based game consoles given to the research users;

reading, by the resource control system. a number of the cloud-based game consoles available;
identifying, by the resource control system, one or more experiments to suspend; and
moving the one or more experiments identified to suspend into a <suspending> state.

12. The method of claim 11 , further comprising determining the one or more experiments to suspend based on at least one of a location of the cloud-based game console in use by the one or more experiments, a priority level of the one or more experiments, an age of the one or more experiments, and/or a user ID of the one or more experiments.

13. The method of claim 11 , further comprising assigning the resource control service at each environment containing the cloud-based game console, wherein each resource control service monitors for the one or more experiments in the <suspending> state.

14. The method of claim 13 , further comprising:
terminating each process under control of the resource control service for the one or more experiments in the <suspending> state;
changing a status of each process under control of the resource control service to a <suspended> state when each process under control of the resource control service is terminated; and
changing the one or more experiments to a <suspended> state when each of the resource control service at each environment containing the computing resources has its respective processes for the one or more experiments in the <suspended> state.

15. The method of claim 14 , further comprising saving state information into a remote storage before terminating each process.

16. The method of claim 14 , further comprising restarting a suspended experiment when the computing resources are determined to be available by the resource management service.

17. A method for managing computing resources of cloud-based game consoles between human users and research users to ensure a sufficient amount of cloud-based game consoles for the human users, the method comprising:
providing a resource management service for the computing resources, the resource management service including:
a first resource management module measuring load due to human activity;
a second resource management module predicting future load due to human activity; and
a third resource management module determining the computing resources given to the research users; and

providing a resource control service for the computing resources, the resource control service including:
a first resource control module reading a number of computing resources available; and
a second resource control module identifying one or more experiments to suspend;

moving the one or more experiments identified to suspend into a <suspending> state;
assigning the resource control service at each environment containing the cloud-based game consoles, wherein each resource control service monitors for the one or more

Assignees

• Sony Group Corp
• Sony Corp America
• Sony Interactive Entertainment LLC

Inventors

• Wurman Peter
• Barrett Leon
• Khandelwal Piyush
• Whitehead Dion
• Douglas Rory
• Aghabozorgi Houmehr
• Beltran Justin V
• Ahad Rabih Abdul
• Azzam Bandaly

Classifications

• G06N3/092

  Reinforcement learning · CPC title

• A63F13/67 Primary

  adaptively or by learning from player actions, e.g. skill level adjustment or by storing successful combat sequences for re-use · CPC title

• A63F13/352

  involving special game server arrangements, e.g. regional servers connected to a national server or a plurality of servers managing partitions of the game world · CPC title

• A63F2300/531

  Server assignment · CPC title

• A63F2300/6027

  using adaptive systems learning from user actions, e.g. for skill level adjustment · CPC title