Augmenting multimedia streaming by anticipating events using artificial intelligence

Patent metadata

Field	Value
Publication number	US-12388889-B2
Application number	US-202117512590-A
Country	US
Kind code	B2
Filing date	Oct 27, 2021
Priority date	Oct 27, 2021
Publication date	Aug 12, 2025
Grant date	Aug 12, 2025

Abstract

Apparatuses, systems, and techniques to predict events within a gameplay session and modify a game broadcast based at least in part on the predicted events. In at least one embodiment, events within the gameplay session are predicted, which once predicted, cause assets to be generated and once the event is detected the assets are included in the game broadcast.

First claim

What is claimed is:

1. A method comprising:
obtaining data associated with a streaming session;
predicting an interval of time during which an in-game highlight event is predicted to occur in the streaming session by analyzing user inputs associated with the streaming session;
based at least in part on the prediction, causing the streaming session to be modified to generate a modified streaming session by at least applying an effect to one or more frames corresponding to the streaming session and the predicted interval of time; and
causing the modified streaming session to be broadcast after the in-game highlight event occurs within the streaming session.

2. The method of claim 1 , wherein the method further comprises:
capturing a set of inputs associated with the streaming session;
generating a determination that the in-game highlight event has occurred based at least in part on the set of inputs; and
causing the effect to no longer be applied to one or more frames corresponding to the modified streaming session.

3. The method of claim 2 , wherein the set of inputs include at least one of an image, audio, or user input.

4. The method of claim 2 , wherein generating the determination that the in-game highlight event has occurred is performed using a model trained to detect objects in one or more image frames corresponding to the streaming session.

5. The method of claim 1 , wherein the method further comprises:
determining an expiration of the predicted interval of time; and
causing the effect to no longer be applied to one or more other frames corresponding to the streaming session.

6. The method of claim 1 , wherein the streaming session corresponds to a gameplay session initiated by a server.

7. The method of claim 1 , wherein data associated with the streaming session includes at least one of: input data obtained from a remote client, one or more image frames corresponding to the streaming session, audio of the streaming session, or audio captured from the remote client.

8. The method of claim 1 , wherein the effect further comprises an overlay displayed in one or more image frames corresponding to the streaming session.

9. The method of claim 1 , wherein the effect further comprises an audio cue included in one or more audio frames corresponding to the streaming session.

10. The method of claim 1 , wherein the user inputs comprise inputs obtained from a controller device associated with a game session.

11. The method of claim 10 , wherein the controller device includes at least one of: a gaming console controller, a track pad, a mouse, a keyboard, or a motion controller.

12. A non-transitory machine-readable medium having stored thereon a set of instructions, which, if performed by one or more processors, cause the one or more processors to:
obtain data generated during a streaming session;
predict an interval of time during which an in-game highlight event is predicted to occur in the streaming session by analyzing user inputs associated with the streaming session;
based at least in part on the interval of time, cause the streaming session to be modified by at least applying an asset to the streaming session corresponding to the predicted interval of time; and
broadcast the modified streaming session after the in-game highlight event occurs within the streaming session.

13. The non-transitory machine-readable medium of claim 12 , wherein the set of instructions further include instructions, which, if performed by the one or more processors, cause the one or more processors to:
capture an image corresponding to the streaming session;
generate a determination that the in-game highlight event has occurred based at least in part on the image; and
cause a state of the asset to be modified.

14. The non-transitory machine-readable medium of claim 12 , wherein the set of instructions further include instructions, which, if performed by the one or more processors, cause the one or more processors to determine the predicted interval of time based at least in part on an event type associated with the in-game highlight event and a type associated with the asset.

15. The non-transitory machine-readable medium of claim 12 , wherein the asset further comprises an effect applied to the streaming session.

16. The non-transitory machine-readable medium of claim 15 , wherein the effect adds a delay between successive frames of the streaming session.

17. The non-transitory machine-readable medium of claim 12 , wherein the set of instructions that cause the one or more processors to predict the in-game highlight event further include instructions, which, if performed by the one or more processors, cause the one or more processors to predict the in-game highlight event based at least in part on audio or video captured at a host computer system associated with the streaming session.

18. The non-transitory machine-readable medium of claim 12 , wherein the user inputs comprise inputs from a controller device logged at a host computer system associated with the streaming session.

19. A system comprising:
one or more processors; and
memory storing instructions that, as a result of being executed by the one or more processors, cause the system to:
cause a determination to be generated, the determination predicting an interval of time during which an in-game highlight event is predicted to occur in a streaming session by analyzing user inputs associated with the streaming session, based at least in part on data associated with a streaming session;
in response to the determination, cause an effect to be applied to the streaming session corresponding to the predicted interval of time; and
broadcast the streaming session with the effect applied after the in-game highlight event occurs within the streaming session.

20. The system of claim 19 , wherein the memory further stores instructions that, as a result of being executed by the one or more processors, cause the system to:
during the predicted interval of time, obtain a set of images of the streaming session; and
generate a second determination that the in-game highlight event occurred prior to an expiration of the predicted interval of time.

21. The system of claim 19 , wherein the memory further stores instructions that, as a result of being executed by the one or more processors, cause the system to, in response to an expiration of the predicted interval of time, cause the effect to no longer be applied to the streaming session.

22. The system of claim 19 , wherein the data associated with the streaming session includes keystroke data obtained from a remote client.

23. The system of claim 19 , wherein the instructions that cause the determination to be generated further include instructions that, as a result of being executed by the one or more processors, cause the system to cause a machine learning model, executed by a server, to generate the determination.

24. The system of claim 23 , wherein the machine learning model generates the determination based at least in part on images of a game session captured by the system.

25. The system of claim 23 , wherein the machine learning model generates the determination based at least in part on information obtained from a game engine.

26. The system of claim 23 , wherein the machine learning model generates the determination based at least in part on information obtained from a game associated w

Assignees

• Nvidia Corp

Inventors

• Sundareson Prabindh
• Pandhare Sachin Dattatray
• Raikar Shyam
• Sharma Toshant
• Yadav Prakash

Classifications

• H04L65/65

  Network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP] · CPC title

• H04N21/2187

  Live feed · CPC title

• A63F13/35

  Details of game servers · CPC title

• G06N20/00

  Machine learning · CPC title

• A63F13/85

  Providing additional services to players · CPC title