Augmenting multimedia streaming by anticipating events using artificial intelligence

Patent metadata

Field	Value
Publication number	US-2025168217-A1
Application number	US-202519025621-A
Country	US
Kind code	A1
Filing date	Jan 16, 2025
Priority date	Oct 27, 2021
Publication date	May 22, 2025
Grant date	—

Abstract

Apparatuses, systems, and techniques to predict events within a gameplay session and modify a game broadcast based at least in part on the predicted events. In at least one embodiment, events within the gameplay session are predicted, which once predicted, cause assets to be generated and once the event is detected the assets are included in the game broadcast.

First claim

What is claimed is:

1 . A method comprising:
predicting an interval of time based at least in part on user activity data associated with subject matter of a stream; modifying one or more frames of the stream that correspond to the interval of time; and causing the modified one or more frames to be broadcast in the stream.

2 . The method of claim 1 , wherein the subject matter of the stream comprises a game, and the interval of time is a prediction of an occurrence of at least one in-game event.

3 . The method of claim 1 , wherein the user activity data includes at least one of video data, audio data, device data, or biometric data.

4 . The method of claim 1 , wherein the subject matter of the stream comprises a game, and the user activity data comprises at least one input obtained from a controller device associated with the game.

5 . The method of claim 4 , wherein the controller device includes at least one of: a gaming console controller, a track pad, a mouse, a keyboard, or a motion controller.

6 . The method of claim 1 , wherein modifying the one or more frames comprises applying an effect to the one or more frames that is not applied to one or more other frames of the stream.

7 . The method of claim 1 , wherein predicting the interval of time is based, at least in part, on information associated with the stream that includes at least one of: data obtained from a remote client, one or more image frames corresponding to the stream, audio of the stream, or audio captured from the remote client.

8 . The method of claim 1 , wherein modifying the one or more frames comprises displaying an overlay in the one or more frames.

9 . The method of claim 1 , wherein modifying the one or more frames comprises including an audio cue in the one or more frames.

10 . A processor comprising:
one or more circuits to at least: predict an event will occur within an interval of time based at least in part on user activity data; modify a stream by at least applying an effect to one or more frames of the stream corresponding the interval of time; and cause the one or more frames with the applied effect to be broadcast in the stream.

11 . The processor of claim 10 , wherein the one or more circuits are to at least:
generate an asset; and cause a state of the asset to be modified after the event is determined to have occurred.

12 . The processor of claim 11 , wherein the event is to be predicted to occur within the interval of time based, at least in part, on an event type associated with the event and a type associated with the asset.

13 . The processor of claim 11 , wherein the one or more circuits are to at least:
apply another effect corresponding to the asset to at least one frame of the stream.

14 . The processor of claim 13 , wherein the other effect adds a delay between two or more successive frames of the at least one frame of the stream.

15 . The processor of claim 10 , wherein the event is to be predicted to occur within the interval of time based, at least in part, on audio or video captured at a host computer system associated with the stream.

16 . The processor of claim 10 , wherein the user activity data comprises inputs obtained from a controller device logged at a host computer system associated with the stream.

17 . The processor of claim 10 , wherein the event is an in-game event.

18 . A system comprising:
one or more processors; and memory storing instructions that, as a result of being executed by the one or more processors, cause the system to:
modify a streaming session by at least applying an effect to one or more frames corresponding a predicted interval of time, wherein the predicted interval of time is to predict when an event is to occur in the streaming session, and the predicted interval of time is to be generated based at least in part on an analysis of user inputs associated with the streaming session; and
cause the modified streaming session to be broadcast.

19 . The system of claim 18 , wherein the memory further stores instructions that, as a result of being executed by the one or more processors, cause the system to:
during the predicted interval of time, obtain a set of images of the streaming session; and generate a determination that the event occurred prior to an expiration of the predicted interval of time.

20 . The system of claim 18 , wherein the memory further stores instructions that, as a result of being executed by the one or more processors, cause the system to:
cause the effect to no longer be applied to the streaming session in response to an expiration of the predicted interval of time.

21 . The system of claim 18 , wherein the user inputs associated with the streaming session includes keystroke data obtained from a remote client.

22 . The system of claim 18 , wherein the instructions further include instructions that cause the system to:
cause a machine learning model to generate the predicted interval of time.

23 . The system of claim 18 , wherein the memory further stores instructions that, as a result of being executed by the one or more processors, cause the system to:
continue causing the effect to be applied beyond expiration of the predicted interval of time as a result of a determination that the event has occurred.

24 . The system of claim 18 , wherein the streaming session is associated with a game, and the event is an in-game event.

Assignees

• Nvidia Corp

Inventors

• Sundareson Prabindh
• Pandhare Sachin Dattatray
• Raikar Shyam
• Sharma Toshant
• Yadav Prakash

Classifications

• H04L65/65

  Network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP] · CPC title

• H04N21/2187

  Live feed · CPC title

• A63F13/35

  Details of game servers · CPC title

• G06N20/00

  Machine learning · CPC title

• A63F13/85

  Providing additional services to players · CPC title