Encoding a video frame as a reference frame based on a scene change hint at a cloud gaming server

Patent metadata

Field	Value
Publication number	US-12289452-B2
Application number	US-202218089187-A
Country	US
Kind code	B2
Filing date	Dec 27, 2022
Priority date	Oct 1, 2019
Publication date	Apr 29, 2025
Grant date	Apr 29, 2025

Abstract

A method for encoding including executing game logic built on a game engine of a video game at a cloud gaming server to generate video frames. The method including executing scene change logic to predict a scene change in the video frames based on game state collected during execution of the game logic. The method including identifying a range of video frames that is predicted to include the scene change. The method including generating a scene change hint using the scene change logic, wherein the scene change hint identifies the range of video frames, wherein the range of video frames includes a first video frame. The method including delivering the first video frame to an encoder. The method including sending the scene change hint from the scene change logic to the encoder. The method including encoding the first video frame as an I-frame based on the scene change hint.

First claim

What is claimed is:

1. A method for encoding, comprising:
receiving at an encoder from a central processing unit (CPU) a notification that a video frame is predicted to include a scene change, wherein a plurality of video frames is generated by a game engine executing game logic corresponding to a video game at a cloud gaming server, wherein the CPU executes the game logic via the game engine, wherein the prediction that the video frame includes the scene change occurs before the video frame is generated;
disabling scene change detection at the encoder;
receiving the video frame at the encoder;
determining at the encoder that the notification has been received, wherein the notification is received by the encoder before the encoder begins encoding the video frame; and
encoding the video frame as a reference frame based on the notification that the video frame is predicted to include a scene change.

2. The method of claim 1 , further comprising:
sending the video frame that is encoded to a streamer for streaming to a client device over a network.

3. The method of claim 1 , wherein the encoding the video frame includes:
encoding a plurality of encoder slices that combined form the reference frame.

4. The method of claim 1 ,
wherein the video frame is not encoded based on a previous reference frame before the video frame is encoded as the reference frame.

5. The method of claim 1 , further comprising:
receiving a second video frame;
determining that no notification was received that corresponds to the second video frame; and
encoding the second video frame based on the reference frame.

6. The method of claim 1 , further comprising:
executing scene change logic to predict the scene change in the plurality of video frames, wherein prediction of the scene change is based on game state collected during execution of the game logic;
generating the notification using the scene change logic; and
forwarding the notification to the encoder.

7. The method of claim 1 , further comprising:
determining that the scene change of the video frame does not meet or exceed a threshold, wherein the threshold reflects a difference between the video frame and a previous video frame;
terminating encoding of the video frame as a reference frame; and
encoding the video frame based on a previous reference frame.

8. A non-transitory computer-readable medium storing a computer program for encoding, the computer-readable medium comprising:
program instructions for receiving at an encoder from a central processing unit (CPU) a notification that a video frame is predicted to include a scene change, wherein a plurality of video frames is generated by a game engine executing game logic corresponding to a video game at a cloud gaming server, wherein the CPU executes the game logic via the game engine, wherein the prediction that the video frame includes the scene change occurs before the video frame is generated;
program instructions for disabling scene change detection at the encoder;
program instructions for receiving the video frame at the encoder;
program instructions for determining at the encoder that the notification has been received, wherein the notification is received by the encoder before the encoder begins encoding the video frame; and
program instructions for encoding the video frame as a reference frame based on the notification that the video frame is predicted to include a scene change.

9. The non-transitory computer-readable medium of claim 8 , further comprising:
program instructions for sending the video frame that is encoded to a streamer for streaming to a client device over a network.

10. The non-transitory computer-readable medium of claim 8 , wherein the program instructions for encoding the video frame includes:
program instructions for encoding a plurality of encoder slices that combined form the reference frame.

11. The non-transitory computer-readable medium of claim 8 ,
wherein in the program instructions the video frame is not encoded based on a previous reference frame before the video frame is encoded as the reference frame.

12. The non-transitory computer-readable medium of claim 8 , further comprising:
program instructions for receiving a second video frame;
program instructions for determining that no notification was received that corresponds to the second video frame; and
program instructions for encoding the second video frame based on the reference frame.

13. The non-transitory computer-readable medium of claim 8 , further comprising:
program instructions for executing scene change logic to predict the scene change in the plurality of video frames, wherein prediction of the scene change is based on game state collected during execution of the game logic;
program instructions for generating the notification using the scene change logic; and
program instructions for forwarding the notification to the encoder.

14. The non-transitory computer-readable medium of claim 8 , further comprising:
program instructions for determining that the scene change of the video frame does not meet or exceed a threshold, wherein the threshold reflects a difference between the video frame and a previous video frame;
program instructions for terminating encoding of the video frame as a reference frame; and
program instructions for encoding the video frame based on a previous reference frame.

15. A computer system comprising:
a processor; and
memory coupled to the processor and having stored therein instructions that, if executed by the computer system, cause the computer system to execute a method for encoding comprising:
receiving at an encoder from a central processing unit (CPU) a notification that a video frame is predicted to include a scene change, wherein a plurality of video frames is generated by a game engine executing game logic corresponding to a video game at a cloud gaming server, wherein the CPU executes the game logic via the game engine, wherein the prediction that the video frame includes the scene change occurs before the video frame is generated;
disabling scene change detection at the encoder;
receiving the video frame at the encoder;
determining at the encoder that the notification has been received, wherein the notification is received by the encoder before the encoder begins encoding the video frame; and
encoding the video frame as a reference frame based on the notification that the video frame is predicted to include a scene change.

16. The computer system of claim 15 , the method further comprising:
sending the video frame that is encoded to a streamer for streaming to a client device over a network.

17. The computer system of claim 15 ,
wherein in the method the video frame is not encoded based on a previous reference frame before the video frame is encoded as the reference frame.

18. The computer system of claim 15 , the method further comprising:
receiving a second video frame;
determining that no notification was received that corresponds to the second video frame; and
encoding the second video frame based on the reference frame.

19. The computer system of claim 15 , the method further comprising:
executing scene change logic to predict the scene change in the plurality of video frames, wherein prediction of the scene change is based on game state collected during execution of the game logic;
generating the notification using the scene change logic; and
forwarding the notification to the encoder.

20. The computer system of

Assignees

• Sony Interactive Entertainment Inc

Inventors

• Cerny Mark E
• Yong Kelvin M

Classifications

• H04N19/179

  the unit being a scene or a shot · CPC title

• H04N19/177

  the unit being a group of pictures [GOP] · CPC title

• A63F13/352

  involving special game server arrangements, e.g. regional servers connected to a national server or a plurality of servers managing partitions of the game world · CPC title

• A63F13/50

  Controlling the output signals based on the game progress · CPC title

• H04N19/142 Primary

  Detection of scene cut or scene change · CPC title