Overlapping scan-in and encode at the server

Patent metadata

Field	Value
Publication number	US-12036471-B2
Application number	US-202217893707-A
Country	US
Kind code	B2
Filing date	Aug 23, 2022
Priority date	Oct 1, 2019
Publication date	Jul 16, 2024
Grant date	Jul 16, 2024

Abstract

A method for cloud gaming. The method including generating a video frame when executing a video game at a server. The method including scanning-in the video frame to an encoder at the server scanline-by-scanline. The method including encoding one or more encoder slices as one or more encoded slices at the encoder, wherein each encoded slice is delivered to a buffer at an encoder fill rate. The method including draining a first encoded slice from the buffer to stream the first encoded slice at a buffer drain rate over a network to a client, wherein the first encoded slice is streamed over the network before fully encoding the one or more slices of the video frame.

First claim

What is claimed is:

1. A method for cloud gaming, comprising:
generating a video frame at a server;
scanning-in the video frame to an encoder before a first next occurrence of a server VSYNC signal;
encoding the video frame to generate a plurality of encoded slices, wherein the encoding of the video frame begins before the scanning-in of the video frame to the encoder has fully completed;
storing the plurality of encoded slices in a buffer; and
streaming the plurality of encoded slices from the buffer to a client device before a second next occurrence of the server VSYNC signal, wherein the streaming of the plurality of encoded slices begins before the scanning-in of the video frame to the encoder has fully completed.

2. The method of claim 1 , further comprising:
beginning the streaming of the plurality of encoded slices before the encoding of the video frame has fully completed.

3. The method of claim 1 ,
wherein the scanning-in of the video frame occurs at a flip time.

4. The method of claim 1 , wherein the encoding the video frame includes:
encoding a plurality of encoder slices of the video frame as the plurality of encoded slices; and
delivering the plurality of encoded slices to the buffer at an encoder fill rate.

5. The method of claim 4 , further comprising:
draining an encoded slice from the buffer at a buffer drain rate; and
streaming the encoded slice over a network to the client device,
wherein the encoded slice is streamed to the client device before the scanning-in of the video frame to the encoder has fully completed.

6. The method of claim 1 , further comprising:
modifying the video frame during the scanning-in of the video frame, such that the video frame that is scanned-in to the encoder has been modified.

7. The method of claim 1 ,
wherein the first next occurrence and the second next occurrence of the server VSYNC signal occur at the same time.

8. A non-transitory computer-readable medium storing a computer program performing a method for cloud gaming, the computer-readable medium comprising:
program instructions for generating a video frame at a server;
program instructions for scanning-in the video frame to an encoder before a first next occurrence of a server VSYNC signal;
program instructions for encoding the video frame to generate a plurality of encoded slices, wherein the encoding of the video frame begins before the scanning-in of the video frame to the encoder has fully completed;
program instructions for storing the plurality of encoded slices in a buffer; and
program instructions for streaming the plurality of encoded slices from the buffer to a client device before a second next occurrence of the server VSYNC signal, wherein the streaming of the plurality of encoded slices begins before the scanning-in of the video frame to the encoder has fully completed.

9. The non-transitory computer-readable medium of claim 8 , further comprising:
program instructions for beginning the streaming of the plurality of encoded slices before the encoding of the video frame has fully completed.

10. The non-transitory computer-readable medium of claim 8 ,
wherein the scanning-in of the video frame occurs at a flip time.

11. The non-transitory computer-readable medium of claim 8 , wherein the program instructions for encoding the video frame includes:
program instructions for encoding a plurality of encoder slices of the video frame as the plurality of encoded slices; and
program instructions for delivering the plurality of encoded slices to the buffer at an encoder fill rate.

12. The non-transitory computer-readable medium of claim 11 , further comprising:
program instructions for draining an encoded slice from the buffer at a buffer drain rate; and
program instructions for streaming the encoded slice over a network to the client device,
wherein the encoded slice is streamed to the client device before the scanning-in of the video frame to the encoder has fully completed.

13. The non-transitory computer-readable medium of claim 8 , further comprising:
program instructions for modifying the video frame during the scanning-in of the video frame, such that the video frame that is scanned-in to the encoder has been modified.

14. The non-transitory computer-readable medium of claim 8 ,
wherein the first next occurrence and the second next occurrence of the server VSYNC signal occur at the same time.

15. A computer system comprising:
a processor; and
memory coupled to the processor and having stored therein instructions that, if executed by the computer system, cause the computer system to execute a method for cloud gaming comprising:
generating a video frame at a server;
scanning-in the video frame to an encoder before a first next occurrence of a server VSYNC signal;
encoding the video frame to generate a plurality of encoded slices, wherein the encoding of the video frame begins before the scanning-in of the video frame to the encoder has fully completed;
storing the plurality of encoded slices in a buffer; and
streaming the plurality of encoded slices from the buffer to a client device before a second next occurrence of the server VSYNC signal, wherein the streaming of the plurality of encoded slices begins before the scanning-in of the video frame to the encoder has fully completed.

16. The computer system of claim 15 , the method further comprising:
beginning the streaming of the plurality of encoded slices before the encoding of the video frame has fully completed.

17. The computer system of claim 15 ,
wherein in the method the scanning-in of the video frame occurs at a flip time.

18. The computer system of claim 15 , wherein in the method the encoding the video frame includes:
encoding a plurality of encoder slices of the video frame as the plurality of encoded slices; and
delivering the plurality of encoded slices to the buffer at an encoder fill rate.

19. The computer system of claim 15 , the method further comprising:
modifying the video frame during the scanning-in of the video frame, such that the video frame that is scanned-in to the encoder has been modified.

20. The computer system of claim 15 ,
wherein in the method the first next occurrence and the second next occurrence of the server VSYNC signal occur at the same time.

Assignees

• Sony Interactive Entertainment Inc

Inventors

• Yong Kelvin M
• Cerny Mark E

Classifications

• A63F13/50

  Controlling the output signals based on the game progress · CPC title

• H04N21/23605

  Creation or processing of packetized elementary streams [PES] · CPC title

• H04N21/4385

  Multiplex stream processing, e.g. multiplex stream decrypting · CPC title

• A63F2300/538

  for performing operations on behalf of the game client, e.g. rendering · CPC title

• H04N21/4781

  Games · CPC title