System and method for compressing video for streaming video game content to remote clients

Patent metadata

Field	Value
Publication number	US-10155160-B2
Application number	US-201514882417-A
Country	US
Kind code	B2
Filing date	Oct 13, 2015
Priority date	Dec 5, 2007
Publication date	Dec 18, 2018
Grant date	Dec 18, 2018

Abstract

Computer-implemented methods for hosting online video games, performing video compression, and streaming video for video games are provided. One method includes receiving a request from a user for playing a low latency video game, processing control signals received from a client, and touting the control signals to a server on which the requested video game is to be executed. The method further includes, executing the low latency video game on the server to generate a compressed video stream that is subsequently decompressed at the client. Some embodiments of the method include dynamically adjusting a determined maximum data rate of the compressed video stream in response to a current detected level of packet loss or latency by modifying the compression quality or image resolution or modifying the image rate.

First claim

What is claimed is:

1. A computer-implemented method for hosting online video games, performing video compression, and streaming video for video games, comprising:
receiving a request from a client of a user for playing a low latency video game;
determining a maximum data rate for a channel used to send and receive data to and from the client using a data rate test stream;
receiving, by a server, control signals from the client in response to an input device of the client being actuated by the user as the user is playing the low latency video game;
responsively executing the low latency video game on the server to generate video output comprising a sequence of images of the low latency video game; and
compressing, by a video compressor, the sequence of images of the low latency video game to generate a compressed video stream, the compressing including,
determining transmission times for images within the sequence of images based on the maximum data rate for the channel and a data size of the image;
for a first image within the sequence of images having a transmission time that is less than a frame time, compressing and transmitting the first image to the client and compressing and transmitting an image immediately following the first image to the client; and
for a second image within the sequence of images having a transmission time that is greater than the frame time, compressing and transmitting the second image to the client and discarding one or more images immediately following the second image, wherein the one or more images that are discarded are not compressed by the video compressor and not transmitted to the client.

2. The method of claim 1 , wherein the compression uses an encoder that processes a standard H.264 algorithm, and the client is configured to use a decoder that processes the standard H.264 algorithm.

3. The method as in claim 1 , wherein an image resolution of an image from the sequence of images is modified based on the determined maximum data rate, the client is configured to scale up the image to maintain a same image size on a display screen of the client.

4. The method as in claim 1 wherein the operations of receiving control signals from the client device over the Internet, executing the video game, compressing the sequence of images, streaming the compressed video stream to the client, and decompressing the compressed video stream are performed with a finite latency of 80 ms or less.

5. The method as in claim 1 wherein the operations of receiving control signals from the client device over the Internet, executing the video game, compressing the sequence of images, streaming the compressed video stream to the client, and decompressing the compressed video stream are performed with a finite latency of 150 ms or less.

6. The method of claim 1 , wherein dynamically adjusting the determined maximum data rate of the compressed video stream in response to a current detected level of packet loss and/or latency;
wherein dynamic adjusting includes one or more of modifying an image rate, a compression ratio, and an image resolution.

7. A computer-implemented method executed by an online video game hosting system, comprising:
receiving a request from a client device for playing a video game at a server of the online video game hosting system;
determining a current data rate of a channel for sending and receiving data between the server and the client device, the data including control signals, compressed video frames, compressed audio, and feedback, wherein the current data rate is based on a maximum available data rate for the channel, the current data rate being periodically adjusted based on changes to the maximum available data rate, said determining the maximum data rate including,
sending a data rate test stream from the server to the client device, the data rate test stream having a data rate that increases to eventually exceed the maximum data rate;
receiving, from the client device, feedback on a level of packet loss and/or latency of the data rate test stream;
determining, from the feedback, that a level of packet loss and/or latency of the data rate test stream is not within a predetermined threshold;
reducing the data rate of the data rate test stream until it is determined from the feedback that a level of packet loss and/or latency is within a predetermined threshold; and
setting the reduced data rate as the maximum data rate of the channel;

receiving, at the server, control signals from the client device, the control signals used at least partially to control actions within the video game being executed by the server, the execution by the server generating a sequence of images for video game; and
by a video compressor, for an image of the sequence of images calculated to have a transmission time less than a frame time as determined by the current data rate and a data size of the image, compressing and transmitting to the client device the image and then compressing and transmitting a subsequent image immediately following the image within the sequence of images, and for an image calculated to have a transmission time that is greater than the frame time, compressing and transmitting the image and discarding one or more images immediately following the image, wherein the one or more images that are discarded are not compressed by the video compressor and not transmitted to the client.

8. The computer-implemented method as recited in claim 7 , wherein execution by the server further comprises:
generating, in response to the control signals, a sequence of audio data, the sequence of audio data synchronized to the sequence of images for the low latency video game; and
compressing and transmitting the sequence of audio data even if one or more images to which the audio data is synchronized are discarded.

9. The computer-implemented method as recited in claim 7 , wherein the frame time is calculated based on an inverse of a frame rate of the compressed video stream, the frame rate defines a number of images within the sequence of images that is configured to be displayed at the client device per unit time.

10. The method as recited in claim 1 , wherein executing the low latency video game further comprises:
generating, in response to the control signals, a sequence of audio data, the sequence of audio data synchronized to the sequence of images for the low latency video game; and
compressing and transmitting the sequence of audio data even if one or more images to which the audio data is synchronized are discarded.

11. The method as recited in claim 1 , wherein determining the maximum data rate of a channel for sending and receiving data between the server and the client device comprises:
sending the data rate test stream from the server to the client device, the data rate test stream having a data rate that increases to eventually exceed the maximum data rate;
receiving, from the client device, feedback on a level of packet loss and/or latency of the data rate test stream;
determining, from the feedback, that a level of packet loss and/or latency of the data rate test stream is not within a predetermined threshold;
reducing the data rate of the data rate test stream until it is determined from the feedback that a level of packet loss and/or latency is within a predetermined threshold; and
setting the reduced data rate as the maximum data rate of the channel.

12. A computer-implemented method executed by an online video game hosting system, comprising:
receiving a request from a client device for playing a video game at a server of the online video game hosting system;
determining a current data rate of a channel for

Assignees

• Sony Interactive Entertainment America Llc

Inventors

• Van Der Laan Roger
• Perlman Stephen G

Classifications

• H04N21/6125

  involving transmission via Internet (transmission by internet of broadcast information H04H60/82) · CPC title

• A47L13/51

  Storing of cleaning tools, e.g. containers therefor · CPC title

• H04N21/437

  Interfacing the upstream path of the transmission network, e.g. for transmitting client requests to a VOD server {(flow control in data networks H04L47/10; streaming protocols, e.g. RTP or RTCP, H04L65/65; scheduling or organising the servicing of application requests in data packet switching networks H04L67/60)} · CPC title

• A47L13/59

  with movable squeezing members · CPC title

• A63F13/358

  Adapting the game course according to the network or server load, e.g. for reducing latency due to different connection speeds between clients · CPC title