Video compression system and method for compensating for bandwidth limitations of a communication channel

Patent metadata

Field	Value
Publication number	US-10130891-B2
Application number	US-35912709-A
Country	US
Kind code	B2
Filing date	Jan 23, 2009
Priority date	Dec 10, 2002
Publication date	Nov 20, 2018
Grant date	Nov 20, 2018

Abstract

A system and method are described below for encoding interactive low-latency video using interframe coding. For example, one embodiment of a computer-implemented method for performing video compression comprises: detecting a maximum data rate of a communication channel between a server and a client; transmitting a video stream comprising a series of sequential frames from the server to the client; detecting that the maximum data rate will be exceeded if a particular frame of the sequence of frames is transmitted from the server to the client over the communication channel; and in lieu of transmitting the frame which could cause the maximum data rate to be exceeded, causing the client to re-render the previous frame of the sequence of frames, thereby effectively reducing the frame rate of the video stream rendered on the client.

First claim

What is claimed is:

1. A computer-implemented method for hosting online video games, performing video compression and streaming video over a network comprising:
receiving control signals by a server, the control signals received from a client device over a network to the server, the control signals configured to be in response to a user input from an input device of the client device being actuated by a user as the user is playing a video game;
executing program code of the video game on the server in response to the received control signals;
generating a plurality of video frames resulting from execution of the video game;
encoding by one or more shared compression resources the plurality of video frames using low latency compression to generate a set of low latency compressed video frames;
detecting by the server, a maximum data rate of a communication channel established between the server and the client device, wherein determining the maximum data rate is enabled by sending an increasingly higher data rate test stream to the client device until at least one of a packet loss rate or an increased latency indicates the maximum data rate has been exceeded and then responsively reducing a data rate of the test stream until at least one of the packet loss rate or the latency have reached acceptable values;
updating a compression process state that sets an amount of compression for the communication channel for the detected maximum data rate;
transmitting the set of low latency compressed video frames from the server to the client device using the compression process state;
while transmitting, continuing to monitor when the maximum data rate for the communication channel has decreased based on measurements of at least one of an increase in the packet loss rate or said latency, and responsively reducing a data rate of the set of low latency compressed video frames until at least one of the packet loss rate or said latency have reached an acceptable level; and
while transmitting, continuing to monitor when the maximum data rate of the communication channel has increased based on continued measurements of at least one of a decrease in packet loss rate or said latency, and responsively increasing the data rate of the set of low latency compressed video frames until at least one of the packet loss rate or said latency reaches an unacceptable level;
wherein the compression process state is updated to adjust the amount of compression for responsively reducing or increasing the data rate;
wherein determining the maximum data rate includes receiving feedback from the client device for determining said packet loss rate or said latency.

2. The method as in claim 1 wherein the set of low latency compressed video frames are compressed in either a first encoding format or a second encoding format, the first encoding format comprising video frames which are dependent on previous video frames and the second encoding format comprising video frames which are not dependent on previous video frames.

3. The method as in claim 2 wherein the first encoding format is selected to reduce the data rate of the set of low latency compressed video frames to ensure that the maximum data rate of the communication channel will not be exceeded.

4. The method as in claim 1 further comprising:
encoding audio generated by the video game to generate a low latency compressed audio stream.

5. The method as in claim 1 further comprising detecting a change in the maximum data rate of the communication channel and dynamically adjusting a data rate of the set of low latency compressed video frames to accommodate the changed maximum data rate.

6. The method as in claim 5 wherein the data rate is adjusted by modifying a frame rate of the set of low latency compressed video frames.

7. The method as in claim 5 wherein the data rate is adjusted by modifying a quality of the set of low latency compressed video frames.

8. The method as in claim 5 wherein the data rate is adjusted by modifying an image resolution of the set of low latency compressed video frames.

9. The method as in claim 8 wherein when the image resolution is modified, the client device is configured to scale up the image to maintain the same image size on a display screen coupled to the client device.

10. The method as in claim 1 further comprising:
pausing the video game when the communication channel to the client device becomes unresponsive.

11. The method as in claim 1 further comprising generating forward error correction (FEC) data to protect at least a portion of the set of low latency compressed video frames.

12. The method as in claim 11 generating forward error correction (FEC) data to protect the portion of the low latency compressed video frames when a loss of a portion of the set of low latency compressed video frames would result in a noticeable visual artifact.

13. The method as in claim 12 wherein forward error correction (FEC) is only used if a disruption in the set of low latency compressed video frames would be more than a specified amount of time.

14. The method as in claim 11 further comprising:
protecting all audio packets generated by the video game using forward error correction (FEC).

15. The method as in claim 14 further comprising:
protecting the control signals received by the server from the client device using forward error correction (FEC).

16. A computer-implemented method for hosting online video games, performing video compression and streaming video over a network comprising:
receiving a control signal by a server, the control signal received from a client device over a communication channel to the server, the control signal being for executing an online video game by the server, wherein executing the online video game includes generating, encoding and transmitting a set of compressed video frames from the server to the client device;
determining a maximum data rate of the communication channel by sending an increasingly higher data rate data stream of the transmitted set of compressed video frames to the client device until at least one of a packet loss rate or an increase in latency indicates the maximum data rate of the communication channel has been exceeded;
detecting that the maximum data rate of the communication channel has been exceeded based on at least one of a detected increase in the packet loss rate or the increase in latency;
reducing a data rate of the set of compressed video frames to a reduced data rate, wherein the reduced data rate results in at least one of the packet loss rate or the latency to indicate an acceptable level; and
updating a compression process state that sets an amount of compression for the communication channel for the reduced data rate, wherein the compression process state is used to encode the set of compressed video frames;
while transmitting the set of compressed video frames, continuing to monitor when the maximum data rate for the communication channel has decreased based on measurements of at least one of an increase in the packet loss rate or said latency, and responsively reducing a data rate of the set of low latency compressed video frames until at least one of the packet loss rate or said latency have reached an acceptable level; and
while transmitting the set of compressed video frames, continuing to monitor when the maximum data rate of the communication channel has increased based on continued measurements of at least one of a decrease in packet loss rate or said latency, and responsively increasing the data rate of the set of low latency compressed video frames

Assignees

• Perlman Stephen G
• Van Der Laan Roger
• Sony Interactive Entertainment America Llc

Inventors

• Perlman Stephen G
• Van Der Laan Roger

Classifications

• A63F13/87 Primary

  Communicating with other players during game play, e.g. by e-mail or chat · CPC title

• A63F13/533

  for prompting the player, e.g. by displaying a game menu · CPC title

• A63F13/335 Primary

  using Internet · CPC title

• H04W88/14

  Backbone network devices · CPC title

• A63F2300/513

  server hierarchy, e.g. local, regional, national or dedicated for different tasks, e.g. authenticating, billing · CPC title