Rate control for low latency video encoding and transmission

Patent metadata

Field	Value
Publication number	US-12081769-B2
Application number	US-202217933057-A
Country	US
Kind code	B2
Filing date	Sep 16, 2022
Priority date	Jun 28, 2018
Publication date	Sep 3, 2024
Grant date	Sep 3, 2024

Abstract

A video encoding system in which pixel data is decomposed into frequency bands prior to encoding. The frequency bands for a slice of a frame may be buffered so that complexity statistics may be calculated across the frequency bands prior to encoding. The statistics may then be used by a rate control component in determining quantization parameters for the frequency bands for modulating the rate in the encoder for the current slice. The quantization parameters for the frequency bands may be calculated jointly to optimize the quality of the displayed frames after decoder reconstruction and wavelet synthesis on a receiving device. Information about one or more previously processed frames may be used in combination with the statistics for a current slice in determining the quantization parameters for the current slice.

First claim

What is claimed is:

1. A video encoding system, comprising:
a wavelet transform component comprising one or more processors configured to:
apply a wavelet transform to pixel blocks from a current video frame to decompose the pixel blocks into a plurality of frequency bands; and
output frequency blocks comprising frequency data for the plurality of frequency bands;

a complexity analysis component comprising one or more processors configured to:
analyze the frequency data in the frequency blocks to calculate complexity statistics across the plurality of frequency bands of the current video frame; and
output the complexity statistics for the plurality of frequency bands of the current video frame;

a block-based encoder configured to:
access a foveated region from a previous video frame;
determine a number of bits that were used to encode co-sited pixel blocks from the previous video frame;
determine quantization parameters for each of the frequency bands of the plurality of frequency bands of the current video frame based, at least in part, on:
(a) the complexity statistics for the plurality of frequency bands of the current video frame;
(b) the other complexity statistics from the previous video frame;
(c) the foveated region of the previous video frame; and
(d) the determined number of bits that were used to encode the co-sited pixel blocks from the previous video frame; and

apply an encoding technique to the frequency blocks of the current video frame, wherein the encoding technique uses the determined quantization parameters for the plurality of frequency bands of the current video frame to compress the frequency data in the frequency blocks; and

a wireless interface configured to transmit the compressed frequency data over a wireless connection to a receiving device.

2. The video encoding system as recited in claim 1 , wherein, to determine the quantization parameters for each of the frequency bands of the plurality of frequency bands of the current video frame, the block-based encoder is further configured to:
determine a region of the current video frame based on the foveated region of the previous video frame; and
determine the quantization parameters for each of the frequency bands of the plurality of frequency bands of the current video frame based, at least in part, on the determined region of the current video frame such that the determined region of the current video frame is encoded to provide higher resolution than a peripheral region of the current video frame outside of the determined region.

3. The video encoding system as recited in claim 1 , wherein, to determine the quantization parameters for each of the frequency bands of the plurality of frequency bands of the current video frame, the block-based encoder is further configured to:
modulate the other complexity statistics from the previous video frame according to the foveated region of the previous video frame; and
determine the quantization parameters for each of the frequency bands of the plurality of frequency bands of the current video frame based, at least in part, on the modulated other complexity statistics from the previous video frame in combination with the complexity statistics for the current video frame.

4. The video encoding system as recited in claim 1 , wherein, to determine the quantization parameters for each of the frequency bands of the plurality of frequency bands of the current video frame, the block-based encoder is further configured to:
examine additional statistics for the previous video frame to determine the number of bits that were used to encode co-sited frequency blocks of the previous video frame.

5. The video encoding system as recited in claim 1 , wherein the complexity statistics comprise one or more of pixel gradient information and variance in size of at least one of the frequency blocks with respect to another of the frequency blocks.

6. The video encoding system as recited in claim 1 ,
wherein the wireless interface is further configured to provide a current available bit rate for the wireless connection as feedback to the block-based encoder; and
wherein, when determining the quantization parameters for each of the frequency bands of the plurality of frequency bands of the current video frame, the block-based encoder is further configured to allocate the current available bit rate among respective ones of the plurality of frequency bands of the current video frame.

7. The video encoding system as recited in claim 1 ,
wherein the wireless interface is further configured to provide a current available bit rate for the wireless connection and a current dropped packet rate on the wireless connection as feedback to the block-based encoder;
wherein, when determining the quantization parameters for each of the frequency bands of the plurality of frequency bands of the current video frame, the block-based encoder is further configured to adjust the quantization parameters for respective ones of the plurality of frequency bands to minimize the dropped packet rate while maximizing available bandwidth usage at the current available bit rate.

8. The video encoding system as recited in claim 1 , wherein the wavelet transform is a two-level wavelet decomposition applied to each pixel block from the current video frame.

9. The video encoding system as recited in claim 1 , wherein the block-based encoder comprises a block-processing pipeline and a rate control component, wherein:
the rate control component is configured to perform the determination of the quantization parameters for each of the frequency bands and is further configured to provide the quantization parameters to the block processing pipeline; and
the block processing pipeline is configured to apply the encoding technique to the frequency blocks of the current video frame.

10. A method, comprising:
one or more processors configured to implement:
applying a wavelet transform to pixel blocks from a current video frame to decompose the pixel blocks into a plurality of frequency bands;
storing frequency data for the plurality of frequency bands into frequency blocks;
analyzing the frequency data in the frequency blocks to calculate complexity statistics across the plurality of frequency bands of the current video frame;
accessing a foveated region from a previous video frame;
determining a number of bits that were used to encode co-sited pixel blocks from the previous video frame;
determining quantization parameters for each of the frequency bands of the plurality of frequency bands of the current video frame based, at least in part, on:
(a) the complexity statistics for the plurality of frequency bands of the current video frame;
(b) the other complexity statistics from the previous video frame;
(c) the foveated region of the previous video frame; and
(d) the determined number of bits that were used to encode the co-sited pixel blocks from the previous video frame;

applying an encoding technique to the frequency blocks, wherein the encoding technique uses the determined quantization parameters for the plurality of frequency bands of the current video frame to compress the frequency data in the frequency blocks; and
transmitting the compressed frequency data over a wireless connection to a receiving device.

11. The method as recited in claim 10 , wherein said determining quantization parameters for each of the frequency bands of the plurality of frequency bands of the current video frame comprises:
determining a region of the current video frame based on the foveated region of the previous video frame; and
determining the quantization parameters for each of the frequency bands of t

Assignees

• Apple Inc

Inventors

• Pan Hao
• Chou Jim C
• Fernandes Felix C

Classifications

• H04L65/752

  adapting media to network capabilities · CPC title

• H04N19/146 Primary

  Data rate or code amount at the encoder output · CPC title

• H04N19/63 Primary

  using sub-band based transform, e.g. wavelets · CPC title

• H04L65/70

  Media network packetisation · CPC title

• H04N19/188

  the unit being a video data packet, e.g. a network abstraction layer [NAL] unit · CPC title