Rate control for low latency video encoding and transmission

Patent metadata

Field	Value
Publication number	US-2023009707-A1
Application number	US-202217933057-A
Country	US
Kind code	A1
Filing date	Sep 16, 2022
Priority date	Jun 28, 2018
Publication date	Jan 12, 2023
Grant date	—

Abstract

A video encoding system in which pixel data is decomposed into frequency bands prior to encoding. The frequency bands for a slice of a frame may be buffered so that complexity statistics may be calculated across the frequency bands prior to encoding. The statistics may then be used by a rate control component in determining quantization parameters for the frequency bands for modulating the rate in the encoder for the current slice. The quantization parameters for the frequency bands may be calculated jointly to optimize the quality of the displayed frames after decoder reconstruction and wavelet synthesis on a receiving device. Information about one or more previously processed frames may be used in combination with the statistics for a current slice in determining the quantization parameters for the current slice.

First claim

1 - 20 . (canceled)

21 . A video encoding system, comprising:
a wavelet transform component comprising one or more processors configured to:
apply a wavelet transform to pixel blocks from a current video frame to decompose the pixel blocks into a plurality of frequency bands; and
output frequency blocks comprising frequency data for the plurality of frequency bands;
a complexity analysis component comprising one or more processors configured to:
analyze the frequency data in the frequency blocks to calculate complexity statistics across the plurality of frequency bands of the current video frame; and
output the complexity statistics for the plurality of frequency bands of the current video frame;
a block-based encoder configured to:
determine quantization parameters for each of the frequency bands of the plurality of frequency bands of the current video frame based, at least in part, on the complexity statistics for the plurality of frequency bands of the current video frame in combination with other complexity statistics from a previous video frame and a foveated region of the previous video frame; and
apply an encoding technique to the frequency blocks of the current video frame, wherein the encoding technique uses the determined quantization parameters for the plurality of frequency bands of the current video frame to compress the frequency data in the frequency blocks; and
a wireless interface configured to transmit the compressed frequency data over a wireless connection to a receiving device.

22 . The video encoding system as recited in claim 21 , wherein, to determine the quantization parameters for each of the frequency bands of the plurality of frequency bands of the current video frame based, at least in part, on the complexity statistics for the plurality of frequency bands of the current video frame in combination with the other complexity statistics from the previous video frame and the foveated region of the previous video frame, the block-based encoder is further configured to:
determine a region of the current video frame based on the foveated region of the previous video frame; and determine the quantization parameters for each of the frequency bands of the plurality of frequency bands of the current video frame based, at least in part, on the determined region of the current video frame such that the determined region of the current video frame is encoded to provide higher resolution than a peripheral region of the current video frame outside of the determined region.

23 . The video encoding system as recited in claim 21 , wherein, to determine the quantization parameters for each of the frequency bands of the plurality of frequency bands of the current video frame based, at least in part, on the complexity statistics for the plurality of frequency bands of the current video frame in combination with the other complexity statistics from the previous video frame and the foveated region of the previous video frame, the block-based encoder is further configured to:
modulate the other complexity statistics from the previous video frame according to the foveated region of the previous video frame; and determine the quantization parameters for each of the frequency bands of the plurality of frequency bands of the current video frame based, at least in part, on the modulated other complexity statistics from the previous video frame in combination with the complexity statistics for the current video frame.

24 . The video encoding system as recited in claim 21 , wherein, to determine the quantization parameters for each of the frequency bands of the plurality of frequency bands of the current video frame based, at least in part, on the complexity statistics for the plurality of frequency bands of the current video frame in combination with the other complexity statistics from the previous video frame and the foveated region of the previous video frame, the block-based encoder is further configured to:
examine additional statistics for the previous video frame to determine an actual number of bits that were used to encode co-sited frequency blocks of the previous video frame; and determine the quantization parameters for the current video frame based, at least in part, on the determined actual number of bits that were used to encode the co-sited blocks of the previous video frame in combination with the complexity statistics for the current video frame.

25 . The video encoding system as recited in claim 21 , wherein the complexity statistics comprise one or more of pixel gradient information and variance in size of at least one of the frequency blocks with respect to another of the frequency blocks.

26 . The video encoding system as recited in claim 21 ,
wherein the wireless interface is further configured to provide a current available bit rate for the wireless connection as feedback to the block-based encoder; and wherein, when determining the quantization parameters for each of the frequency bands of the plurality of frequency bands of the current video frame, the block-based encoder is further configured to allocate the current available bit rate among respective ones of the plurality of frequency bands of the current video frame.

27 . The video encoding system as recited in claim 21 ,
wherein the wireless interface is further configured to provide a current available bit rate for the wireless connection and a current dropped packet rate on the wireless connection as feedback to the block-based encoder; wherein, when determining the quantization parameters for each of the frequency bands of the plurality of frequency bands of the current video frame, the block-based encoder is further configured to adjust the quantization parameters for respective ones of the plurality of frequency bands to minimize the dropped packet rate while maximizing available bandwidth usage at the current available bit rate.

28 . The video encoding system as recited in claim 21 , wherein the wavelet transform is a two-level wavelet decomposition applied to each pixel block from the current video frame.

29 . The video encoding system as recited in claim 21 , wherein the block-based encoder comprises a block-processing pipeline and a rate control component, wherein:
the rate control component is configured to perform the determination of the quantization parameters for each of the frequency bands and is further configured to provide the quantization parameters to the block processing pipeline; and the block processing pipeline is configured to apply the encoding technique to the frequency blocks of the current video frame.

30 . A method, comprising:
one or more processors configured to implement:
applying a wavelet transform to pixel blocks from a current video frame to decompose the pixel blocks into a plurality of frequency bands;
storing frequency data for the plurality of frequency bands into frequency blocks;
analyzing the frequency data in the frequency blocks to calculate complexity statistics across the plurality of frequency bands of the current video frame;
determining quantization parameters for each of the frequency bands of the plurality of frequency bands of the current video frame based, at least in part, on the complexity statistics for the plurality of frequency bands of the current video frame in combination with other complexity statistics from a previous video frame and a foveated region of the previous video frame;
applying an encoding technique to the frequency blocks, wherein the encoding technique uses the determined quantization parameters for the plurality of frequency bands of the current video frame to compress the frequency data in the frequency blo

Assignees

• Apple Inc

Inventors

• Pan Hao
• Chou Jim C
• Fernandes Felix C

Classifications

• H04N19/188

  the unit being a video data packet, e.g. a network abstraction layer [NAL] unit · CPC title

• H04N19/14

  Coding unit complexity, e.g. amount of activity or edge presence estimation (H04N19/146 takes precedence) · CPC title

• H04N19/115

  Selection of the code volume for a coding unit prior to coding · CPC title

• H04N19/176

  the region being a block, e.g. a macroblock · CPC title

• H04L65/70

  Media network packetisation · CPC title