Machine learning model for video with real-time rate control

Patent metadata

Field	Value
Publication number	US-2025080751-A1
Application number	US-202418816444-A
Country	US
Kind code	A1
Filing date	Aug 27, 2024
Priority date	Aug 30, 2023
Publication date	Mar 6, 2025
Grant date	—

Abstract

Methods and systems for rate control include determining an encoding parameter value to use for an input set of video frames based on a current channel capacity, using a machine learning model that accepts the input set of video frames and the current channel capacity as inputs. The input set of video frames are encoded using the encoding parameter to generate encoded video that has a bitrate below the current channel capacity. The encoded video is transmitted.

First claim

What is claimed is:

1 . A computer-implemented method for rate control, comprising:
determining an encoding parameter value to use for an input set of video frames based on a current channel capacity, using a machine learning model that accepts the input set of video frames and the current channel capacity as inputs; encoding the input set of video frames using the encoding parameter to generate encoded video that has a bitrate below the current channel capacity; and transmitting the encoded video.

2 . The method of claim 1 , further comprising determining the current channel capacity based on channel quality information received from user equipment.

3 . The method of claim 1 , wherein the machine learning model includes a prediction head model that is trained to generate a parameter value that, when used to encode the input set of video frames, results in the encoded video being at or below the current channel capacity.

4 . The method of claim 3 , wherein the prediction head model is a deep neural network model that includes conditional group normalization using the current channel capacity as a condition.

5 . The method of claim 4 , wherein the prediction head model includes a plurality of convolutional layers, each followed by a respective conditional group normalization.

6 . The method of claim 1 , wherein the encoding parameter is a quantization parameter.

7 . The method of claim 1 , further comprising altering the determined encoding parameter value to decrease video quality before encoding the video.

8 . The method of claim 1 , wherein determining the encoding parameter value includes extracting features from the input set of video frames and processing the features with the current channel capacity in a prediction head model.

9 . The method of claim 1 , wherein the encoded video is transmitted to a medical professional to aid in medical decision making.

10 . The method of claim 1 , further comprising performing a treatment action responsive to the encoded video, including automatically altering a patient's treatment in response to a patient activity shown in the encoded video.

11 . The method of claim 1 , wherein determining the encoding parameter value includes maximization of average video quality of a live video feed, subject to the current channel capacity available for the set of video frames and minimization of a probability of packet drop and video artifacts in transmission of the encoded video.

12 . A system for rate control, comprising:
a hardware processor; and a memory that stores a computer program which, when executed by the hardware processor, causes the hardware processor to:
determine an encoding parameter value to use for an input set of video frames based on a current channel capacity, using a machine learning model that accepts the input set of video frames and the current channel capacity as inputs;
encode the input set of video frames using the encoding parameter to generate encoded video that has a bitrate below the current channel capacity; and
transmit the encoded video.

13 . The system of claim 12 , wherein the computer program further causes the hardware processor to determine the current channel capacity based on channel quality information received from user equipment.

14 . The system of claim 12 , wherein the machine learning model includes a prediction head model that is trained to generate a parameter value that, when used to encode the input set of video frames, results in the encoded video being at or below the current channel capacity.

15 . The system of claim 14 , wherein the prediction head model is a deep neural network model that includes conditional group normalization using the current channel capacity as a condition.

16 . The system of claim 15 , wherein the prediction head model includes a plurality of convolutional layers, each followed by a respective conditional group normalization.

17 . The system of claim 12 , wherein the computer program further causes the hardware processor to alter the determined encoding parameter value to decrease video quality before encoding the video.

18 . The system of claim 12 , wherein the computer program further causes the hardware processor to extract features from the input set of video frames and to process the features with the current channel capacity in a prediction head model.

19 . The system of claim 12 , wherein the encoded video is transmitted to a medical professional to aid in medical decision making.

20 . The system of claim 12 , wherein the computer program further causes the hardware processor to perform a treatment action responsive to the encoded video, including automatically alteration of a patient's treatment in response to a patient activity shown in the encoded video.

Assignees

• Nec Lab America Inc

Inventors

• Khojastepour Mohammad
• Chakradhar Srimat
• Mortaheb Matin

Classifications

• H04N19/124

  Quantisation · CPC title

• H04N19/115

  Selection of the code volume for a coding unit prior to coding · CPC title

• H04N19/149 Primary

  by estimating the code amount by means of a model, e.g. mathematical model or statistical model · CPC title

• H04N19/172 Primary

  the region being a picture, frame or field · CPC title