Encoding output for streaming applications based on client upscaling capabilities

Patent metadata

Field	Value
Publication number	US-11818192-B2
Application number	US-202217683140-A
Country	US
Kind code	B2
Filing date	Feb 28, 2022
Priority date	Feb 28, 2022
Publication date	Nov 14, 2023
Grant date	Nov 14, 2023

Abstract

In various examples, the decoding and upscaling capabilities of a client device are analyzed to determine encoding parameters and operations used by a content streaming server to generate encoded video streams. The quality of the upscaled content of the client device may be monitored by the streaming servers such that the encoding parameters may be updated based on the monitored quality. In this way, the encoding operations of one or more streaming servers may be more effectively matched to the decoding and upscaling abilities of one or more client devise such that an increased number of client devices may be served by the streaming servers.

First claim

What is claimed is:

1. A method comprising:
receiving data indicating one or more types of one or more video upscaling algorithms available to a client device;
selecting one or more encoding parameters for one or more streams of video data to the client device based at least on identifying, using the data, the one or more types of the one or more video upscaling algorithms, wherein a first configuration for the one or more encoding parameters corresponds to a neural network-based upscaling algorithm being identified and a second configuration for the one or more encoding parameters corresponds to a non-neural network-based upscaling algorithm being identified;
encoding one or more portions of the one or more streams of video data using the one or more encoding parameters based at least on the selecting; and
transmitting the one or more portions of the one or more streams to the client device, the transmitting causing the client device to generate an upscaled version of one or more video frames represented by the video data using at least one of the one or more video upscaling algorithms.

2. The method of claim 1 , wherein the selecting of the one or more encoding parameters selects a resolution for the one or more streams of video data and different types of video upscaling algorithms being identified corresponds to different resolutions for the video data.

3. The method of claim 1 , wherein the selecting of the one or more encoding parameters is further based at least on one or more visual characteristics of video content represented by the video data, and different types of video upscaling algorithms being identified corresponds to different encoding parameters for the one or more visual characteristics of the video data.

4. The method of claim 1 , further comprising updating the one or more encoding parameters based at least on a visual quality level of the upscaled version of the one or more video frames relative to one or more corresponding video frames encoded to generate the one or more streams of video data.

5. The method of claim 1 , where the encoding occurs during the one or more streams of video data to the client device.

6. The method of claim 1 , wherein the selecting of the one or more encoding parameters is further based at least on identifying one or more types of visual content is represented by the video data, and different types of video upscaling algorithms being identified corresponds to different encoding parameters for the one or more types of visual content represented by the video data.

7. The method of claim 1 , wherein a first configuration of the one or more encoding parameters for the video data corresponds to a first type of video upscaling algorithm being identified and a second configuration of the one or more encoding parameters corresponds to a second type of video upscaling algorithm being identified.

8. The method of claim 1 , wherein the one or more video upscaling algorithms include one or more video upscaling algorithms implemented using a convolutional neural network.

9. The method of claim 1 , wherein the selecting is further based at least on one or more of:
a minimum stream resolution requested by the client device, a minimum quality level requested by the client device, or a minimum bitrate requested by the client device.

10. A processor comprising:
one or more circuits to:
generate at least a portion of one or more streams of video data to a client device using one or more encoding parameters based at least on:
receiving data indicating one or more types of one or more video upscaling algorithms available to the client device;
determining the one or more encoding parameters for the one or more streams based at least on identifying, using the data, the one or more types of the one or more video upscaling algorithms, wherein a first configuration for the one or more encoding parameters corresponds to a neural network-based upscaling algorithm being identified and a second configuration for the one or more encoding parameters corresponds to a non-neural network-based upscaling algorithm being identified; and
encoding the at least a portion of the one or more streams of video data using the one or more encoding parameters based at least on the determining.

11. The processor of claim 10 , wherein the generating is at a resolution selected based at least on the one or more video upscaling algorithms.

12. The processor of claim 10 , wherein the one or more video upscaling algorithms indicate one or more upscaling ratios available to the client device and the at least a portion of the one or more streams of video data are generated based at least on the one or more upscaling ratios.

13. The processor of claim 10 , wherein the generating is further based at least on one or more characteristics of video content represented by the video data.

14. The processor of claim 10 , wherein the processor is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system for generating synthetic data;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.

15. A system comprising:
one or more processing units to execute operations comprising:
receiving data indicating one or more types of one or more video upscaling algorithms available to a client device;
selecting one or more encoding parameters for one or more streams of video data to the client device based at least on identifying, using the data, the one or more types of the one or more video upscaling algorithms, wherein a first configuration for the one or more encoding parameters corresponds to a neural network-based upscaling algorithm being identified and a second configuration for the one or more encoding parameters corresponds to a non-neural network-based upscaling algorithm being identified;
encoding one or more portions of the one or more streams of video data using the one or more encoding parameters based at least on the selecting; and
transmitting the one or more portions of the one or more streams to the client device, the transmitting causing the client device to generate an upscaled version of one or more video frames represented by the video data using at least one of the one or more video upscaling algorithms.

16. The system of claim 15 , wherein the selecting of the one or more encoding parameters selects a resolution for the one or more streams of video data.

17. The system of claim 15 , wherein the selecting of the one or more encoding parameters is further based at least on one or more visual characteristics of video content represented by the video data.

18. The system of claim 15 , further comprising updating the one or more encoding parameters based at least on a visual quality level of the upscaled version of the one or more video frames relative to one or more corresponding video frames encoded to generate the one or more streams of video data.

19. The system of claim 15 , wherein the selecting of the one or more en

Assignees

• Nvidia Corp

Inventors

• Sundareson Prabindh
• Pandhare Sachin
• Raikar Shyam

Classifications

• H04N19/59

  involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution · CPC title

• H04N19/146

  Data rate or code amount at the encoder output · CPC title

• H04N19/105

  Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction · CPC title

• H04L65/756 Primary

  adapting media to device capabilities · CPC title

• H04L65/762

  at the source  (reformatting of additional data in video distribution servers H04N21/2355) · CPC title