Video streaming scaling using virtual resolution adjustment

Patent metadata

Field	Value
Publication number	US-12368908-B2
Application number	US-202217724360-A
Country	US
Kind code	B2
Filing date	Apr 19, 2022
Priority date	Apr 19, 2022
Publication date	Jul 22, 2025
Grant date	Jul 22, 2025

Abstract

In various examples, network conditions associated with a video stream are observed over time to determine a virtual scaling factor that may be applied to the images of one or more frames of the video stream to generate scaled images that may be appended with a padding region that maintains the original resolution of the video and/or video stream such that a receiving device may crop the padding region from the received video stream during the decoding process without restarting or including additional intra-coded frames.

First claim

What is claimed is:

1. A method comprising:
selecting a target resolution for a stream of images corresponding to frames of a video stream based at least on one or more conditions associated with the video stream, the target resolution being a different resolution than each of a current resolution being used to encode the stream of images to the video stream and a resolution of the frames;
based at least on the selecting, transitioning the stream of images from the current resolution to the target resolution over iterative steps, wherein each step of two or more of the iterative steps includes:
updating a step size for an adjustment of the stream of images to an intermediate resolution,
performing the adjustment of the stream of images to the intermediate resolution,
padding at least one image from the stream of images with one or more pixels, and
encoding the at least one image that is padded with the one or more pixels to one or more of the frames of the video stream; and

transmitting the one or more of the frames over a network using one or more packets.

2. The method of claim 1 , wherein the transmitting includes storing, using the one or more packets, data representing one or more parameters indicating one or more of a resolution of the at least one image within a frame or a location of the at least one image within a frame.

3. The method of claim 1 , further comprising:
causing a decoding of the one or more packets to one or more decoded frames, the decoding including scaling, from the target resolution to the resolution of the frames, one or more images corresponding to the one or more decoded frames; and
causing a display of the one or more decoded frames.

4. The method of claim 3 , wherein the adjustment uses a different step size for a vertical resolution dimension than for a horizontal resolution dimension.

5. The method of claim 1 , wherein the padding the at least one image comprises padding the at least one image with a padding region that includes the one or more pixels, wherein the one or more pixels have color values that are configured to reduce or eliminate one or more visual artifacts caused by decoding the at least one image at a boundary with the padding region.

6. The method of claim 1 , wherein the adjustment aligns the intermediate resolution to a multiple of a block size of an encoding algorithm used in the encoding of the at least one image.

7. The method of claim 1 , wherein the one or more conditions correspond to a target bitrate associated with the video stream.

8. The method of claim 1 , wherein in response to a decrease in a complexity value of video content in the stream of images, the selecting of the target resolution is to increase the current resolution.

9. A system comprising:
one or more processing units to perform operations comprising:
determining a target resolution for a stream of images corresponding to frames of a video stream based at least on one or more conditions associated with the video stream;
based at least on the determining the target resolution, transitioning the stream of images from a current resolution to the target resolution over each step of two or more of the iterative steps includes:
updating a step size for an adjustment of the stream of images to an intermediate resolution,
performing the adjustment of the stream of images to the intermediate resolution,
padding at least one image from the stream of images with one or more pixels, and
encoding the at least one image that is padded with the one or more pixels to one or more of the frames of the video stream; and

transmitting, using one or more packets, the one or more of the frames to a device.

10. The system of claim 9 , wherein encoding the one or more of the frames includes encoding data representing one or more parameters indicating one or more of a size of the one or more pixels or a location of the one or more pixels within the one or more of the frames.

11. The system of claim 9 , wherein the operations further comprise padding an image of the at least one image with the one or more pixels, wherein the one or more pixels have color values that are selected to match edge pixels of the image.

12. The system of claim 9 , wherein the operations further comprise causing the device to crop a frame of the one or more of the frames, upscale the cropped frame, and display the upscaled frame.

13. The system of claim 9 , wherein the one or more conditions correspond to a target bitrate associated with the video stream.

14. The system of claim 9 , wherein the adjustment comprises scaling the one or more of the frames by an amount that is aligned to a multiple of a pixel size defining a video compression processing boundary used in the encoding.

15. The system of claim 9 , wherein the one or more conditions correspond to a complexity value indicating a level and magnitude of complexity in encoding the video stream.

16. The system of claim 9 , wherein the operations further comprise, upon completion of the transitioning, maintaining the target resolution for a plurality of the frames until a new target resolution is selected for the stream of images.

17. The system of claim 9 , wherein the system is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system for generating synthetic data;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.

18. At least one processor comprising:
one or more circuits to:
select a target resolution for a stream of images corresponding to frames of a video stream based at least on one or more conditions associated with the video stream, and
based at least on the selecting, to transition the stream of images from a current resolution to the target resolution over iterative steps, wherein each step of two or more of the iterative steps includes:
updating a step size for an adjustment of the stream of images to an intermediate resolution,
performing the adjustment of the stream of images to the intermediate resolution,
padding at least one image from the stream of images with one or more pixels, and
encoding the at least one image that is padded with the one or more pixels to one or more of the frames of the video stream.

19. The at least one processor of claim 18 , wherein the one or more circuits are to store, in one or more packets representing the one or more of the frames, data representing one or more parameters indicating one or more of a size of the at least one image within one or more of the frames or a location of the at least one image within the one or more of the frames.

20. The at least one processor of claim 18 , wherein the at least one processor is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing light transport simulation;
a system fo

Assignees

• Nvidia Corp

Inventors

• Gopalakrishna Rao Bhavani
• Cook Nicholas
• Howard James

Classifications

• H04N21/440263

  by altering the spatial resolution, e.g. for displaying on a connected PDA · CPC title

• H04L67/131

  Protocols for games, networked simulations or virtual reality · CPC title

• H04N19/59

  involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution · CPC title

• H04N19/29

  involving scalability at the object level, e.g. video object layer [VOL] · CPC title

• H04N21/234363 Primary

  by altering the spatial resolution, e.g. for clients with a lower screen resolution · CPC title