Dual detail video encoding with region specific quality control

Patent metadata

Field	Value
Publication number	US-12536706-B2
Application number	US-202318523298-A
Country	US
Kind code	B2
Filing date	Nov 29, 2023
Priority date	Nov 29, 2023
Publication date	Jan 27, 2026
Grant date	Jan 27, 2026

Abstract

Described herein are, among other things, techniques for implementing a dual detail encoding scheme in a distributed display system. At the host computer, an application may render a frame for a scene at a first resolution. The host computer may generate a two-dimensional (2D) array of pixels for the frame, encode the 2D array of pixels, and transmit the encoded pixel data to the display device. The 2D array of pixels comprises, for each eye, a copy of the frame downscaled to a second resolution less than the first resolution, and a copy of a high-attention subregion of the scene. The 2D array of pixels may be encoded using a technique that controls a quality parameter at compression unit granularity, such as macroblock or coding tree unit granularity. At the display device, the encoded pixel data is decoded to obtain the 2D array of pixels for the frame.

First claim

The invention claimed is:

1 . A computing system, comprising:
one or more processors; and one or more memories collectively storing computer-executable instructions that, when collectively executed by the one or more processors, cause the computing system to:
receive, from a graphics-rendering application, a left eye frame and a right eye frame for a scene, the left eye frame and the right eye frame rendered at a first resolution;
generate a two-dimensional (2D) array of pixels for the scene, the 2D array of pixels comprising:
a left eye base frame comprising the left eye frame downscaled to a second resolution less than the first resolution;
a right eye base frame comprising the right eye frame downscaled to the second resolution;
a left eye high-attention subregion comprising a portion of the left eye frame, the left eye high-attention subregion configured to be integrated with the left eye base frame when displayed by a display device; and
a right eye high-attention subregion comprising a portion of the right eye frame, the right eye high-attention subregion configured to be integrated with the right eye base frame when displayed by the display device;
select quality levels for each of a plurality of compression units of the 2D array of pixels;
encode the 2D array of pixels according to the selected quality levels; and
transmit the encoded 2D array of pixels to the display device.

2 . The computing system of claim 1 , wherein, to select quality levels for each of the plurality of compression units of the 2D array of pixels, the computing system:
for each of the left eye and right eye high-attention subregions, selects a first quality level for compression units in a central focus area of the high-attention subregion, and selects a second quality level for compression units in a peripheral area of the high-attention subregion surrounding the central focus area, the second quality level lower than the first quality level; and for each of the left eye and right eye base frames, selects a third quality level for compression units in an overlapping area of the base frame that is to be overlapped with the peripheral area of the respective high-attention subregion, and selects a fourth quality level for compression units in a background area of the base frame that surrounds the overlapping area of the base frame.

3 . The computing system of claim 2 , wherein, for each of the left eye and right eye high-attention subregions, the computing system selects quality levels such that there is a gradual transition between the first quality level and the second quality level.

4 . The computing system of claim 2 , wherein, for each of the left eye and right eye base frames, the computing system selects quality levels such that there is a gradual transition between the third quality level and the fourth quality level.

5 . The computing system of claim 1 , wherein the selection of the quality levels for the compression units comprises providing respective offset values to a quality parameter that is adaptively controlled by rate control logic of the computing system.

6 . The computing system of claim 1 , wherein, for each of the left eye and right eye base frames, the computing system selects a low quality level for an area of the base frame determined to not be visible to a user of the display device.

7 . The computing system of claim 1 , wherein, for each of the left eye and right eye high-attention subregions, the computing system selects a low quality level for an area of the high-attention subregion determined to not be visible to a user of the display device.

8 . The computing system of claim 1 , wherein the computing system determines each of the left eye and right eye high-attention subregions based at least in part on eye tracking data received from the display device.

9 . The computing system of claim 1 , wherein the computer-executable instructions, when executed by the one or more processors, further cause the computing system to:
determine whether respective positions of the left eye and right eye high-attention subregions are based at least in part on eye tracking data received from the display device; and responsive to a determination that the positions of the left eye and right eye high-attention subregions are based at least in part on eye tracking data received from the display device, select a quality level of a background area of the frame to be a quality level that is lower than a quality level that is selected when it is determined that the positions of the left eye and right eye high-attention subregions are not based at least in part on eye tracking data received from the display device.

10 . The computing system of claim 1 , wherein:
pixels of the left eye high-attention subregion correspond one-to-one with original pixels of the portion of the left eye frame; and pixels of the right eye high-attention subregion correspond one-to-one with original pixels of the portion of the right eye frame.

11 . The computing system of claim 1 , wherein generating the 2D array of pixels comprises arranging the pixels in a vertical packing layout having:
one of the left eye high-attention subregion and the right eye high-attention subregion at a top of the 2D array of pixels; one of the left eye base frame and the right eye base frame below the one of the left eye high-attention subregion and the right eye high-attention subregion; the other of the left eye high-attention subregion and the right eye high-attention subregion below the one of the left eye base frame and the right eye base frame; and the other of the left eye base frame and the right eye base frame at a bottom of the 2D array of pixels.

12 . The computing system of claim 1 , further comprising a transceiver, wherein transmitting the encoded 2D array of pixels to the display device comprises wirelessly transmitting, via the transceiver, one or more data packets carrying the encoded 2D array of pixels.

13 . A method comprising:
receiving, from a graphics-rendering application, a left eye frame and a right eye frame for a scene, the left eye frame and the right eye frame rendered at a first resolution; generating a two-dimensional (2D) array of pixels for the scene, the 2D array of pixels comprising:
a left eye base frame comprising the left eye frame downscaled to a second resolution less than the first resolution;
a right eye base frame comprising the right eye frame downscaled to the second resolution;
a left eye high-attention subregion comprising a portion of the left eye frame, the left eye high-attention subregion configured to be integrated with the left eye base frame when displayed by a display device; and
a right eye high-attention subregion comprising a portion of the right eye frame, the right eye high-attention subregion configured to be integrated with the right eye base frame when displayed by the display device;
selecting quality levels for each of a plurality of compression units of the 2D array of pixels; encoding the 2D array of pixels according to the selected quality levels; and transmitting the encoded 2D array of pixels to the display device.

14 . The method of claim 13 , wherein selecting quality levels for each of the plurality of compression units of the 2D array of pixels comprises:
for each of the left eye and right eye high-attention subregions, selecting a first quality level for compression units in a central focus area of the high-attention subregion, and selecting a second quality level for compression units in a peripheral area of the high-attention subregion surrounding the central f

Assignees

• Valve Corp

Inventors

• Jackson Benedict Joseph
• Lohr Charles

Classifications

• G06T3/40

  Scaling of whole images or parts thereof, e.g. expanding or contracting · CPC title

• G06F3/147

  using display panels · CPC title

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• A63F13/52 Primary

  involving aspects of the displayed game scene · CPC title

• G06T9/00 Primary

  Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00) · CPC title