High dynamic range support for legacy applications

Patent metadata

Field	Value
Publication number	US-11954830-B2
Application number	US-202017125705-A
Country	US
Kind code	B2
Filing date	Dec 17, 2020
Priority date	Dec 17, 2020
Publication date	Apr 9, 2024
Grant date	Apr 9, 2024

Abstract

High dynamic range (HDR) support is provided for legacy application programs, such as games that are configured to display standard dynamic range (SDR) frames. HDR frames may be synthesized without modifying the legacy application program. The buffer creation process of the legacy application program is intercepted and modified before creation of the SDR format buffer so that the buffer is configured to use an HDR format. A location of an intermediate buffer storing HDR rendered data is determined by intercepting and analyzing graphics driver calls in a command stream produced by the legacy application program. The HDR rendered data is combined with user interface content extracted from the SDR frames. Additionally, any post processing effects used by the legacy application program to produce the SDR frames may be predicted and applied to the HDR rendered data to synthesize the HDR frames for display on a modern HDR display device.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
receiving one or more commands to produce standard dynamic range (SDR) frames, wherein the one or more commands to produce the SDR frames include one or more commands for producing, for at least one SDR frame of the SDR frames, an intermediate raw high dynamic range (HDR) frame to be converted to the at least one SDR frame; and
generating, for the at least one SDR frame, at least one enhanced HDR frame by:
obtaining user interface content that is included in the at least one SDR frame; and
applying the obtained user interface content and processing the intermediate raw HDR frame by a neural network model to generate an enhanced HDR frame, the enhanced HDR frame comprising the user interface content and preserving at least one of a luminance or a dynamic range of at least a portion of the intermediate raw HDR frame.

2. The computer-implemented method of claim 1 , wherein the obtaining the user interface content that is included in the at least one SDR frame comprises processing, by the neural network model, the at least one SDR frame to obtain at least the user interface content.

3. The computer-implemented method of claim 2 , wherein the processing the intermediate raw HDR frame by the neural network model comprises:
applying style information to the intermediate raw HDR frame, wherein the style information is obtained from the processing, by the neural network model, the at least one SDR frame.

4. The computer-implemented method of claim 2 , wherein the processing the intermediate raw HDR frame by the neural network model comprises:
applying post processing effects to the intermediate raw HDR frame, wherein the style information is obtained from the processing, by the neural network model, the at least one SDR frame.

5. The computer-implemented method of claim 1 , wherein the at least one SDR frame and the intermediate raw HDR frame are produced during execution of an application program that is configured to produce the at least one SDR frame for display by an SDR-compatible display device.

6. The computer-implemented method of claim 5 , wherein the user interface content is not included in the intermediate raw HDR frame.

7. The computer-implemented method of claim 1 , wherein the obtaining the user interface content that is included in the at least one SDR frame comprises extracting, by a second neural network model, the user interface content from the at least one SDR frame.

8. The computer-implemented method of claim 1 , further comprising displaying the enhanced HDR frame by an HDR-compatible display device.

9. The computer-implemented method of claim 1 , further comprising:
extracting buffer setup information from a command stream generated by an application program that, when executed, produces the at least one SDR frame and the intermediate raw HDR frame; and
inserting, into the command stream, buffer setup commands configured to set up a buffer configured to store the enhanced HDR frame.

10. The computer-implemented method of claim 1 , further comprising:
extracting display setup information from a command stream generated by an application program that, when executed, produces the at least one SDR frame and the intermediate raw HDR frame; and
modifying display setup commands to configure an HDR display device to display the enhanced HDR frame instead of the at least one SDR frame.

11. A system, comprising:
a memory configured to store an intermediate raw high dynamic range (HDR) frame to be converted to at least one SDR frame; and
a processor configured to:
receive one or more commands to produce standard dynamic range (SDR) frames, wherein the one or more commands to produce the SDR frames include one or more commands for producing, for at least one SDR frame of the SDR frames, the intermediate raw high dynamic range (HDR) frame; and
generate, for the at least one SDR frame, at least one enhanced HDR frame by:
obtaining user interface content that is included in the at least one SDR frame; and
applying the obtained user interface content and processing the intermediate raw HDR frame by a neural network model to generate an enhanced HDR frame, the enhanced HDR frame comprising the user interface content and preserving at least one of a luminance or a dynamic range of at least a portion of the intermediate raw HDR frame.

12. The system of claim 11 , wherein the neural network model is configured to process the SDR frame to obtain at least the user interface content.

13. The system of claim 12 , wherein the neural network model is configured to process the SDR frame to further obtain style information, and wherein the neural network model is further configured to apply the style information to the intermediate raw HDR frame.

14. The system of claim 12 , wherein the neural network model is configured to process the SDR frame to further obtain post processing effects, and wherein the neural network model is further configured to apply the post processing effects to the intermediate raw HDR frame.

15. The system of claim 11 , wherein the at least one SDR format frame and the intermediate raw HDR frame are produced during execution of an application program that is configured to produce the at least one SDR frame for display by an SDR-compatible display device.

16. The system of claim 15 , wherein the user interface content is not included in the intermediate raw HDR frame.

17. The system of claim 11 , wherein the obtaining the user interface content that is included in the at least one SDR frame comprises extracting, by a second neural network models, the user interface content from the at least one SDR frame.

18. The system of claim 11 , further comprising displaying the enhanced HDR frame by an HDR-compatible display device.

19. The system of claim 11 , wherein the processor is further configured to:
extract buffer setup information from a command stream generated by an application program that, when executed, produces the at least one SDR frame and the intermediate raw HDR frame; and
insert, into the command stream, buffer setup commands configured to set up a buffer configured to store the enhanced HDR frame.

20. A non-transitory computer-readable media storing computer instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:
receiving one or more commands to produce standard dynamic range (SDR) frames, wherein the one or more commands to produce the SDR frames include one or more commands for producing, for at least one SDR frame of the SDR frames, an intermediate raw high dynamic range (HDR) frame to be converted to the at least one SDR frame; and
generating, for the at least one SDR frame, at least one enhanced HDR frame by:
obtaining user interface content that is included in the at least one SDR frame; and
applying the obtained user interface content and processing the intermediate raw HDR frame by a neural network model to generate an enhanced HDR frame, the enhanced HDR frame comprising the obtained user interface content and preserving at least one of a luminance and or a dynamic range of at least a portion of the corresponding intermediate raw HDR frame.

Assignees

• Nvidia Corp

Inventors

• Kumar Shaveen

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06T5/009 Primary

  Physics · mapped topic

• G06N3/08

  Learning methods · CPC title

• G06T5/20

  using local operators · CPC title