Temporally coherent local tone mapping of HDR video

Patent metadata

Field	Value
Publication number	US-9361679-B2
Application number	US-201414121091-A
Country	US
Kind code	B2
Filing date	Jul 28, 2014
Priority date	Jul 28, 2014
Publication date	Jun 7, 2016
Grant date	Jun 7, 2016

Abstract

Approaches are described for filtering a first image frame in a sequence of image frames. A tone-mapping pipeline applies a spatiotemporal filter to each pixel of the first image frame, based on a forward optical flow and a backward optical flow, to produce a base layer image. The tone-mapping pipeline applies a temporal filter to each pixel of the first image frame, based on the forward optical flow and the backward optical flow, to produce a temporally filtered frame. The tone-mapping pipeline produces a detail layer image based on the base layer image and the temporally filtered frame.

First claim

What is claimed is:

1. A method for filtering a first image frame in a sequence of image frames, comprising:
applying a spatiotemporal filter to each pixel of the first image frame, based on a forward optical flow and a backward optical flow, to produce a base layer image;
applying a first temporal filter to each pixel of the first image frame, based on the forward optical flow and the backward optical flow, to produce a temporally filtered frame, wherein at least one of the spatiotemporal filter and the first temporal filter comprises a diffusion estimate term comprising a weighted average at a first pixel position and a fidelity term that introduces a bias towards the first image frame; and
producing a detail layer image based on the base layer image and the temporally filtered frame.

2. The method of claim 1 , wherein the bias is controlled by a parameter that is multiplied by the fidelity term.

3. The method of claim 2 , wherein the parameter is equal to one.

4. The method of claim 1 , wherein the spatiotemporal filter and the first temporal filter are iteratively applied to the first image frame for a first quantity of iterations.

5. The method of claim 4 , wherein each iteration in the first quantity of iterations is performed in parallel with every other iteration in the first quantity of iterations.

6. The method of claim 1 , wherein applying the spatiotemporal filter comprises:
applying a spatial filter to each pixel of the first image frame to produce a partial base layer image; and
applying a second temporal filter to each pixel of the partial base layer image, based on the forward optical flow and the backward optical flow, to produce the base layer Image.

7. The method of claim 1 , further comprising storing an animation file on a computer-readable medium, wherein the animation file comprises the output of the applying the spatiotemporal filter, applying the first temporal filter, and producing steps.

8. A method for filtering a first image frame in a sequence of image frames, comprising:
applying a spatiotemporal filter to each pixel of the first image frame, based on a forward optical flow and a backward optical flow, to produce a base layer image, wherein applying the spatiotemporal filter comprises:
applying a spatial filter to each pixel of the first image frame to produce a partial base layer image, wherein applying the spatial filter comprises computing a horizontal permeability between a first pixel in the first image frame and another pixel on the same horizontal row of pixels, and
applying a first temporal filter to each pixel of the partial base layer image, based on the forward optical flow and the backward optical flow, to produce the base layer image;

applying a second temporal filter to each pixel of the first image frame, based on the forward optical flow and the backward optical flow, to produce a temporally filtered frame; and
producing a detail layer image based on the base layer image and the temporally filtered frame.

9. A method for filtering a first image frame in a sequence of image frames, comprising:
applying a spatiotemporal filter to each pixel of the first image frame, based on a forward optical flow and a backward optical flow, to produce a base layer image, wherein applying the spatiotemporal filter comprises:
applying a spatial filter to each pixel of the first image frame to produce a partial base layer image, wherein applying the spatial filter comprises computing a vertical permeability between a first pixel in the first image frame and another pixel on the same vertical column of pixels, and
applying a first temporal filter to each pixel of the partial base layer image, based on the forward optical flow and the backward optical flow, to produce the base layer image;

applying a second temporal filter to each pixel of the first image frame, based on the forward optical flow and the backward optical flow, to produce a temporally filtered frame; and
producing a detail layer image based on the base layer image and the temporally filtered frame.

10. A method for filtering a first image frame in a sequence of image frames, comprising:
applying a spatiotemporal filter to each pixel of the first image frame, based on a forward optical flow and a backward optical flow, to produce a base layer image, wherein applying the spatiotemporal filter comprises:
applying a first temporal filter to each pixel of the first image frame, based on the forward optical flow and the backward optical flow, to produce a temporally filtered frame, wherein applying the first temporal filter comprises computing a temporal permeability between a first pixel in the first image frame and at least one of a first corresponding pixel in a first warped image frame of a previous image frame and a second corresponding pixel in a second warped image frame of a later image frame; and
producing a detail layer image based on the base layer image and the temporally filtered frame.

11. A non-transitory computer-readable storage medium including instructions that, when executed by a processor, cause the processor to filter a first image frame in a sequence of image frames, by performing an operation comprising:
applying a spatiotemporal filter to each pixel of the first image frame, based on a forward optical flow and a backward optical flow, to produce a base layer image;
applying a first temporal filter to each pixel of the first image frame, based on the forward optical flow and the backward optical flow, to produce a temporally filtered frame, wherein at least one of the spatiotemporal filter and the first temporal filter comprises a diffusion estimate term comprising a weighted average at a first pixel position and a fidelity term that introduces a bias towards the first image frame; and
producing a detail layer image based on the base layer image and the temporally filtered frame.

12. The non-transitory computer-readable storage medium of claim 11 , wherein the bias is controlled by a parameter that is multiplied by the fidelity term.

13. The non-transitory computer-readable storage medium of claim 12 , wherein the parameter is equal to one.

14. The non-transitory computer-readable storage medium of claim 11 , wherein the spatiotemporal filter and the first temporal filter are iteratively applied to the first image frame for a first quantity of iterations.

15. The non-transitory computer-readable storage medium of claim 14 , wherein each iteration in the first quantity of iterations is performed in parallel with every other iteration in the first quantity of iterations.

16. The non-transitory computer-readable storage medium of claim 11 , wherein applying the spatiotemporal filter comprises:
applying a spatial filter to each pixel of the first image frame to produce a partial base layer image; and
applying a second temporal filter to each pixel of the partial base layer image, based on the forward optical flow and the backward optical flow, to produce the base layer image.

17. The non-transitory computer-readable storage medium of claim 11 , wherein the operation further comprises storing an animation file on a computer-readable medium, wherein the animation file comprises the output of the applying the spatiotemporal filter, applying the first temporal filter, and producing steps.

Assignees

• Disney Entpr Inc
• Eth Zurich Eidgenoessische Tech Hochschule Zurich

Inventors

• Aydin Tunc Ozan
• Croci Simone
• Smolic Aljosa Aleksej Andrej
• Stefanoski Nikolce
• Gross Markus

Classifications

• H04N19/117 Primary

  Filters, e.g. for pre-processing or post-processing (sub-band filter banks H04N19/635) · CPC title

• G06T2207/20024

  Filtering details · CPC title

• G06T2207/20208

  High dynamic range [HDR] image processing · CPC title

• G06T2207/20182

  Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering · CPC title

• G06T5/40 Primary

  using histogram techniques · CPC title