Pre-processing for video noise reduction

Patent metadata

Field	Value
Publication number	US-2017195591-A1
Application number	US-201614988630-A
Country	US
Kind code	A1
Filing date	Jan 5, 2016
Priority date	Jan 5, 2016
Publication date	Jul 6, 2017
Grant date	—

Abstract

Embodiments of the present invention are directed to methods and systems for performing automatic noise reduction in video. According to one aspect of the invention, a video noise-reducing system is provided consisting of a noise estimator, a motion classifier, two stages of filters, each including a spatial and temporal filter, and a combiner. The system adapts to noise level and to scene content to find at each location in the image a balance of noise reduction and detail preservation. Temporal Infinite Impulse Response (IIR) filtering provides a high level of detail-preserving noise reduction where motion allows, while non linear spatial filtering provides edge-preserving noise reduction in areas where the temporal filter would introduce motion artifacts. A spatial-temporal combiner provides smooth transition and balance between the two filtering modes; this block also enables use of external cues to produce a visually pleasing output based on ambient conditions.

First claim

What is claimed is:

1 . A method for pre-processing captured video data, the method comprising:
receiving video data captured by a camera device, the video data comprising a plurality of frames; calculating a local noise level estimate of a current frame of the plurality of frames; determining a motion classification of a plurality of regions in the current frame of the captured video data based on the local noise level estimate; and performing noise reduction on the current frame based on the local noise level estimation and the motion classification.

2 . The method according to claim 1 , wherein the plurality of regions comprises a plurality of non-overlapping blocks in the current frame.

3 . The method according to claim 2 , wherein the calculating a local noise level estimate comprises hierarchically identifying a plurality of homogenous areas within the plurality of non-overlapping blocks.

4 . The method according to claim 3 , wherein the identifying the plurality of homogenous areas is based on calculating a strength of a plurality of directional activity operators.

5 . The method according to claim 4 , wherein a directional activity operator of the plurality of directional activity operators comprises a mean of absolute deviation of the plurality of homogenous areas.

6 . The method according to claim 1 , wherein the calculating the local noise level estimate comprises performing adaptive spatial filtering on the current frame.

7 . The method according to claim 6 , wherein the performing spatial filtering comprises applying the weighted sum of the plurality of homogenous areas to a first homogenous area in the current frame.

8 . The method according to claim 5 , wherein the calculating the local noise level estimate comprises performing adaptive temporal filtering on the plurality of frames.

9 . The method according to claim 8 , wherein the performing temporal filtering comprises applying a first-order Infinite Impulse Response filter over the current frame and a previous frame of the plurality of frames.

10 . The method according to claim 5 , wherein the calculating the local noise level estimate comprises processing a plurality of filtering stages, each filtering stage of the plurality of filtering stages comprising an adaptive spatial filter and an adaptive temporal filter.

11 . The method according to claim 1 , wherein the determining a motion classification comprises calculating a sum of square difference (SSD) of the plurality of regions.

12 . The method according to claim 1 , wherein the performing the noise reduction on the current frame of the captured video data is further based on one or more external cues.

13 . The method according to claim 12 , wherein an external cue of the one or more external cues comprises at least one external cue from the group consisting of:
a data corresponding to lighting conditions of the current frame; a user input; a measurement from an image signal processor (ISP) of the camera device.

14 . The method according to claim 1 , wherein the determining a motion classification of a plurality of regions in the current frame further comprises determining a motion classification of the plurality of regions in a plurality of previous frames.

15 . A system for performing noise-reduction in video, the system comprising:
a camera device comprising an image sensor configured to generate captured video data comprising a plurality of frames; a memory device configured to store a plurality of programmed instructions; a processor configured to execute the plurality of programmed instructions in order to implement:
a noise estimator for computing a local noise level estimate for a current frame of the plurality of frames;
a motion classifier for determining a motion classification of a plurality of regions in the current frame of the captured video data based on the local noise level estimate; and
a combiner configured to perform noise reduction on the current frame based on the local noise level estimate and the motion classification.

16 . The system according to claim 15 , wherein the noise estimator is further configured to perform adaptive spatial filtering on the current frame by applying the weighted sum of a plurality of homogenous areas to a first homogenous area in the current frame.

17 . The system according to claim 15 , wherein the noise estimator is further configured to perform adaptive temporal filtering on the local noise level estimate by applying a first-order Infinite Impulse Response filter over the current frame and a previous frame of the plurality of frames.

18 . The system according to claim 15 , wherein the noise estimator is configured to apply a plurality of filtering stages, each filtering stage of the plurality of filtering stages comprising an adaptive spatial filter and an adaptive temporal filter.

19 . The system according to claim 15 , wherein the combiner is further configured to perform the noise reduction based on one or more external cues.

20 . The system according to claim 19 , wherein an external cue of the one or more external cues comprises at least one external cue from the group consisting of:
a data corresponding to lighting conditions of the current frame; a user input; a measurement from an image signal processor (ISP) of the camera device.

21 . A non-transitory computer-readable medium comprising a plurality of programmed instructions, which, when executed by a processor in a computing device, is configured to perform noise reduction on frames of video data from an image capture device, the plurality of programmed instructions comprising:
instructions to receive video data captured by an image capture device, the video data comprising a plurality of frames; instructions to calculate a local noise level estimate of a current frame of the plurality of frames; instructions to determine a motion classification of a plurality of regions in the current frame of the captured video data based on the local noise level estimate; and instructions to perform noise reduction on the current frame based on the local noise level estimation and the motion classification.

Assignees

• Nvidia Corp

Inventors

• Avadhanam Niranjan
• Viscito Eric
• Allagadapa Varun
• Kottana Thrinadh

Classifications

• H04B1/3833 Primary

  Hand-held transceivers · CPC title

• H04N23/683

  performed by a processor, e.g. controlling the readout of an image memory · CPC title

• H04N23/6811

  based on the image signal · CPC title

• H04N5/374

  Electricity · mapped topic

• H04N5/23254

  Electricity · mapped topic