Unified optimization method for end-to-end camera image processing for translating a sensor captured image to a display image

Patent metadata

Field	Value
Publication number	US-9905196-B2
Application number	US-201615276626-A
Country	US
Kind code	B2
Filing date	Sep 26, 2016
Priority date	Jan 21, 2014
Publication date	Feb 27, 2018
Grant date	Feb 27, 2018

Abstract

A computer implemented method of determining a latent image from an observed image is disclosed. The method comprises implementing a plurality of image processing operations within a single optimization framework, wherein the single optimization framework comprises solving a linear minimization expression. The method further comprises mapping the linear minimization expression onto at least one non-linear solver. Further, the method comprises using the non-linear solver, iteratively solving the linear minimization expression in order to extract the latent image from the observed image, wherein the linear minimization expression comprises: a data term, and a regularization term, and wherein the regularization term comprises a plurality of non-linear image priors.

First claim

What is claimed is:

1. A computer implemented method of generating a latent image from an observed image, said method comprising:
implementing a plurality of image processing operations within an optimization framework, wherein said optimization framework comprises solving a linear minimization expression;
mapping said linear minimization expression onto at least one non-linear solver; and
using said non-linear solver, iteratively solving said linear minimization expression in order to generate said latent image from said observed image.

2. The method of claim 1 , wherein said linear minimization expression comprises a regularization term, wherein said regularization term comprises a plurality of non-linear image priors.

3. The method of claim 2 , wherein said regularization term comprises three image priors, wherein said three image priors are selected from the group consisting of: total variation image prior, a denoising image prior, and a cross-channel gradient correlation image prior.

4. The method of claim 3 , wherein said denoising prior can be selected from a group consisting of: BM3D type, NLM type, and sliding DCT type.

5. The method of claim 1 , wherein said single optimization framework is implemented to execute on a graphics processing unit (GPU).

6. The method of claim 1 , wherein said non-linear solver is selected from a group consisting of: a primal-dual solver and an ADMM solver.

7. The method of claim 1 , wherein said optimization framework comprises a forward image formation model, wherein said forward image formation model comprises a sequence of independent linear transformations.

8. The method of claim 7 , wherein said forward image formation model can be selected from a group consisting of: joint Bayer demosaicking and denoising, interlaced HDR reconstruction, image fusion from color camera arrays, super-resolution, and joint image stack denoising and demosaicking.

9. A non-transitory computer-readable storage medium having stored thereon, computer executable instructions that, if executed by a computer system cause the computer system to perform a method of determining a latent image from an observed image, said method comprising:
implementing a plurality of image processing operations within an optimization framework, wherein said optimization framework comprises solving a linear minimization expression;
mapping said linear minimization expression onto at least one non-linear solver; and
using said non-linear solver, iteratively solving said linear minimization expression in order to generate said latent image from said observed image.

10. The non-transitory computer-readable medium of claim 9 , wherein said linear minimization expression comprises a regularization term, wherein said regularization term comprises a plurality of non-linear image priors.

11. The non-transitory computer-readable medium as described in claim 10 , wherein said regularization term comprises three image priors, wherein said three image priors are selected from the group consisting of: total variation image prior, a denoising image prior, and a cross-channel gradient correlation image prior.

12. The non-transitory computer-readable medium as described in claim 11 , wherein said denoising prior can be selected from a group consisting of: BM3D type, NLM type, and sliding DCT type.

13. The non-transitory computer-readable medium as described in claim 9 , wherein said optimization framework is implemented to execute on a graphics processing unit (GPU).

14. The non-transitory computer-readable medium as described in claim 9 , wherein said non-linear solver is selected from a group consisting of: a primal-dual solver and an ADMM solver.

15. The non-transitory computer-readable medium as described in claim 9 , wherein said optimization framework comprises a forward image formation model, wherein said forward image formation model comprises a sequence of independent linear transformations.

16. The non-transitory computer-readable medium as described in claim 15 , wherein said forward image formation model can be selected from a group consisting of: joint Bayer demosaicking and denoising, interlaced HDR reconstruction, image fusion from color camera arrays, super-resolution, and joint image stack denoising and demosaicking.

17. A system for providing a latent image from an observed image, said system comprising:
a memory storing information related to an image construction framework;
a processor coupled to said memory, said processor operable to implement a method of providing a latent image from an observed image, said method comprising:
integrating a plurality of image processing operations within an optimization framework, wherein said optimization framework comprises solving a linear minimization equation;
mapping said linear minimization equation onto at least one non-linear solver; and
using said non-linear solver, iteratively solving said linear minimization equation in order to generate said latent image from said observed image.

18. The system of claim 17 , wherein said linear minimization equation comprises: a data term, a regularization term.

19. The system of claim 17 , wherein said optimization framework is implemented to execute on a graphics processing unit (GPU).

20. The system of claim 17 , wherein said non-linear solver is selected from a group consisting of: a primal-dual solver and an ADMM solver.

Assignees

• Nvidia Corp

Inventors

• Pajak Dawid Stanislaw
• Heide Felix
• Reddy Nagilla Dikpal
• Rouf Mushfiqur
• Kautz Jan
• Pulli Kari
• Gallo Orazio

Classifications

• G09G5/02 Primary

  characterised by the way in which colour is displayed {(details of colour display specific for CRTs G09G1/28; specific for flat matrix panels other than liquid crystal displays G09G3/2003; specific for liquid crystal displays G09G3/3607)} · CPC title

• G09G5/363 Primary

  Graphics controllers · CPC title

• G06T3/4015

  Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns · CPC title

• G09G2320/066

  for control of contrast · CPC title

• G09G2320/0238

  Improving the black level · CPC title