Systems and methods for estimating sky light probes for outdoor images

Patent metadata

Field	Value
Publication number	US-9860453-B2
Application number	US-201514614214-A
Country	US
Kind code	B2
Filing date	Feb 4, 2015
Priority date	Nov 26, 2014
Publication date	Jan 2, 2018
Grant date	Jan 2, 2018

Abstract

Methods and systems for estimating HDR sky light probes for outdoor images are disclosed. A precaptured sky light probe database is leveraged. The database includes a plurality of HDR sky light probes captured under a plurality of different illumination conditions. A HDR sky light probe is estimated from an outdoor image by fitting a three dimensional model to an object of interest in the image and solving an inverse optimization lighting problem for the 3D model where the space of possible HDR sky light probes is constrained by the HDR sky light probes of the database.

First claim

What is claimed is:

1. A system, comprising:
a camera for capturing an outdoor image;
one or more processors; and
one or more non-transitory computer-readable mediums operatively coupled to at least one of the one or more processors and having instructions stored thereon that, when executed by at least one of the one or more processors, cause at least one of the one or more processors to:
fit a three-dimensional (3D) model to an object of interest in the outdoor image; and
estimate a sky light probe for the outdoor image by applying an inverse optimization lighting algorithm based on the fitted 3D model, wherein applying an inverse optimization lighting algorithm based on the fitted 3D model comprises constraining lighting of the optimization light algorithm to a set of a plurality of precaptured sky light probes captured under a plurality of sky illumination conditions.

2. The system of claim 1 , wherein illumination parameters of the estimated sky light probe comprise sun illumination parameters based on a sun model and sky illumination parameters based on a sky model, wherein the sun illumination parameters include a position of the sun.

3. The system of claim 1 , wherein applying an inverse optimization light algorithm comprises optimizing a directional lighting model over the space of precaptured sky light probes to estimate a sun position.

4. The system of claim 3 , wherein applying an inverse optimization light algorithm comprises:
initializing a hemispherical lighting model based on the estimated sun position; and
optimizing the hemispherical lighting model over the space of precaptured sky light probes.

5. The system of claim 1 , wherein the one or more non-transitory computer-readable mediums when executed by at least one of the one or more processors, further cause at least one of the one or more processors to:
insert a virtual three-dimensional (3D) object into the outdoor image; and
relight the virtual 3D object using the estimated sky light probe.

6. The system of claim 1 , wherein:
the captured outdoor image is a low dynamic range (LDR) image;
the precaptured sky light probes are high dynamic range (HDR) sky light probes; and
the estimated sky light probe is an HDR sky light probe.

7. The system of claim 2 , wherein the sun model is based on a double exponential sun model.

8. The system of claim 7 , wherein the sky model is based on a Preetham sky model.

9. The system of claim 1 , wherein the object of interest is a human face.

10. The system of claim 1 , further comprising a storage that stores the precaptured sky light probes.

11. A method, comprising:
capturing an outdoor image with a camera;
fitting a three-dimensional (3D) model to an object of interest in the outdoor image; and
estimating a sky light probe for the outdoor image by applying an inverse optimization lighting algorithm based on the fitted 3D model, wherein applying an inverse optimization lighting algorithm based on the fitted 3D model comprises constraining lighting of the optimization light algorithm to a set of a plurality of precaptured sky light probes captured under a plurality of sky illumination conditions.

12. The method of claim 11 , wherein illumination parameters of the estimated sky light probe comprise sun illumination parameters based on a sun model and sky illumination parameters based on a sky model, wherein the sun illumination parameters include a position of the sun.

13. The method of claim 11 , wherein applying an inverse optimization light algorithm comprises optimizing a directional lighting model over the space of precaptured sky light probes to estimate a sun position.

14. The method of claim 13 , wherein applying an inverse optimization light algorithm comprises:
initializing a hemispherical lighting model based on the estimated sun position; and
optimizing the hemispherical lighting model over the space of precaptured sky light probes.

15. The method of claim 11 , further comprising:
inserting a virtual object into the outdoor image; and
relighting the virtual object using the estimated sky light probe.

16. The method of claim 11 , wherein:
the captured outdoor image is a low dynamic range (LDR) image;
the precaptured sky light probes are high dynamic range (HDR) sky light probes; and
the estimated sky light probe is an HDR sky light probe.

17. The method of claim 11 , wherein the object of interest is a human face.

Assignees

• Disney Entpr Inc

Inventors

• Matthews Iain
• Lalonde Jean-Francois

Classifications

• H04N5/265 Primary

  Mixing · CPC title

• H04N23/741

  by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors · CPC title

• H04N23/71 Primary

  Circuitry for evaluating the brightness variation · CPC title

• H04N5/2355

  Electricity · mapped topic

• H04N5/23229

  Electricity · mapped topic