Predicting a light probe for an outdoor image

Patent metadata

Field	Value
Publication number	US-9639773-B2
Application number	US-201314091270-A
Country	US
Kind code	B2
Filing date	Nov 26, 2013
Priority date	Nov 26, 2013
Publication date	May 2, 2017
Grant date	May 2, 2017

Abstract

Methods and systems for predicting light probes for outdoor images are disclosed. A light probe database is created to learn a mapping from the outdoor image's features to predicted outdoor light probe illumination parameters. The database includes a plurality of images, image features for each of the plurality of images, and a captured light probe for each of the plurality of images. A light probe illumination model based on a sun model and sky model is fitted to the captured light probes. The light probe for the outdoor image may be predicted based on the database dataset and fitted light probe models.

First claim

What is claimed is:

1. A method, comprising:
predicting a light probe for an outdoor image based on a learned mapping from the outdoor image's features to light probe illumination parameters;
wherein the light probe illumination parameters comprise sun parameters and sky parameters;
wherein the learned mapping is based on a light probe database and a double exponential sun model or von-Mises Fisher sun model for the sun parameters and a sky model for the sky parameters, and wherein the light probe database comprises:
a plurality of images comprising a plurality of objects at a plurality of different locations captured under a plurality of illumination conditions;
image features for each of the plurality of images; and
a plurality of captured light probes associated with the plurality of images.

2. The method of claim 1 , wherein the outdoor image's features comprise features computed from a surface normal.

3. The method of claim 2 , wherein image features for each of the plurality of images in the light probe database comprise a normal map.

4. The method of claim 1 , wherein the sun parameters comprise sun position, sun angular variation, and sun color.

5. The method of claim 1 , wherein the sky model is based on one of a Preetham model, a cosine model, or a von-Mises Fisher model.

6. The method of claim 1 , wherein the sun model and the sky model are fitted to the light probe database captured light probes to create fitted models.

7. The method of claim 1 , further comprising inserting a virtual object into the outdoor image and lighting the virtual object using the predicted light probe.

8. The method of claim 4 , wherein the sky parameters comprise sky color and sky angular variation.

9. A method, comprising:
predicting a light probe for an outdoor image based on a learned mapping from the outdoor image's features to light probe illumination parameters;
wherein the light probe illumination parameters comprise sun parameters and sky parameters, wherein the sun parameters comprise sun position, sun angular variation, and sun color, wherein the sky parameters comprise sky color and sky angular variation;
wherein the learned mapping is based on a sun model for the sun parameters and a sky model for the sky parameters;
wherein the learned mapping is based on a light probe database comprising:
a plurality of images comprising a plurality of locations captured under a plurality of illumination conditions;
image features for each of the plurality of images; and
a plurality of captured light probes associated with the plurality of images; and wherein predicting a light probe for an outdoor image comprises:

predicting the sun position based on a probabilistic model; and
predicting the sun angular variation, sun colors, sky colors, and sky angular variation based on a regression technique.

10. The method of claim 9 , wherein the sun position is predicted relative to the camera that took the outdoor image.

11. The method of claim 9 , wherein the regression technique is a kernel regression or a linear regression.

12. A system, comprising:
a camera configured to take an outdoor image; and
a computer configured to predict a light probe for the outdoor image based on a learned mapping from the outdoor image's features to light probe illumination parameters:
wherein the light probe illumination parameters comprise sun parameters and sky parameters;
wherein the learned mapping is based on a light probe database and a double exponential sun model or von-Mises Fisher sun model for the sun parameters and a sky model from the sky parameters, and wherein the light probe database comprises:
a plurality of images comprising a plurality of objects at a plurality of different locations captured under a plurality of illumination conditions;
image features for each of the plurality of images; and
a plurality of captured light probes associated with the plurality of images.

13. The system of claim 12 , wherein the outdoor image's features comprises features computed from a surface normal.

14. The system of claim 13 , wherein image features for each of the plurality of images in the light probe database comprise a normal map.

15. The system of claim 12 , wherein the sun parameters comprise sun position; sun angular variation, and sun color.

16. The system of claim 12 , wherein the sky model is based on one of a Preetham model, a cosine model, or a von-Mises Fisher model.

17. The system of claim 12 , wherein the sun model and the sky model are fitted to the light probe database captured light probes to create fitted models.

18. The system of claim 12 , wherein the computer is further configured to: insert a virtual object into the outdoor image; and
light the virtual object using the predicted light probe.

19. The system of claim 15 , wherein the sky parameters comprise sky color and sky angular variation.

20. The system of claim 19 , wherein predicting a light probe for an outdoor image comprises:
predicting the sun position based on a probabilistic model; and
predicting the sun angular variation, sun colors, sky colors, and sky angular variation based on a regression technique.

21. The system of claim 20 , wherein the sun position is predicted relative to the camera that took the outdoor image.

22. The system of claim 20 , wherein the regression technique is a kernel regression or a linear regression.

23. A method of creating a light probe database, comprising:
capturing, with a camera, a plurality of images of a plurality of objects at plurality of different locations under a plurality of illumination conditions;
capturing, with a camera, a light probe for each of the plurality of images;
recovering image features for each of the plurality of captured images; wherein recovering image features for each of the plurality of captured images comprises recovering a normal map for each of the plurality of captured images; and
calibrating and aligning the captured images and captured light probes.

24. The method of claim 23 , wherein the light probe database is used to predict a light probe for an outdoor image.

Assignees

• Disney Entpr Inc

Inventors

• Lalonde Jean-Francois
• Matthews Iain

Classifications

• G06T15/506 Primary

  Illumination models · CPC title

• G06V10/60

  relating to illumination properties, e.g. using a reflectance or lighting model · CPC title

• G06K9/00697

  Physics · mapped topic

• G06K9/4661 Primary

  Physics · mapped topic

• G06V20/38

  Outdoor scenes · CPC title