Inserting objects into content

Patent metadata

Field	Value
Publication number	US-9330500-B2
Application number	US-201113314723-A
Country	US
Kind code	B2
Filing date	Dec 8, 2011
Priority date	Dec 8, 2011
Publication date	May 3, 2016
Grant date	May 3, 2016

Abstract

An image into which one or more objects are to be inserted is obtained. Based on the image, both a 3-dimensional (3D) representation and a light model of the scene in the image are generated. One or more objects are added to the 3D representation of the scene. The 3D representation of the scene is rendered, based on the light model, to generate a modified image that is the obtained image modified to include the one or more objects.

First claim

What is claimed is:

1. A method implemented in one or more computing devices, the method comprising:
obtaining a single image of a scene;
generating, based on the single image of the scene, a 3D model of the scene;
generating, based on the single image of the scene and in the absence of other images of the scene, a light model including 3D light source locations, directions, and intensities for the scene;
correcting the light model by automatically refining the locations of light sources projected onto the 3D model of the scene, the automatically refining including choosing light parameters by at least one of the one or more computing devices minimizing an objective function that is based on a rendered image of the scene and the single image of the scene in the absence of other images of the scene, the objective function being further based on applying a per-pixel weighting that places less emphasis on pixels near the ground;
inserting one or more objects into the 3D model of the scene; and
modifying the appearance of the single image, based on both the corrected light model and the 3D model of the scene, producing a modified image of the scene that depicts the scene as if the one or more objects were actually part of the scene.

2. A method as recited in claim 1 , the generating the light model comprising generating the light model based on the single image in the absence of additional data regarding lighting collected from the physical scene depicted in the single image.

3. A method as recited in claim 1 , the generating the 3D model comprising automatically identifying scene boundaries and receiving a user input adjusting the scene boundaries.

4. A method as recited in claim 3 , further comprising:
receiving a user input identifying extruding geometry on which an object can be inserted;
converting the extruded geometry to a 3D model; and
adding the 3D model of the extruded geometry to the 3D model of the scene.

5. A method as recited in claim 3 , further comprising receiving a user input identifying a surface that would occlude an object inserted behind the surface.

6. A method as recited in claim 1 , the single image of the scene comprising a frame of video of the scene, and the modifying including modifying the video of the scene.

7. A method as recited in claim 1 , the generating the light model comprising:
identifying, based on the single image, locations of one or more interior lighting sources in the single image; and
maintaining, as at least part of the light model, the locations of one or more interior lighting sources in the single image.

8. A method as recited in claim 7 , the identifying locations of one or more interior lighting sources in the single image comprising receiving user input identifying locations of interior lighting sources, and the automatically refining comprising automatically refining the user identified locations.

9. A method as recited in claim 1 , the generating the light model comprising:
identifying, based on the image, locations of one or more shafts of light in the single image;
identifying an amount of shadowing of each of multiple pixels in the one or more shafts of light;
determining a direction of each of the one or more shafts of light; and
maintaining, as at least part of the light model, the locations of one or more shafts of light in the single image, the amount of shadowing of each of multiple pixels in the one or more shafts of light, and the direction of each of the one or more shafts of light.

10. A method as recited in claim 9 , the identifying locations of one or more shafts of light in the single image comprising receiving a user input identifying the locations of the one or more shafts of light.

11. A method as recited in claim 9 , the determining comprising receiving a user input identifying one or more sources of the one or more shafts of light, and determining the direction based on locations in polygons encompassing the one or more sources and the one or more shafts of light.

12. A method as recited in claim 9 , the determining the direction of each of the one or more shafts of light comprising determining, for one of the one or more shafts of light, the direction of the one shaft of light based on a user input of an estimate of the direction of the one shaft of light.

13. A method as recited in claim 1 , further comprising using an intrinsic decomposition technique to estimate an albedo and direct light from the single image by solving an additional objective function.

14. A method as recited in claim 13 , wherein the albedo is an albedo ρ, the direct light is a direct light D, γ1 is a first weight, γ2 is a second weight, γ3 is a third weight, m is a scalar mask, D 0 is an initial direct lighting estimate, and the additional objective function is:

argmin

ρ
,
D

⁢

∑

i
∈
pixels

⁢



Δ
⁢

⁢
ρ



i

+

γ
1

⁢

m
i

⁡

(

∇
ρ

)

i
2

+

γ
2

(

D
i

-

D

0
i

⁢

)

2

+

γ
3

⁡

(

∇
D

)

i
2

.

15. A method as recited in claim 1 , the generating the light model further comprising:
receiving user input identifying locations of on

Assignees

• Karsch Kevin
• Hedau Varsha Chandrashekhar
• Forsyth David A
• Hoiem Derek
• Univ Illinois

Inventors

• Karsch Kevin
• Hedau Varsha Chandrashekhar
• Forsyth David A
• Hoiem Derek

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title