Optical illumination mapping

Patent metadata

Field	Value
Publication number	US-9418629-B2
Application number	US-201313841894-A
Country	US
Kind code	B2
Filing date	Mar 15, 2013
Priority date	Mar 15, 2013
Publication date	Aug 16, 2016
Grant date	Aug 16, 2016

Abstract

Techniques for augmenting an appearance of a first object. Embodiments include capturing a visual scene for display. Here, the visual scene includes a physical object and wherein the visual scene is captured using one or more camera devices. The physical object is identified as a first predetermined object type, based on one or more object identifiers associated with the physical object. Embodiments also retrieve visual characteristics information corresponding to the first predetermined object type. A sequence of frames that includes the first object is then rendered for display, where the appearance the first object in the rendered sequence of frames is augmented based on the retrieved visual characteristics information and an appearance of the physical object in the captured visual scene.

First claim

What is claimed is:

1. A method of augmenting an appearance of a first virtual object, comprising:
capturing a visual scene for display, wherein the visual scene includes a physical object and wherein the visual scene is captured using one or more camera devices;
identifying the physical object as a object type, by analyzing the visual scene to determine a plurality of edges of the physical object within the visual scene and further based on coloration of the physical object within the visual scene;
retrieving a predefined object profile corresponding to the first object type, wherein the predefined object profile specifies a measure of natural coloration of objects of the first object type;
determining a plurality of attributes of a light source within an environment in which the physical object is located, based on an appearance of the physical object in the captured visual scene and the retrieved predefined object profile corresponding to the first object type, wherein the plurality of attributes include at least (i) a position of the light source within the environment and (ii) a measure of coloration of the light source, and wherein the measure of coloration of the light source is determined by comparing the appearance of the physical object within the captured visual scene to the measure of natural coloration of objects of the first object type;
simulating lighting effects for the first virtual object based on the plurality of attributes of the light source; and
rendering, by operation of one or more computer processors, a sequence of frames for display that includes the first virtual object, wherein the appearance of the first virtual object in the rendered sequence of frames is augmented by applying the simulated lighting effects to the first virtual object, thereby simulating an application of the light source within the environment to the first virtual object.

2. The method of claim 1 , wherein the plurality of attributes of the light source include an angle of the light source, an indication of whether the light source is omnidirectional, a color of the light source, and an intensity of the light source, and
wherein the predefined object profile corresponding to the first object type specifies a reflectivity value of objects of the first object type and a shaded reflectivity value of objects of the first object type.

3. The method of claim 1 , further comprising at least one of:
projecting the rendered sequence of frames, using one or more projection devices; and
displaying the rendered sequence of frames on a display device of an augmented reality device.

4. The method of claim 1 , further comprising:
determining visual attributes of the physical object within the captured visual scene, based on the appearance of the physical object within the captured visual scene and the retrieved predefined object profile, wherein the visual attributes include (i) one or more shadows relating to the physical object, (ii) indirect global illumination information relating to the physical object, and (iii) one or more reflections relating to the physical object,
wherein rendering a sequence of frames for display that include the first virtual object is further based on the determined visual attributes of the physical object.

5. The method of claim 1 , wherein the measure of natural coloration of objects of the first object type describes a coloration of the objects under predefined lighting conditions.

6. The method of claim 1 , further comprising:
determining a measure of indirect global illumination for the environment as a whole, and separate from the plurality of attributes of the light source; and
wherein the appearance of the first virtual object in the rendered sequence of frames is further augmented based on the determined measure of indirect global illumination for the environment.

7. The method of claim 1 , wherein the first virtual object includes a reflective surface, and further comprising:
determining a position and orientation of the first virtual object within a three-dimensional scene representing the environment; and
determining at least one physical object within the environment positioned in a direction of the reflective surface of the first virtual object,
wherein an appearance of the reflective surface within the rendered sequence of frames is generated based on the predefined object profile corresponding to the first object type, an appearance of the determined at least one physical object within the environment within the captured visual scene, and the plurality of attributes of the light source within the environment.

8. A non-transitory computer-readable medium containing a program that, when executed by operation of one or more processors, performs an operation for augmenting an appearance of a first virtual object, the operation comprising:
capturing a visual scene for display, wherein the visual scene includes a physical object and wherein the visual scene is captured using one or more camera devices;
identifying the physical object as a first object type, by analyzing the visual scene to determine a plurality of edges of the physical object within the visual scene and further based on coloration of the physical object within the visual scene;
retrieving a predefined object profile corresponding to the first object type, wherein the predefined object profile specifies a measure of natural coloration of objects of the first object type;
determining a plurality of attributes of a light source within an environment in which the physical object is located, based on an appearance of the physical object in the captured visual scene and the retrieved predefined object profile corresponding to the first object type, wherein the plurality of attributes include at least (i) a position of the light source within the environment and (ii) a measure of coloration of the light source, and wherein the measure of coloration of the light source is determined by comparing the appearance of the physical object within the captured visual scene to the measure of natural coloration of objects of the first object type;
simulating lighting effects for the first virtual object based on the plurality of attributes of the light source; and
rendering, by operation of one or more computer processors, a sequence of frames for display that includes the first virtual object, wherein the appearance of the first virtual object in the rendered sequence of frames is augmented by applying the simulated lighting effects to the first virtual object, thereby simulating an application of the light source within the environment to the first virtual object.

9. The non-transitory computer-readable medium of claim 8 , wherein the plurality of attributes of the light source include an angle of the light source, an indication of whether the light source is omnidirectional, a color of the light source, and an intensity of the light source, and
wherein the predefined object profile corresponding to the first object type specifies a reflectivity value of objects of the first object type and a shaded reflectivity value of objects of the first object type.

10. The non-transitory computer-readable medium of claim 8 , the operation further comprising at least one of:
projecting the rendered sequence of frames, using one or more projection devices; and
displaying the rendered sequence of frames on a display device of an augmented reality device.

11. The non-transitory computer-readable medium of claim 8 , the operation further comprising:
determining visual attributes of the physical object within the captured visual scene, based on the appearance of the physical object within the captured visual scene

Assignees

• Disney Entpr Inc

Inventors

• Calian Dan Andrei
• Mitchell Kenneth John

Classifications

• G06T15/04

  Texture mapping · CPC title

• G09G5/377 Primary

  for mixing or overlaying two or more graphic patterns (G09G5/02, G09G5/397 take precedence) · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T2215/16

  Using real world measurements to influence rendering · CPC title