Environment synthesis for lighting an object

Patent metadata

Field	Value
Publication number	US-11694392-B2
Application number	US-201916372882-A
Country	US
Kind code	B2
Filing date	Apr 2, 2019
Priority date	May 22, 2018
Publication date	Jul 4, 2023
Grant date	Jul 4, 2023

Abstract

Various implementations disclosed herein include devices, systems, and methods that render a reflective surface of a computer-generated reality (“CGR”) object based on synthesis in a CGR environment. In order to render a reflective surface of the CGR object, one exemplary implementation involves synthesizing an environment map of a CGR environment representing a portion of a physical scene based on observed characteristics of the physical scene. In an implementation, generation of a complete environment map includes identifying pixels of the environment map with no corresponding texture and generating synthesized texture based on textural information associated with one or more camera images of the physical scene. In an implementation, a CGR object is rendered in the CGR environment, wherein an appearance of a reflective surface of the CGR object is determined based on the complete environment map of the CGR environment.

First claim

What is claimed is:

1. A method of rendering a reflective surface of a computer-generated reality (“CGR”) object based on surface synthesis in a CGR environment, comprising:
at a device with one or more processors and a computer readable medium:
determining a first geometry and textural information of a CGR environment, wherein the CGR environment is based on one or more camera images of a physical scene;
determining that the CGR environment includes a surface having a captured surface portion and a missing portion, the surface in the CGR environment representing a surface in the physical scene and the missing portion of the surface in the CGR environment corresponding to an uncaptured portion of the surface in the physical scene;
generating an extension portion for the missing portion based on an assessment of whether the captured surface portion has been cutoff by a boundary defined by a captured geometry, wherein the extension portion is generated by extending a 3D geometry of the captured surface portion based on a characteristic of the captured surface portion;
determining a second geometry of the CGR environment representing the physical scene based on the extension portion;
determining second textural information, wherein the second textural information is synthesized for the extension portion by:
determining a characteristic of the captured surface portion that matches a stored reference characteristic of an object; or
identifying a repeating pattern or texture of the captured surface portion;

generating an environment map using the determined second geometry and the second textural information; and
rendering a CGR object in the CGR environment, wherein an appearance of a reflective surface of the CGR object is determined based on the environment map.

2. The method of claim 1 , further comprising:
determining a three-dimensional (“3D”) position of the CGR object in the CGR environment; and
determining a reflectance of the CGR object, wherein the appearance of the reflective surface of the CGR object is based at least in part on the 3D position and the reflectance of the CGR object.

3. The method of claim 2 , wherein a size of the extension portion is determined based at least in part on the 3D position, a geometry, and the reflectance of the CGR object.

4. The method of claim 1 , further comprising:
blending the extension portion and at least one adjacent portion.

5. The method of claim 1 , further comprising:
selecting a center of projection of the environment map based on a three-dimensional (“3D”) position of the CGR object, wherein generating the environment map is based at least in part on the center of projection of the environment map.

6. The method of claim 1 , wherein the second geometry is determined based at least in part on a plurality of extension portions.

7. The method of claim 1 , further comprising:
generating a plurality of environment maps; and
rendering multiple CGR objects in the CGR environment, wherein appearances of reflective surfaces of the multiple CGR objects are determined based on the plurality of environment maps.

8. The method of claim 1 , wherein the textural information comprises information about color, pattern, reflectance, or irradiance.

9. The method of claim 1 , further comprising:
identifying a darkest texture from the one or more camera images of the physical scene, wherein the textural information is based at least on part on the darkest texture.

10. The method of claim 1 , further comprising:
determining that the environment map includes pixels with no corresponding texture;
generating synthesized texture for the pixels with no corresponding texture; and
enhancing the environment map by replacing the pixels with no corresponding texture with the synthesized texture.

11. The method of claim 1 , wherein the CGR object in the CGR environment is rendered on a screen or on a semi-transparent display apparatus.

12. The method of claim 1 , wherein the extension portion is generated by extending the 3D geometry of the captured surface portion to extend a 3D model of the physical scene to include the extension portion.

13. The method of claim 1 , wherein the extension portion is sized based on the characteristic of the captured surface portion.

14. A system for rendering a reflective surface of a computer-generated reality (“CGR”) object based on environment map synthesis in a CGR environment, comprising:
a device with one or more processors; and
a computer readable medium including instructions that, when executed by the one or more processors, cause the system to:
determine a first geometry and textural information of a CGR environment, wherein the CGR environment is based on one or more camera images of a physical scene;
determine that the CGR environment includes a surface having a captured surface portion a missing portion, the surface in the CGR environment representing a surface in the physical scene and the missing portion of the surface in the CGR environment corresponding to an uncaptured portion of the surface in the physical scene;
generate an extension portion for the missing portion based on an assessment of whether the captured surface portion has been cutoff by a boundary defined by a captured geometry, wherein the extension portion is generated by extending a 3D geometry of the captured surface portion based on a characteristic of the captured surface portion;
determine a second geometry of the CGR environment representing the physical scene based on the extension portion;
determine second textural information, wherein the second textural information is synthesized for the extension portion based on identifying one or more patches of the captured surface portion;
generate an environment map using the determined second geometry and the second textural information; and
render a CGR object in the CGR environment, wherein an appearance of a reflective surface of the CGR object is determined based on the environment map.

Assignees

• Apple Inc

Inventors

• Kurz Daniel
• Somanath Gowri
• Holl Tobias

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T7/73

  using feature-based methods · CPC title

• G06T7/40

  Analysis of texture (depth or shape recovery from texture G06T7/529) · CPC title

• G06T15/50 Primary

  Lighting effects · CPC title

• G06T15/04 Primary

  Texture mapping · CPC title