Mixed reality object rendering based on environment lighting

Patent metadata

Field	Value
Publication number	US-11164366-B2
Application number	US-201816479547-A
Country	US
Kind code	B2
Filing date	Jan 16, 2018
Priority date	Jan 23, 2017
Publication date	Nov 2, 2021
Grant date	Nov 2, 2021

Abstract

Implementations of the subject matter described herein relate to mixed reality rendering of objects. According to the embodiments of the subject matter described herein, while rendering an object, a wearable computing device takes lighting conditions in the real world into account, thereby increasing the reality of the rendered object. In particular, the wearable computing device acquires environment lighting information of an object to be rendered and renders the object to a user based on the environment lighting information. In this way, the object rendered by the wearable computing device can be more real and accurate. The user will thus have a better interaction experience.

First claim

The invention claimed is:

1. A wearable computing device comprising a head-mounted display, the wearable computing device comprising:
a processing unit;
a memory coupled to the processing unit and storing instructions which, when executed by the processing unit, perform acts including:
acquiring, in real-time, environment lighting information of an object to be rendered, the environment lighting information including information concerning light intensities in a plurality of directions in an environment where the object is located;
generating, in real-time, a lighting map associated with the object based on the environment lighting information;
down-sampling, in real-time, the lighting map to generate a set of down-sampled lighting maps having different resolutions; and
rendering, via the head-mounted display, the object to a user based on the set of down-sampled lighting maps, wherein the rendering is based on diffuse reflectance intensities of points on the object, the diffuse reflectance intensities being determined based on a lighting map having a lowest resolution of the set of down-sampled lighting maps.

2. The device according to claim 1 , wherein acquiring the environment lighting information of the object to be rendered comprises:
receiving an image of the environment where the object is located captured by a camera that operatively communicates with the wearable computing device; and
acquiring the environment lighting information based on the image.

3. The device according to claim 2 , wherein acquiring the environment lighting information of the object to be rendered further comprises:
in response to detecting that the object is different from a previously rendered object, determining an exposure compensation parameter associated with the object; and
providing the exposure compensation parameter to the camera, to enable the camera to adjust, based on the exposure compensation parameter, an exposure time used for capturing the image of the environment where the object is currently located.

4. The device according to claim 2 , wherein the image is a panorama image of the environment where the object is located.

5. The device according to claim 1 , wherein generating the lighting map associated with the object based on the environment lighting information comprises:
converting the environment lighting information to a cube map as the lighting map.

6. The device according to claim 1 , wherein down-sampling an initial lighting map comprises:
iteratively averaging pixels in the initial lighting map by a predefined resolution reduction factor to obtain respective down-sampled lighting maps in the set of down-sampled lighting maps.

7. The device according to claim 1 , wherein rendering the object based on the set of down-sampled lighting maps further comprises:
determining specular reflectance intensities of points on the object by using the set of down-sampled lighting maps; and
rendering the object further based on the specular reflectance intensities.

8. The device according to claim 7 , wherein the specular reflectance intensities of points on the object are determined based on a surface roughness coefficient of the respective points on the object.

9. A method implemented by a wearable computing device comprising a head-mounted display, the wearable computing device comprising:
acquiring, in real-time, environment lighting information of an object to be rendered, the environment lighting information including information concerning light intensities in a plurality of directions in an environment where the object is located;
generating, in real-time, a lighting map associated with the object based on the environment lighting information;
down-sampling, in real-time, the lighting map to generate a set of down-sampled lighting maps having different resolutions; and
rendering, via the head-mounted display, the object to a user based on the set of down-sampled lighting maps, wherein the rendering is based on diffuse reflectance intensities of points on the object, the diffuse reflectance intensities being determined based on a lighting map having a lowest resolution of the set of down-sampled lighting maps.

10. The method according to claim 9 , wherein acquiring the environment lighting information of the object to be rendered comprises:
receiving an image of the environment where the object is located captured by a camera that operatively communicates with the wearable computing device; and
acquiring the environment lighting information based on the image.

11. The method according to claim 10 , wherein acquiring the environment lighting information of the object to be rendered further comprises:
in response to detecting that the object is different from a previously rendered object, determining an exposure compensation parameter associated with the object; and
providing the exposure compensation parameter to the camera, to enable the camera to adjust, based on the exposure compensation parameter, an exposure time used for capturing the image of the environment where the object is currently located.

12. The method according to claim 9 , wherein down-sampling an initial lighting map comprises:
iteratively averaging pixels in the initial lighting map by a predefined resolution reduction factor to obtain respective down-sampled lighting maps in the set of down-sampled lighting maps.

13. The method according to claim 9 , wherein rendering the object based on the set of down-sampled lighting maps further comprises:
determining specular reflectance intensities of points on the object by using the set of down-sampled lighting maps; and
rendering the object further based on the specular reflectance intensities.

14. The method according to claim 13 , wherein the specular reflectance intensities of points on the object are determined based on a surface roughness coefficient of the respective points on the object.

15. A computer program product stored in a non-transient storage medium and including machine executable instructions which, when running on a wearable computing device comprising a head-mounted display, cause the wearable computing device to:
acquire, in real-time, environment lighting information of an object to be rendered, the environment lighting information including information concerning light intensities in a plurality of directions in an environment where the object is located;
generating, in real-time, a lighting map associated with the object based on the environment lighting information;
down-sampling, in real-time, the lighting map to generate a set of down-sampled lighting maps having different resolutions; and
render, via the head-mounted display, the object to a user based on the set of down-sampled lighting maps, wherein the rendering is based on diffuse reflectance intensities of points on the object, the diffuse reflectance intensities being determined based on a lighting map having a lowest resolution of the set of down-sampled lighting maps.

16. The computer program product of claim 15 , wherein the acquiring the environment lighting information of the object to be rendered comprises:
receiving a panoramic image of the environment where the object is located captured by a camera integrated on the wearable computing device; and
acquiring the environment lighting information based in part on the image.

17. The computer program product of claim 16 , wherein at least a portion of the environment lighting information is acquired directly from the environment where the object is located.

18. The

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Chen Guojun
• Dong Yue
• Tong Xin

Classifications

• H04N23/698

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• H04N23/73

  by influencing the exposure time · CPC title

• G06T15/04

  Texture mapping · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T2215/12

  Shadow map, environment map · CPC title