Ambient light based mixed reality object rendering

Patent metadata

Field	Value
Publication number	US-11461978-B2
Application number	US-201816635485-A
Country	US
Kind code	B2
Filing date	Jun 21, 2018
Priority date	Aug 25, 2017
Publication date	Oct 4, 2022
Grant date	Oct 4, 2022

Abstract

Implementations of the subject matter described herein relate to mixed reality object rendering based on ambient light conditions. According to the embodiments of the subject matter described herein, while rendering an object, a wearable computing device acquires light conditions of the real world, thereby increasing the reality of the rendered object. In particular, the wearable computing device is configured to acquire an image of an environment where the wearable computing device is located. The image is adjusted based on a camera parameter used when the image is captured. Subsequently, ambient light information is determined based on the adjusted image. In this way, the wearable computing device can obtain more real and accurate ambient light information, so as to render to the user an object with enhanced reality. Accordingly, the user can have a better interaction experience.

First claim

The invention claimed is:

1. A wearable computing device, comprising:
a processing unit;
a memory coupled to the processing unit and storing instructions which, when executed by the processing unit, perform acts comprising:
capturing an image of an environment where the wearable computing device is located, the image captured by a camera that operatively communicates with the wearable computing device;
adjusting pixel color values and pixel opacity of the captured image based on camera parameters used by the camera in capturing the image resulting in an adjusted captured image that provides a baseline from which to identify ambient light directions, the camera parameters including one or more of an exposure time, light sensitivity, and a gamma correction parameter;
determining, based on the adjusted captured image, ambient light information that indicates light intensities in a plurality of directions in the environment; and
adjusting specular and reflection of an object in a subsequently captured image based on the ambient light information.

2. The device according to claim 1 , wherein the camera is onboard the wearable device and wherein capturing the image of an environment where the wearable computing device is located comprises:
determining, based on a parameter indicating a Field of View range of the camera, a plurality of shooting directions required for covering the environment; and
causing the camera to capture a plurality of images according to the plurality of shooting directions.

3. The device according to claim 1 , wherein the determining ambient light information of the wearable computing device comprises:
generating a panorama image of the environment based on the at least one adjusted image; and
mapping the panorama image to a cube map indicating the ambient light information.

4. The device according to claim 1 , wherein the acts further comprise:
determining whether there is original ambient light information for the environment where the wearable computing device is located; and
in response to determining that there is the original ambient light information, updating the original ambient light information by using the determined ambient light information.

5. The device according to claim 4 , wherein the updating the original ambient light information by using the determined ambient light information comprises:
dividing the original ambient light information into a first plurality of portions;
dividing the determined ambient light information into a second plurality of portions; and
in response to a difference between one of the first plurality of portions and a corresponding one of the second plurality of portions exceeding a threshold difference, modifying the original ambient light information with the determined ambient light information.

6. The device according to claim 1 , wherein the acts further comprise:
rendering an object to a user of the wearable computing device based on the ambient light information.

7. The device according to claim 6 , wherein the rendering an object to a user of the wearable computing device based on the ambient light information comprises:
generating an initial light map associated with the object based on the ambient light information;
down-sampling the initial light map to generate a set of down-sampled light maps having different resolutions; and
rendering the object based on the set of down-sampled light maps.

8. The device according to claim 1 , wherein adjusting pixel color values of the captured image based on camera parameters includes raising the pixel color values to the gamma correction parameter and dividing by a result of multiplying the light sensitivity by the exposure time.

9. A method implemented by a wearable computing device, comprising:
capturing an image of an environment where the wearable computing device is located, the image being captured by a camera that operatively communicates with the wearable computing device;
adjusting pixel color values and pixel opacity of the captured image based on camera parameters used by the camera in capturing the image resulting in an adjusted captured image that provides a baseline from which to identify ambient light directions, the camera parameters including one or more of an exposure time, light sensitivity, and a gamma correction parameter;
determining, based on the adjusted, captured image, ambient light information that indicates light intensities in a plurality of directions in the environment; and
adjusting specular and reflection of an object in a subsequently captured image based on the ambient light information.

10. The method according to claim 9 , wherein the camera is onboard the wearable device and wherein the capturing the image of an environment where the wearable computing device is located comprises:
determining, based on a parameter indicating a Field of View range of the camera, a plurality of shooting directions required for covering the environment; and
causing the camera to capture a plurality of images according to the plurality of shooting directions.

11. The method according to claim 9 , wherein the determining ambient light information of the wearable computing device comprises:
generating a panorama image of the environment based on the at least one adjusted image; and
mapping the panorama image to a cube map indicating the ambient light information.

12. The method according to claim 9 , further comprising:
determining whether there is original ambient light information for the environment where the wearable computing device is located; and
in response to determining that there is the original ambient light information, updating the original ambient light information by using the determined ambient light information.

13. The method according to claim 12 , wherein the updating the original ambient light information by using the determined ambient light information comprises:
dividing the original ambient light information into a first plurality of portions;
dividing the determined ambient light information into a second plurality of portions; and
in response to a difference between one of the first plurality of portions and a corresponding one of the second plurality of portions exceeding a threshold difference, modifying the original ambient light information with the determined ambient light information.

14. A non-transitory computer storage medium storing a computer program product and comprising machine executable instructions which, when running on a wearable computing device, cause the wearable computing device to:
capture an image of an environment where the wearable computing device is located, the image being captured by a camera that operatively communicates with the wearable computing device;
adjust pixel color values and pixel opacity of the captured image based on a camera parameter used by the camera in capturing the image resulting in an adjusted, captured image that provides a baseline from which to identify ambient light directions, the camera parameters including one or more of an exposure time, light sensitivity, and a gamma correction parameter;
determine, based on the at least one adjusted, captured image, ambient light information that indicates light intensities in a plurality of directions in the environment, and
adjusting specular and reflection of an object in a subsequently captured image based on the ambient light information.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Chen Guojun
• Dong Yue
• Tong Xin
• Ju Yingnan
• Yu Pingchao

Classifications

• G06T15/506 Primary

  Illumination models · CPC title

• G06T2215/16

  Using real world measurements to influence rendering · CPC title

• G06F1/163

  Wearable computers, e.g. on a belt · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06F3/0304 Primary

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title