Ambient light based mixed reality object rendering

Patent metadata

Field	Value
Publication number	US-11727654-B2
Application number	US-202217942776-A
Country	US
Kind code	B2
Filing date	Sep 12, 2022
Priority date	Aug 25, 2017
Publication date	Aug 15, 2023
Grant date	Aug 15, 2023

Abstract

Implementations of the subject matter described herein relate to mixed reality object rendering based on ambient light conditions. According to the embodiments of the subject matter described herein, while rendering an object a wearable computing device acquires light conditions of the real world, thereby increasing the reality of the rendered object. In particular, the wearable computing deice is configured to acquire an image of an environment where the wearable computing deice is located. The image is adjusted based on a cement parameter used when the image is captured. Subsequently, ambient light information is determined based on the adjusted image. In this way, the wearable computing deice can obtain more real and accurate emblem light information, so as to render to the user an object with enhanced reality. Accordingly, the user can have a better interaction experience.

First claim

The invention claimed is:

1. A wearable computing device, comprising:
a processing unit; and
a memory coupled to the processing unit and storing instructions which, when executed by the processing unit, perform operations comprising:
receiving image data of an image of an environment in which the wearable computing device is situated;
adjusting pixel color values and pixel opacity of the image based on camera parameters used by a camera in capturing the image resulting in an adjusted image, the camera parameters including one or more of an exposure time, light sensitivity, and a gamma correction parameter;
determining, based on the adjusted image, ambient light information that indicates light intensities in the environment; and
adjusting specular or reflection of an object in a subsequent image of the environment based on the ambient light information.

2. The device according to claim 1 , wherein the camera is onboard the wearable computing device, wherein the image data is received from the camera and wherein capturing the image of the environment comprises:
determining, based on a parameter indicating a field of view range of the camera, a plurality of shooting directions required for covering the environment; and
causing the camera to capture a plurality of images according to the plurality of shooting directions.

3. The device according to claim 1 , wherein the determining ambient light information of the wearable computing device comprises:
generating a panorama image of the environment based on the adjusted image; and
mapping the panorama image to a cube map indicating the ambient light information.

4. The device according to claim 1 , wherein the operations further comprise:
determining whether there is original ambient light information for e environment where the wearable computing device is located; and
in response to determining that there is the original ambient light information, updating the original ambient light information by using the determined ambient light information.

5. The device according to claim 4 , wherein the updating the original ambient light information by using the determined ambient light information comprises:
dividing the original ambient light information into a first plurality of portions;
dividing the determined ambient light information into a second plurality of portions; and
in response to a difference between one of the first plurality of portions and a corresponding one of the second plurality of portions exceeding a threshold difference, modifying the original ambient light information with the determined ambient light information.

6. The device according to claim 1 , wherein the operations further comprise:
rendering an object to a user of the wearable computing device based on the ambient light information.

7. The device according to claim 6 , wherein the rendering an object to a user of the wearable computing device based on the ambient light information comprises:
generating an initial light map associated with the object based on the ambient light information;
down-sampling the initial light map to generate a set of down-sampled light maps having different resolutions; and
rendering the object based on the set of down-sampled light maps.

8. A method implemented by a wearable computing device, comprising:
receiving image data of an image of an environment in which the wearable computing device is situated;
adjusting pixel color values and pixel opacity of the image based on camera parameters used by a camera in capturing the image resulting in an adjusted image, the camera parameters including one or more of an exposure time, light sensitivity, and a gamma correction parameter;
determining, based on the adjusted image, ambient light information that indicates light intensities; and
adjusting specular or reflection of an object in a subsequent image of the environment based on the ambient light information.

9. The method according to claim 8 , wherein the camera is onboard the wearable computing device and wherein the capturing the image of the environment where the wearable computing device is located comprises:
determining, based on a parameter indicating a field of view range of the camera, a plurality of shooting directions required for coveting the environment; and
causing the camera to capture a plurality of images according to the plurality of shooting directions.

10. The method according to claim 8 , wherein the determining ambient light information of the wearable computing device comprises:
generating a panorama image of the environment based on the adjusted image; and
mapping the panorama image to a cube map indicating the ambient light information.

11. The method according to claim 8 , further comprising:
determining whether there is original ambient light information for the environment where the wearable computing device is located; and
in response to determining that there is the original ambient light information, updating the original ambient light information by using the determined ambient light information.

12. The method according to claim 11 , wherein the updating the original ambient light information by using the determined ambient light information comprises:
dividing the original ambient light information into a first plurality of portions;
dividing the determined ambient light information into a second plurality of portions; and
in response to a difference between one of the first plurality of portions and a corresponding one of the second plurality of portions exceeding a threshold difference, modifying the original ambient light information with the determined ambient light information.

13. The method according to claim 8 , further comprising:
rendering an object to a user of the wearable computing device based on the ambient light information.

14. The method according to claim 13 , wherein the rendering an object to a user of the wearable computing device based on the ambient light information comprises:
generating an initial light map associated with the object based on the ambient light information;
down-sampling the initial light map to generate a set of down-sampled light maps having different resolutions; and
rendering the object based on the set of down-sampled light maps.

15. A non-transitory computer storage medium storing a computer program product and comprising machine executable instructions which, when miming on a wearable computing device, cause the wearable computing device to perform operations comprising:
receiving image data of an image of an environment in which the wearable computing device is situated;
adjusting pixel color values and pixel opacity of the image based on a camera parameter used by a camera in capturing the image resulting in an adjusted image; the camera parameter including one or more of an exposure time, light sensitivity, and a gamma correction parameter;
determining, based on the adjusted image, ambient light information that indicates light intensities in the environment, and
adjusting specular or reflection of an object in a subsequent image of the environment based on the ambient light information.

16. The non-transitory computer storage medium according to claim 15 , wherein adjusting pixel color values of the image based on camera parameters includes raising the pixel color values to the gamma correction parameter and dividing by a result of multiplying the light sensitivity by the exposure time.

17. The non-transitory computer storage medium according to claim 15 , wherein the operations fu

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Chen Guojun
• Dong Yue
• Tong Xin
• Ju Yingnan
• Yu Pingchao

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F1/163

  Wearable computers, e.g. on a belt · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06T15/506 Primary

  Illumination models · CPC title

• G06F3/0304 Primary

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title