Discrimination between virtual objects and real objects in a mixed reality scene

Patent metadata

Field	Value
Publication number	US-11776166-B2
Application number	US-202117497783-A
Country	US
Kind code	B2
Filing date	Oct 8, 2021
Priority date	Oct 8, 2021
Publication date	Oct 3, 2023
Grant date	Oct 3, 2023

Abstract

A method for providing a mixed reality (MR) experience is provided, including: rendering a mixed reality view for a user on an MR headset, wherein the mixed reality view includes virtual elements; during the rendering of the mixed reality view, capturing sensor data by the MR headset, and analyzing the sensor data to detect real-world objects in a local environment in which the MR headset is disposed; determining a risk to the user posed by one or more of the identified real-world objects; responsive to determining the risk, then adjusting an appearance of at least one virtual element.

First claim

The invention claimed is:

1. A method for providing a mixed reality (MR) experience, comprising:
rendering a mixed reality view for a user on an MR headset, wherein the mixed reality view includes virtual elements;
tracking a gaze of the user during the rendering of the mixed reality view;
during the rendering of the mixed reality view, capturing sensor data by the MR headset, and analyzing the sensor data to detect real-world objects in a local environment in which the MR headset is disposed;
determining a risk to the user posed by one or more of the identified real-world objects;
responsive to determining the risk, then adjusting an appearance of at least one virtual element; and,
following the adjusting of the appearance, responsive to detecting the gaze of the user looking at the one or more of the identified real-world objects posing the risk, including detecting the gaze directed toward the one or more of the identified real-world objects for at least a predefined duration, then stopping the adjusting of the appearance of the at least one virtual element.

2. The method of claim 1 , wherein determining the risk to the user is based on a proximity of the identified real-world objects to the user.

3. The method of claim 1 , wherein determining the risk to the user includes determining that a virtual element obstructs the user's view of the one or more of the identified real-world objects.

4. The method of claim 1 , wherein determining the risk to the user includes calculating a risk factor associated to the one or more of the identified real-world objects, and determining the risk factor to exceed a predefined threshold.

5. The method of claim 4 , wherein an amount of the adjusting of the appearance of at least one virtual element is based on the risk factor.

6. The method of claim 1 , wherein determining the risk to the user includes determining a trajectory of the one or more of the identified real-world objects.

7. The method of claim 1 , wherein determining the risk to the user includes determining a trajectory of the user.

8. The method of claim 1 , wherein adjusting the appearance of at least one virtual element includes adjusting a transparency of at least a portion of the at least one virtual element.

9. The method of claim 1 , wherein adjusting the appearance of at least one virtual element includes rendering a tag on the at least one virtual element.

10. The method of claim 1 , further comprising:
further responsive to determining the risk, then activating a haptic feedback for the user.

11. The method of claim 1 , wherein adjusting the appearance of the at least one virtual element includes changing a color of an outline or surface of the at least one virtual element.

12. A non-transitory computer readable medium having program instructions embodied thereon that, when executed by at least one computing device, cause said at least one computing device to perform a method for providing a mixed reality (MR) experience, said method comprising:
rendering a mixed reality view for a user on an MR headset, wherein the mixed reality view includes virtual elements;
tracking a gaze of the user during the rendering of the mixed reality view;
during the rendering of the mixed reality view, capturing sensor data by the MR headset, and analyzing the sensor data to detect real-world objects in a local environment in which the MR headset is disposed;
determining a risk to the user posed by one or more of the identified real-world objects;
responsive to determining the risk, then adjusting an appearance of at least one virtual element; and,
following the adjusting of the appearance, responsive to detecting the gaze of the user looking at the one or more of the identified real-world objects posing the risk, including detecting the gaze directed toward the one or more of the identified real-world objects for at least a predefined duration, then stopping the adjusting of the appearance of the at least one virtual element.

13. The non-transitory computer readable medium of claim 12 , wherein determining the risk to the user is based on a proximity of the identified real-world objects to the user.

14. The non-transitory computer readable medium of claim 12 , wherein determining the risk to the user includes determining that a virtual element obstructs the user's view of the one or more of the identified real-world objects.

15. The non-transitory computer readable medium of claim 12 , wherein determining the risk to the user includes calculating a risk factor associated to the one or more of the identified real-world objects, and determining the risk factor to exceed a predefined threshold.

16. The non-transitory computer readable medium of claim 15 , wherein an amount of the adjusting of the appearance of at least one virtual element is based on the risk factor.

17. The non-transitory computer readable medium of claim 12 , wherein determining the risk to the user includes determining a trajectory of the one or more of the identified real-world objects.

18. The non-transitory computer readable medium of claim 12 , wherein determining the risk to the user includes determining a trajectory of the user.

19. The non-transitory computer readable medium of claim 12 , wherein adjusting the appearance of at least one virtual element includes adjusting a transparency of at least a portion of the at least one virtual element.

20. The non-transitory computer readable medium of claim 12 , wherein adjusting the appearance of at least one virtual element includes rendering a tag on the at least one virtual element.

Assignees

• Sony Interactive Entertainment LLC

Inventors

• Yang Yiwei
• Benedetto Warren

Classifications

• G06T11/00 Primary

  Two-dimensional [2D] image generation · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06F3/016

  Input arrangements with force or tactile feedback as computer generated output to the user · CPC title

• G06T7/20

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title

• G06V20/20

  in augmented reality scenes · CPC title