Reconstructing mixed reality contextually derived actions

Patent metadata

Field	Value
Publication number	US-11467939-B2
Application number	US-202016906802-A
Country	US
Kind code	B2
Filing date	Jun 19, 2020
Priority date	Jun 19, 2020
Publication date	Oct 11, 2022
Grant date	Oct 11, 2022

Abstract

A computer implemented method includes recording data representative of a user experience sensed by a mixed reality device during a mixed reality experience of a user in a space, processing the recorded data to infer one or more actions performed by the user in the space during the mixed reality experience, and creating an augmented replay of the mixed reality experience based on the recorded data and inferred actions performed by the user during the mixed reality experience.

First claim

The invention claimed is:

1. A computer implemented method comprising:
recording data representative of a user experience sensed by a mixed reality device during a mixed reality experience of a user in a space;
processing the recorded data to infer one or more actions performed by the user in the space during the mixed reality experience; and
creating an augmented replay of the mixed reality experience based on the recorded data and inferred actions performed by the user during the mixed reality experience.

2. The method of claim 1 wherein the augmented replay includes an avatar representing the user, wherein the avatar is performing the inferred one or more actions.

3. The method of claim 1 wherein the recorded data includes user position data at multiple times during the mixed reality experience within the space and wherein processing the recorded data comprises:
determining a speed of the user from the position over time data; and
inferring a walking action in response to the determined speed.

4. The method of claim 1 wherein the recorded data includes user position data at multiple times during the mixed reality experience within the space and wherein processing the recorded data comprises:
determining a speed of the user from the position over time data; and
inferring a running action in response to the determined speed.

5. The method of claim 1 wherein the recorded data includes sounds occurring during the mixed reality experience and wherein processing the recorded data comprises:
identifying a sound associated with an action; and
inferring the action based on the identified sound.

6. The method of claim 1 wherein the recorded data includes a role of the user, and wherein processing the recorded data includes inferring an intent of the user during one of the one or more inferred actions as a function of the role.

7. The method of claim 6 and further comprising:
receiving feedback regarding the inferred intent;
modifying the recorded data based on the received feedback; and
modifying the augmented replay based on the modified recorded data.

8. The method of claim 1 and further comprising inferring a secondary action based on one or more inferred actions, and wherein the augmented replay is generated as a function of the inferred secondary action.

9. The method of claim 1 and further comprising:
querying the processed recorded data for inferred actions; and
providing results of the query.

10. A machine-readable storage device having instructions for execution by a processor of a machine to cause the processor to perform operations to perform a method, the operations comprising:
recording data representative of a user experience sensed by a mixed reality device during a mixed reality experience of a user in a space;
processing the recorded data to infer one or more actions performed by the user in the space during the mixed reality experience; and
creating an augmented replay of the mixed reality experience based on the recorded data and inferred actions performed by the user during the mixed reality experience.

11. The device of claim 10 wherein the augmented replay includes an avatar representing the user, wherein the avatar is performing the inferred one or more actions.

12. The device of claim 10 wherein the recorded data includes user position data at multiple times during the mixed reality experience within the space and wherein processing the recorded data includes operations comprising:
determining a speed of the user from the position over time data; and
inferring a walking action in response to the determined speed or inferring a running action in response to the determined speed.

13. The device of claim 10 wherein the recorded data includes sounds occurring during the mixed reality experience and wherein processing the recorded data includes operations comprising:
identifying a sound associated with an action; and
inferring the action based on the identified sound.

14. The device of claim 10 wherein the recorded data includes a role of the user, and wherein processing the recorded data includes operations to infer an intent of the user during one of the one or more inferred actions as a function of the role.

15. The device of claim 14 and further including operations comprising:
receiving feedback regarding the inferred intent;
modifying the recorded data based on the received feedback; and
modifying the augmented replay based on the modified recorded data.

16. The device of claim 10 and further including operations comprising inferring a secondary action based on one or more inferred actions, and wherein the augmented replay is generated as a function of the inferred secondary action.

17. The device of claim 10 and further includes operations comprising:
querying the processed recorded data for inferred actions; and
providing results of the query.

18. A device comprising:
a processor; and
a memory device coupled to the processor and having a program stored thereon for execution by the processor to perform operations comprising:
recording data representative of a user experience sensed by a mixed reality device during a mixed reality experience of a user in a space;
processing the recorded data to infer one or more actions performed by the user in the space during the mixed reality experience; and
creating an augmented replay of the mixed reality experience based on the recorded data and inferred actions performed by the user during the mixed reality experience.

19. The device of claim 18 wherein the augmented replay includes an avatar representing the user, wherein the avatar is performing the inferred one or more actions, and further including operations comprising:
inferring a secondary action based on one or more inferred actions, wherein the augmented replay is generated as a function of the inferred secondary action;
receiving feedback regarding the inferred intent;
modifying the recorded data based on the received feedback; and
modifying the augmented replay based on the modified recorded data.

20. The device of claim 18 wherein the recorded data includes user position data at multiple times during the mixed reality experience within the space and wherein processing the recorded data includes operations comprising:
determining a speed of the user from the position over time data; and
inferring a walking action in response to the determined speed or inferring a running action in response to the determined speed.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Hill Erik A
• Paulovich Jonathan Gustav
• Schultz Michael William
• Pierpont Dylan Edmund
• Maricich Gabriel Robert
• Harrison Charles L
• Omotani Todd A
• Heasley Kathleen Carol

Classifications

• G06F11/323 Primary

  Visualisation of programs or trace data · CPC title

• A63F13/428

  involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes · CPC title

• A63F13/5255

  according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character · CPC title

• G06F11/302

  where the computing system component is a software system · CPC title

• A63F13/211 Primary

  using inertial sensors, e.g. accelerometers or gyroscopes · CPC title