Mapping a real-world room for a shared artificial reality environment

Patent metadata

Field	Value
Publication number	US-12100111-B2
Application number	US-202217936521-A
Country	US
Kind code	B2
Filing date	Sep 29, 2022
Priority date	Sep 29, 2022
Publication date	Sep 24, 2024
Grant date	Sep 24, 2024

Abstract

A room manager can generate mappings for a real-world room that support a shared XR environment. For example, the real-world room can include real-world objects and surfaces, such as a table(s), chair(s), wall(s), door(s), window(s), etc. The room manager can generate XR object definitions based on information received about the real-world room, object(s), and surface(s). For example, the room manager can implement a flow that guides a user equipped with an XR system to provide information for the XR object definitions, such as real-world surfaces that map to the XR object(s), borders (e.g., measured using a component of the XR system), such as borders on real-world surfaces, semantic information (e.g., number of seat assignments at an XR table, size of XR objects, etc.), and other suitable information. Implementations generate previews of the shared XR environment, such as a local preview and a remote preview.

First claim

We claim:

1. A method for providing a shared XR environment comprising a local version for one or more local users and a remote version for one or more remote users, the method comprising:
rendering, during the shared XR environment by a first XR system of a first local user, the local version, wherein, in the local version,
the remote users are represented using one or more avatars and the local users are represented using a passthrough video capture of the local users in the real-world room,
audio for local users is selectively muted at the first XR system and audio for remote users is output by the first XR system, and
at least part of the real-world room is displayed in passthrough by the first XR system using a video capture of the real-world room, the passthrough video capture being augmented with an XR board displayed at a location that is mapped to predefined boundaries on a vertical surface within the real-world room; and

dynamically rendering, by the first XR system, input from the first local user at the XR board when the first local user physically approaches the predefined boundaries on the vertical surface within the real-world room that are mapped to the XR board and performs detected body movements that correspond to the rendered input, wherein,
the remote version is rendered, during the shared XR environment, by a second XR system of a first remote user, and in the remote version, the remote users and local users are represented using avatars,
audio for local users and remote users is output by the second XR system, and
a virtual room that corresponds to the real-world room is displayed by the second XR system, the display of the virtual room including the XR board that dynamically renders the input that corresponds to the detected body movements of the first local user.

2. The method of claim 1 , wherein, in the remote version, the second XR system displays the XR board at a relative location in the virtual room that corresponds to a relative location of the XR board in the passthrough video capture of the real-world room.

3. The method of claim 1 , wherein the local version and the remote version include an XR table, the XR table being displayed using a passthrough video capture of a real-world table in the local version and the XR table being displayed as a virtual object in the remote version.

4. The method of claim 3 , wherein the local version and the remote version display an XR screen relative to the XR table, and the XR screen displays a video capture of a video user of the shared XR environment.

5. The method of claim 3 , wherein, in the local version and the remote version, the one or more local users and one or more remote users comprise seat assignments at the XR table.

6. The method of claim 1 , wherein an other local version of the shared XR environment is rendered by a third XR system to a second local user, the first local user is represented by the passthrough video capture of the first local user when seated at the XR table in the other local version displayed by the third XR system, and the first local user is represented by an avatar representation of the first local user when seated at the XR table in the remote version displayed by the second XR system.

7. The method of claim 1 , wherein the first remote user is located in an other physical space that is remote from the real-world room, and additional predefined boundaries on an other vertical surface at the other physical space are mapped to the XR board.

8. The method of claim 7 , wherein input from the first remote user is dynamically rendered by the XR board when the first remote user physically approaches the additional predefined boundaries, on the other vertical surface within the other physical space, that are mapped to the XR board and performs detected body movements that correspond to the rendered input.

9. The method of claim 1 , wherein the audio output by the first XR system and the second XR system comprises spatial audio based on relative positioning for the first local user and first remote user in the shared XR environment.

10. A non-transitory computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process for providing a shared XR environment comprising a local version for one or more local users and a remote version for one or more remote users, the process comprising:
rendering, during the shared XR environment by a first XR system of a first local user, the local version, wherein, in the local version,
the remote users are represented using one or more avatars and the local users are represented using a passthrough video capture of the local users in the real-world room,
audio for local users is selectively muted at the first XR system and audio for remote users is output by the first XR system, and
at least part of the real-world room is displayed in passthrough by the first XR system using a video capture of the real-world room, the passthrough video capture being augmented with an XR board displayed at a location that is mapped to predefined boundaries on a vertical surface within the real-world room; and

dynamically rendering, by the first XR system, input from the first local user at the XR board when the first local user physically approaches the predefined boundaries on the vertical surface within the real-world room that are mapped to the XR board and performs detected body movements that correspond to the rendered input, wherein,
the remote version is rendered, during the shared XR environment, by a second XR system of a first remote user, and in the remote version, the remote users and local users are represented using avatars,
audio for local users and remote users is output by the second XR system, and
a virtual room that corresponds to the real-world room is displayed by the second XR system, the display of the virtual room including the XR board that dynamically renders the input that corresponds to the detected body movements of the first local user.

11. The non-transitory computer-readable storage medium of claim 10 , wherein, in the remote version, the second XR system displays the XR board at a relative location in the virtual room that corresponds to a relative location of the XR board in the passthrough video capture of the real-world room.

12. The non-transitory computer-readable storage medium of claim 10 , wherein the local version and the remote version include an XR table, the XR table being displayed using a passthrough video capture of a real-world table in the local version and the XR table being displayed as a virtual object in the remote version.

13. The non-transitory computer-readable storage medium of claim 12 , wherein the local version and the remote version display an XR screen relative to the XR table, and the XR screen displays a video capture of a video user of the shared XR environment.

14. The non-transitory computer-readable storage medium of claim 12 , wherein, in the local version and the remote version, the one or more local users and one or more remote users comprise seat assignments at the XR table.

15. The non-transitory computer-readable storage medium of claim 10 , wherein an other local version of the shared XR environment is rendered by a third XR system to a second local user, the first local user is represented by the passthrough video capture of the first local user when seated at the XR table in the other local version displayed by the third XR system, and the first local user is represented by an avatar representation of the first local user when seated at the

Assignees

• Meta Platforms Tech Llc

Inventors

• Wanbo Björn
• Lebeau Michael James
• Steptoe William Arthur Hugh
• Mallinson Jonathan
• Wilson Steven James
• Rajendran Vasanth Kumar
• Baran Vasyl

Classifications

• G06F3/16

  Sound input; Sound output (speech processing G10L) · CPC title

• G02B2027/0178

  Eyeglass type (eyeglass details G02C) · CPC title

• G02B27/0172 Primary

  characterised by optical features · CPC title

• G06T2219/024

  Multi-user, collaborative environment · CPC title

• G06T7/13

  Edge detection · CPC title