Artificial reality collaborative working environments

Patent metadata

Field	Value
Publication number	US-11770384-B2
Application number	US-202217675852-A
Country	US
Kind code	B2
Filing date	Feb 18, 2022
Priority date	Sep 15, 2020
Publication date	Sep 26, 2023
Grant date	Sep 26, 2023

Abstract

Aspects of the present disclosure are directed to creating and administering artificial reality collaborative working environments and providing interaction modes for them. An XR work system can provide and control such artificial reality collaborative working environments to enable, for example, A) links between real-world surfaces and XR surfaces; B) links between multiple real-world areas to XR areas with dedicated functionality; C) maintaining access, while inside the artificial reality working environment, to real-world work tools such as the user's computer screen and keyboard; D) various hand and controller modes for different interaction and collaboration modalities; E) use-based, multi-desk collaborative room configurations; and F) context-based auto population of users and content items into the artificial reality working environment.

First claim

We claim:

1. A method comprising:
tracking a real-world location of a real-world keyboard;
generating a 3D keyboard model at a location in an artificial reality environment relative to the real-world location;
determining that one or both of a user's hands are within a threshold distance of the real-world keyboard;
identifying contours of the one or both of the user's hands;
enabling a hand passthrough mode that presents, in the artificial reality environment, a display of a portion of the real-world that corresponds to the identified hand contours;
receiving a stream of a display of a real-world computer; and
displaying content from the display of the real-world computer, in the artificial reality environment, with a location based on one or more established anchor points in the artificial reality environment;
wherein user movements in relation to the 3D keyboard model correspond to input to the real-world keyboard that is routed to the real-world computer.

2. The method of claim 1 further comprising:
interfacing with the real-world keyboard; and
causing the real-world keyboard to display a pattern of lights;
wherein the tracking the real-world location of the real-world keyboard is based on recognizing the pattern of lights.

3. The method of claim 1 wherein the method is performed by an XR device and the method further comprises:
interfacing with the real-world computer, other than the XR device, with which the real-world keyboard is connected; and
receiving, from the real-world computer, an identification of a type of the real-world keyboard;
wherein the generating the 3D keyboard model is performed, in part, by selecting a 3D model corresponding to the type of the real-world keyboard.

4. The method of claim 1 further comprising:
identifying a height of a dedicated space on a surface of a real-world desk; and
setting the one or more anchor points, in relation to the dedicated space, where each anchor point is identified as a location in the artificial reality environment for placing virtual content.

5. The method of claim 1 ,
wherein the method is performed by an XR device; and
wherein the stream of the display of the real-world computer is received from an application, installed on the real-world computer, that has an established communication channel with the XR device.

6. The method of claim 1 , wherein the identifying contours of the one or both of the user's hands is performed by monitoring captured images that depict the one or both of the user's hands to create a virtual model of the one or both of the user's hands and using the model to determine one or more of hand outlines, hand shapes, hand positions, or any combination thereof.

7. The method of claim 1 ,
wherein the identifying the contours of the one or both of the user's hands comprises determining one or more outlines of the one or both of the user's hands; and
wherein the enabling the hand passthrough mode includes displaying of a portion of the real-world that is a set amount around one the or more determined outline of the one or both of the user's hands.

8. The method of claim 1 further comprising:
determining that the one or both of the user's hands are no longer within the threshold distance of the real-world keyboard; and
in response, disabling the hand passthrough mode, including stopping display of the portion of the real-world and displaying a virtual model of the one or both of the user's hands corresponding to one or more real-world locations of the one or both of the user's hands.

9. A computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process comprising:
tracking a real-world location of a real-world keyboard;
determining that one or both of a user's hands are within a threshold distance of the real-world keyboard;
identifying contours of the one or both of the user's hands;
enabling a hand passthrough mode that presents, in the artificial reality environment, a display of a portion of the real-world that corresponds to the identified hand contours;
receiving a stream of a display of a real-world computer; and
displaying content from the display of the real-world computer, in the artificial reality environment, with a location based on one or more anchor points in the artificial reality environment;
wherein user movements in relation to a 3D keyboard, in the artificial reality environment, correspond to input to the real-world keyboard that is routed to the real-world computer.

10. The computer-readable storage medium of claim 9 wherein the process is performed by an XR device and the process further comprises:
receiving, from the real-world computer, an identification of a type of the real-world keyboard; and
generating the 3D keyboard by selecting a 3D model corresponding to the type of the real-world keyboard.

11. The computer-readable storage medium of claim 9 , wherein the process further comprises:
identifying a height of a dedicated space on a surface of a real-world desk; and
setting the one or more anchor points, in relation to the dedicated space, where each of the one or more anchor points is identified as a location in the artificial reality environment for placing virtual content.

12. The computer-readable storage medium of claim 9 , wherein the process further comprises:
wherein the process is performed by an XR device; and
wherein the stream of the display of the real-world computer is received from an application, installed on the real-world computer, that has an established communication channel with the XR device.

13. The computer-readable storage medium of claim 9 , wherein the identifying contours of the one or both of the user's hands is performed by monitoring captured images that depict the one or both of the user's hands to create a virtual model of the one or both of the user's hands and using the model to determine one or more of hand outlines, hand shapes, hand positions, or any combination thereof.

14. The computer-readable storage medium of claim 9 ,
wherein the identifying the contours of the one or both of the user's hands comprises determining one or more outlines of the one or both of the user's hands; and
wherein the enabling the hand passthrough mode includes displaying of a portion of the real-world that is a set amount around one the or more determined outline of the one or both of the user's hands.

15. The computer-readable storage medium of claim 9 , wherein the process further comprises coordinating authorization to allow the stream of the display of the real-world computer by:
causing an authorization request to be provided in a foreground of the real-world computer such that keyboard strokes from the real-world keyboard are routed to the authorization request; and
providing a notification, in the artificial reality environment, instructing the user to press a key that will authorize the stream, of the display of the real-world computer, in response to the authorization request.

16. A computing system comprising:
one or more processors; and
one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising:
tracking a real-world location of a real-world keyboard;
determining that one or both of a user's hands are within a threshold distance of the real-world keyboard;
presenting, in the artificial reality environment, a portion of the real-world that corresponds to the user's hands and/or the real-world keyboard;
receiving a stream of

Assignees

• Meta Platforms Tech Llc

Inventors

• Lebeau Michael James
• Freire Santos Manuel Ricardo
• Anpilogovs Aleksejs
• Sorkine Hornung Alexander
• Wanbo Björn
• Treacy Connor
• Lee Fangwei
• Ruiz Federico
• Mallinson Jonathan
• Mayoh Jonathan Richard
• Tanner Marcus
• Inversin Panya
• Ray Sarthak
• Shen Sheng
• Steptoe William Arthur Hugh
• Marra Alessia
• Noris Gioacchino
• Readinger Derrick
• Lock Jeffrey Wai-King
• Witthuhn Jeffrey
• Spurlock Jennifer Lynn
• Laich Larissa Heike
• Santos Javier Alejandro Sierra

Classifications

• H04L63/107 Primary

  wherein the security policies are location-dependent, e.g. entities privileges depend on current location or allowing specific operations only from locally connected terminals · CPC title

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06F3/0219

  Special purpose keyboards · CPC title

• G06F3/1423

  controlling a plurality of local displays, e.g. CRT and flat panel display · CPC title

• G06T19/003

  Navigation within 3D models or images · CPC title