Systems for detecting gestures performed within activation-threshold distances of artificial-reality objects to cause operations at physical electronic devices, and methods of use thereof

Patent metadata

Field	Value
Publication number	US-12498792-B2
Application number	US-202318310505-A
Country	US
Kind code	B2
Filing date	May 1, 2023
Priority date	Jun 17, 2022
Publication date	Dec 16, 2025
Grant date	Dec 16, 2025

Abstract

A method includes, while presenting an artificial-reality environment that includes a virtual object and a representation of a physical object, detecting, based on data from a first group of sensors of a wrist-wearable device, that the wrist-wearable device is within a first distance of an electronic device responsive to user gestures and within another threshold distance of the virtual object. The electronic device is represented within the artificial-reality environment. The method includes, while the wrist-wearable device is within the first activation-threshold distance of the electronic device and is also within a second activation-threshold distance of the virtual object, determining, based on data from a second set of sensors of the wrist-wearable device, that a gesture corresponds to an operation at the electronic device. And the method includes, based on determining that the user gesture corresponds to the operation, causing the physical electronic device to perform the operation.

First claim

What is claimed is:

1 . A method, comprising:
while an artificial-reality environment that includes a virtual object for controlling a physical electronic device is being presented to a user by a head-wearable device, distinct from the physical electronic device:
in accordance with determining that the user is within a first activation-threshold distance for controlling the physical electronic device, causing an indication to be provided, via the head-wearable device, indicating that one or more gestures are available for controlling the physical electronic device;
detecting, based on data from a first group of sensors of a wrist-wearable device worn by the user, wherein the first group of sensors includes a depth sensor, and does not include any biopotential-signal sensors, that the wrist-wearable device is within the first activation-threshold distance of the physical electronic device that is responsive to user gestures and is also within a second activation-threshold distance of the virtual object for controlling the physical electronic device, wherein the physical electronic device is represented within the artificial-reality environment that includes the virtual object for controlling the physical electronic device;
while the wrist-wearable device is within the first activation-threshold distance of the physical electronic device and is also within the second activation-threshold distance of the virtual object for controlling the physical electronic device, determining, based at least in part on data from a second group of sensors of the wrist-wearable device, wherein the second group of sensors includes a biopotential-signal sensor, and does not include any depth sensors, that a user gesture corresponds to an operation at the physical electronic device; and
in response to determining that the user gesture corresponds to the operation at the physical electronic device, causing the physical electronic device to perform the operation and updating the virtual object based on performance of the operation.

2 . The method of claim 1 , further comprising:
determining, based on data received from the head-wearable device, that the user is oriented such that the user has a first gaze direction, wherein the first gaze direction is within a threshold angular range of a first direction between a location of the user and a location of the physical electronic device; and in accordance with detecting a user gesture while the first gaze direction of the user is within the threshold angular range of the first direction, causing the operation to be performed at the physical electronic device.

3 . The method of claim 2 , further comprising:
determining, based on data received from the head-wearable device, that the user has changed orientations such that the user has a second gaze direction, wherein the second gaze direction is not within the threshold angular range of the first direction to the physical electronic device relative to the location of the user; and forgoing providing an indication to the user that the user gesture is available at the physical electronic device in accordance with the user having the second gaze direction.

4 . The method of claim 3 , further comprising:
in accordance with determining that the second gaze direction is within the threshold angular range of a second direction to a second physical electronic device relative to the location of the user:
causing a second indication to be provided, via the head-wearable device, indicating that one or more gestures, detectable by the second group of sensors, are available for causing operations to be performed at the second physical electronic device.

5 . The method of claim 1 , further comprising:
detecting, based on data from the second group of sensors of the wrist-wearable device, a second user gesture, distinct from the user gesture, that corresponds to a second operation at the physical electronic device, distinct from the operation; and in response to detecting the second user gesture, causing the second operation to be performed at the physical electronic device.

6 . The method of claim 5 , further comprising:
detecting, based on data from the second group of sensors of the wrist-wearable device, a third user gesture, the third user gesture having a same hand movement as the user gesture, that corresponds to a third operation at the physical electronic device, distinct from the operation, wherein:
the user gesture is a surface gesture within a surface threshold distance of a physical surface, and
the third user gesture is an in-air gesture beyond the surface threshold distance of the physical surface.

7 . The method of claim 1 , wherein determining that the user is within the first activation-threshold distance of the physical electronic device further includes determining that the user is in a same room as the physical electronic device.

8 . The method of claim 1 , wherein the user gesture comprises movement from a first location to a second location.

9 . The method of claim 8 , further comprising:
during the movement from the first location to the second location:
providing an intermediate haptic event at the wrist-wearable device, the intermediate haptic event indicating that the movement between the first location and the second location corresponds to the user gesture.

10 . The method of claim 1 , wherein updating a display element of the virtual object comprises:
providing an artificial-reality animation of the virtual object toward the physical electronic device; and ceasing to display the virtual object.

11 . The method of claim 1 , wherein:
the physical electronic device is an adjustable light source; and the user gesture corresponds to an operation to adjust a brightness of the adjustable light source.

12 . The method of claim 11 , further comprising:
displaying the virtual object near the adjustable light source; and in response to the user gesture to adjust the brightness of the adjustable light source, updating the display of the virtual object near the adjustable light source.

13 . The method of claim 1 , wherein:
the physical electronic device is a speaker; and the operation performed at the speaker adjusts a volume control of the speaker.

14 . An electronic device, comprising:
one or more processors; and memory, comprising instructions, which, when executed by the one or more processors, cause the electronic device to perform operations for:
while an artificial-reality environment that includes a virtual object for controlling a physical electronic device is being presented to a user by a head-wearable device, distinct from the physical electronic device:
in accordance with determining that the user is within a first activation-threshold distance for controlling the physical electronic device, causing an indication to be provided, via the head-wearable device, indicating that one or more gestures are available for controlling the physical electronic device;
detecting, based on data from a first group of sensors of a wrist-wearable device worn by the user, wherein the first group of sensors includes a depth sensor, and does not include any biopotential-signal sensors, that the wrist-wearable device is within the first activation-threshold distance of the physical electronic device that is responsive to user gestures and is also within a second activation-threshold distance of the virtual object for controlling the physical electronic device, wherein the physical electronic device is represented within the artificial-reality environment that includes the virtual object for controlling

Assignees

• Meta Platforms Tech Llc

Inventors

• Whitmire Eric
• Sodhi Rajinder
• Levi Andre
• Lee Austin
• Lee Szeyin

Classifications

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06F3/0346

  with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors · CPC title

• G06F3/016

  Input arrangements with force or tactile feedback as computer generated output to the user · CPC title

• G06F3/165

  Management of the audio stream, e.g. setting of volume, audio stream path · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title