Intermediary device for offloading processing operations for an artificial-reality system and for enabling interactions and input paradigms, and systems and methods of use thereof

Patent metadata

Field	Value
Publication number	US-12468556-B2
Application number	US-202418803030-A
Country	US
Kind code	B2
Filing date	Aug 13, 2024
Priority date	Aug 14, 2023
Publication date	Nov 11, 2025
Grant date	Nov 11, 2025

Abstract

A non-transitory computer-readable storage medium including instructions for distributing tasks between a handheld intermediary processing device (HIPD) and communicatively coupled devices is disclosed. The instructions cause one or more processors to, while a HIPD and a head-wearable device are communicatively coupled and share operational data for performing one or more computational tasks, identify one or more back-end tasks and one or more front-end tasks associated with performing the one or more computational tasks. The instructions further cause performance of the one or more back-end tasks at the HIPD including updating the operational data to create updated operational data. The instructions further cause performance of the one or more front-end tasks at the head-wearable device using the updated operational data such that a representation based on the one or more computational tasks is presented to the user by the head-wearable device.

First claim

What is claimed is:

1 . A non-transitory computer-readable storage medium including instructions that, when executed by one or more processors, cause the one or more processors to:
while a handheld intermediary processing device and a head-wearable device are communicatively coupled and share operational data for performing one or more computational tasks:
identify one or more back-end tasks and one or more front-end tasks associated with performing the one or more computational tasks;
cause performance of the one or more back-end tasks at the handheld intermediary processing device including updating the operational data to create updated operational data; and
cause performance of the one or more front-end tasks at the head-wearable device using the updated operational data such that a representation based on the one or more computational tasks is presented to a user by the head-wearable device.

2 . The non-transitory computer-readable storage medium of claim 1 , wherein the instructions, that when executed, further cause the one or more processors to:
while the handheld intermediary processing device and a wrist-wearable device are communicatively coupled and share additional operational data for performing one or more additional computational tasks:
identify one or more additional back-end tasks and one or more additional front-end tasks associated with performing the one or more additional computational tasks;
cause performance of the one or more additional back-end tasks at the handheld intermediary processing device including updating the additional operational data to create updated additional operational data; and
cause performance of the one or more additional front-end tasks at one or more of the wrist-wearable device and the head-wearable device based on the updated additional operational data such that a representation of the one or more additional computational tasks is presented to the user.

3 . The non-transitory computer-readable storage medium of claim 2 , wherein:
the additional operational data includes neuromuscular signal data captured by the wrist-wearable device; the one or more additional computational tasks include hand gesture recognition; and the one or more additional back-end tasks include:
determining an in-air hand gesture performed by the user based on the neuromuscular signal data;
in response to a determination that the in-air hand gesture is associated with an input command, updating the additional operational data based on the input command, such that the updated additional operational data includes a representation of the input command associated with the in-air hand gesture; and
causing performance of the one or more additional front-end tasks at one or more of the wrist-wearable device and the head-wearable device based on the updated additional operational data such that a representation of the one or more additional computational tasks is presented to the user.

4 . The non-transitory computer-readable storage medium of claim 3 , wherein:
the operational data includes image data captured by the head-wearable device; the one or more computational tasks include hand gesture recognition; and the one or more back-end tasks include:
determining an in-air hand gesture performed by the user based on the image data; and
in response to a determination that the in-air hand gesture is associated with an input command, updating the operational data based on the input command, such that the updated operational data includes a representation of the input command associated with the in-air hand gesture.

5 . The non-transitory computer-readable storage medium of claim 4 , wherein:
the one or more back-end tasks include:
determining an in-air hand gesture performed by the user based on the image data and the neuromuscular signal data, and
in response to a determination that the in-air hand gesture is associated with an input command, updating at least one of the operational data and the additional operational data based on the input command, such that at least one of the updated operational data and the updated additional operational data includes a representation of the input command associated with the in-air hand gesture.

6 . The non-transitory computer-readable storage medium of claim 4 , wherein the instructions, that when executed, further cause the one or more processors to:
in accordance with a determination that a user's hand is within a field-of view of the head-wearable device:
determine, by the one or more back-end tasks, the in-air hand gesture performed by the user based on the image data, and
in response to the determination that the in-air hand gesture is associated with the input command, update, by the one or more back-end tasks, the operational data based on the input command, such that the updated operational data includes the representation of the input command associated with the in-air hand gesture; and
in accordance with a determination that the user's hand is not within the field-of view of the head-wearable device:
determine, by the one or more back-end tasks, the in-air hand gesture performed by the user based on the neuromuscular signal data, and
in response to the determination that the in-air hand gesture is associated with the input command, update, by the one or more back-end tasks, the additional operational data based on the input command, such that the updated additional operational data includes the representation of the input command associated with the in-air hand gesture.

7 . The non-transitory computer-readable storage medium of claim 1 , wherein the instructions, that when executed, further cause the one or more processors to:
before the handheld intermediary processing device and the head-wearable device are communicatively coupled, detect, by the handheld intermediary processing device, the head-wearable device within a proximity to the handheld intermediary processing device and associated with the handheld intermediary processing device; and in accordance with a determination that the head-wearable device is within the proximity to the handheld intermediary processing device and associated with the handheld intermediary processing device, communicatively couple the head-wearable device and the handheld intermediary processing device.

8 . The non-transitory computer-readable storage medium of claim 7 , wherein the proximity is twenty feet.

9 . The non-transitory computer-readable storage medium of claim 7 , wherein the detection, by the handheld intermediary processing device, of the head-wearable device is within the proximity to the handheld intermediary processing device and associated with the handheld intermediary processing device is based on:
receiving, from the handheld intermediary device, a signal to communicatively couple with the head-wearable device within the proximity of the handheld intermediary processing device.

10 . The non-transitory computer-readable storage medium of claim 7 , wherein the detection, by the handheld intermediary processing device, of the head-wearable device is within the proximity to the handheld intermediary processing device and associated with the handheld intermediary processing device is based on:
receiving, from the head-wearable device, a signal to communicatively couple with the handheld intermediary processing device within the proximity of the head-wearable device.

11 . The non-transitory computer-readable storage medium of claim 1 , wherein the identification of the one or more back-end tasks and the one or more front-end tasks associated with performing the one or more computational tasks comprises one or more of

Assignees

• Meta Platforms Tech Llc

Inventors

• Henry Emron Jackson
• Bravate Joseph N
• Martinez Roger Ibars
• Ma Eric
• Pla I Conesa Pol
• Reardon Thomas Robert
• Rojas Chris
• Roth Ian Joseph
• Sparks Bryan
• Tank Vikram

Classifications

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06V40/28

  Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113) · CPC title

• G06F1/163

  Wearable computers, e.g. on a belt · CPC title

• G06F3/0383

  Signal control means within the pointing device · CPC title