Reducing power consumption of extended reality devices

Patent metadata

Field	Value
Publication number	US-2025068228-A1
Application number	US-202318478853-A
Country	US
Kind code	A1
Filing date	Sep 29, 2023
Priority date	Aug 24, 2023
Publication date	Feb 27, 2025
Grant date	—

Abstract

Examples describe a method performed by an extended reality (XR) device that implements a multi-camera object tracking system. The XR device accesses object tracking data associated with an object in a real-world environment. Based on the object tracking data, the XR device activates a low-power mode of the multi-camera object tracking system. In the low-power mode, a state of the object in the real-world environment is determined by using the multi-camera object tracking system.

First claim

What is claimed is:

1 . A method performed by an extended reality (XR) device that implements a multi-camera object tracking system, the method comprising:
accessing object tracking data associated with an object in a real-world environment; activating, based on the object tracking data, a low-power mode of the multi-camera object tracking system; and in the low-power mode, determining a state of the object in the real-world environment using the multi-camera object tracking system.

2 . The method of claim 1 , wherein the object is a hand of a user of the XR device.

3 . The method of claim 2 , wherein the object tracking data comprises handedness data that identifies the hand of the user.

4 . The method of claim 1 , wherein activating the low-power mode comprises dynamically selecting a subset of cameras of the multi-camera object tracking system for determining the state of the object.

5 . The method of claim 1 , wherein determining the state of the object comprises activating a dynamic switching function that causes the XR device to switch between different subsets of cameras of the multi-camera object tracking system to determine the state of the object, such that, at a first point in time, a first subset of the cameras is selected to determine the state of the object, and at a second point in time, a second subset of the cameras is selected to determine the state of the object.

6 . The method of claim 5 , wherein the dynamic switching function causes the XR device to cycle through the different subsets of cameras of the multi-camera object tracking system in a sequence that is based on the object tracking data.

7 . The method of claim 1 , wherein the state of the object comprises at least a location associated with the object, and wherein the object tracking data comprises a predicted location of the object within a three-dimensional reference coordinate system of the XR device.

8 . The method of claim 7 , wherein the predicted location of the object is generated based on at least one of: historic tracking data; or a predicted pose of the XR device.

9 . The method of claim 7 , wherein activating the low-power mode comprises selecting, based on the predicted location of the object relative to a field of view of each respective camera of the multi-camera object tracking system, a subset of cameras of the multi-camera object tracking system for determining the state of the object in the real-world environment.

10 . The method of claim 7 , wherein determining the state of the object comprises:
projecting, for each camera of the multi-camera object tracking system and based on a predetermined calibration of the camera, the predicted location of the object onto a two-dimensional camera view image to obtain a two-dimensional projected location; selecting, based on the projected locations, a subset of cameras of the multi-camera object tracking system; and using only the subset of cameras to determine the state of the object in the real-world environment.

11 . The method of claim 10 , wherein each camera in the subset of cameras is selected based on the projected location relative to at least one reference location within the respective camera view image.

12 . The method of claim 11 , wherein, for each camera in the subset of cameras, the camera is selected based on the projected location meeting a predefined condition, wherein the predefined condition is one of: a distance between the projected location and the at least one reference location is less than a threshold value; a difference between the distance between the projected location and the at least one reference location for one or more other cameras of the multi-camera object tracking system and the distance between the projected location and the at least one reference location for the camera exceeds a threshold value; the projected location is within a predetermined zone in the respective camera view image relative to the at least one reference location; or the projected location is outside of a predetermined zone in the respective camera view image relative to the at least one reference location.

13 . The method of claim 12 , wherein the camera is selected based on the predefined condition being met with respect to multiple consecutive frames.

14 . The method of claim 11 , wherein projecting the predicted location of the object comprises projecting at least one point of interest associated with the object onto the camera view image and determining a bounding box associated with the at least one point of interest, the method further comprising, for each camera of the multi-camera object tracking system, determining a distance between the bounding box and the at least one reference location within the respective camera view image, each camera in the subset of cameras being selected based on the determined distance.

15 . The method of claim 1 , wherein activating the low-power mode comprises reducing a sampling rate of at least a subset of cameras of the multi-camera object tracking system.

16 . The method of claim 1 , wherein activating the low-power mode comprises activating a reduced image processing setting for at least a subset of cameras of the multi-camera object tracking system.

17 . The method of claim 1 , wherein activating the low-power mode comprises:
selecting, based on the object tracking data, a subset of cameras of the multi-camera object tracking system for determining the state of the object; and switching off each camera of the multi-camera object tracking system that is excluded from the selected subset of cameras.

18 . The method of claim 1 , wherein the object tracking data comprises at least one of: detection data indicating whether the object has been detected; a predicted location of the object within a three-dimensional reference coordinate system of the XR device; a predicted appearance zone of the object; predicted motion of the object; handedness data of a user of the XR device;
historic power consumption data for at least one camera of the multi-camera object tracking system; or expected inference performance for at least one camera of the multi-camera object tracking system with respect to the object.

19 . An extended reality (XR) device comprising:
a multi-camera object tracking system; a memory that stores instructions; and at least one processor configured by the instructions to perform operations comprising:
accessing object tracking data associated with an object in a real-world environment;
activating, based on the object tracking data, a low-power mode of the multi-camera object tracking system; and
in the low-power mode, determining a state of the object in the real-world environment using the multi-camera object tracking system.

20 . A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by at least one processor of an extended reality (XR) device that implements a multi-camera object tracking system, cause the at least one processor to perform operations comprising:
accessing object tracking data associated with an object in a real-world environment; activating, based on the object tracking data, a low-power mode of the multi-camera object tracking system; and in the low-power mode, determining a state of the object in the real-world environment using the multi-camera object tracking system.

Assignees

• Snap Inc

Inventors

• Chatzikalymnios Evangelos
• Faeulhammer Thomas
• Wolf Daniel
• Zhou Kai

Classifications

• G06V40/107

  Static hand or arm · CPC title

• G06F1/3287 Primary

  by switching off individual functional units in the computer system · CPC title