Dynamic sensor selection for visual inertial odometry systems

Patent metadata

Field	Value
Publication number	US-12265222-B2
Application number	US-202318368288-A
Country	US
Kind code	B2
Filing date	Sep 14, 2023
Priority date	Jun 29, 2020
Publication date	Apr 1, 2025
Grant date	Apr 1, 2025

Abstract

Visual-inertial tracking of an eyewear device using sensors. The eyewear device monitors the sensors of a visual inertial odometry system (VIOS) that provide input for determining a position of the device within its environment. The eyewear device determines the status of the VIOS based information from the sensors and adjusts the plurality of sensors (e.g., by turning on/off sensors, changing the sampling rate, of a combination thereof) based on the determined status. The eyewear device then determines the position of the eyewear device within the environment using the adjusted plurality of sensors.

First claim

What is claimed is:

1. A method for visual-inertial tracking with an eyewear device, the method comprising:
monitoring a plurality of sensors of a visual inertial odometry system (VIOS), wherein each of the plurality of sensors provide input for determining a position of the eyewear device within an environment;
determining a status of the VIOS;
selecting a sampling rate for one of the plurality of sensors based on the determined status;
sampling the one of the plurality of sensors at the selected sampling rate; and
determining the position of the eyewear device within the environment using the one of the plurality of sensors at the selected sampling rate.

2. The method of claim 1 , further comprising adjusting the plurality of sensors based on the determined status.

3. The method of claim 2 , wherein the adjusting further comprises: selecting a subset of the plurality of sensors and powering off the remaining sensors, wherein the adjusting comprises:
selecting a subset of the plurality of sensors; and
powering off the remaining sensors;

wherein the determining the position of the eyewear device comprises determining the position of the eyewear device within the environment using the subset of sensors.

4. The method of claim 2 , wherein the adjusting includes:
adjusting at least one sensor of the plurality of sensors in response to new information, wherein adjusting includes altering rate, resolution, quality, power on, or power off.

5. The method of claim 2 , wherein the determining the status of the VIOS includes:
determining at least one of a motion parameter value or an uncertainty parameter value of the eyewear device; and
mapping the determined at least one of the motion parameter value or the uncertainty parameter value to one of a plurality of VIOS status configuration options.

6. The method of claim 5 , wherein the VIOS status configuration options include at least a low power level and a high power level and wherein the adjusting includes:
placing the VIOS in the low power level when the motion parameter value and the uncertainty parameter value are low; and
placing the VIOS in the high power level when the motion parameter values and the uncertainty parameter value are high.

7. The method of claim 1 , wherein the plurality of sensors include an inertial measurement unit (IMU) and a first camera.

8. The method of claim 7 , wherein the first camera is a first visible light camera, and wherein the plurality of sensors further includes one or more of a second visible light camera, a first depth camera, a second depth camera, another IMU, a radar system, or a GPS.

9. The method of claim 7 , wherein the determining the status of the VIOS includes:
capturing images with the first camera;
identifying a physical environment of the eyewear device;
comparing the identified physical environment to a prior physical environment;
identifying new information in the identified physical environment; and
determining the status of the VIOS based on the new information in the identified physical environment.

10. An eyewear device with visual-inertial tracking, the eyewear device comprising:
a visual inertial odometry system (VIOS) including a plurality of sensors, wherein the plurality of sensors include an inertial measurement unit (IMU) and a first camera, wherein each of the plurality of sensors provide input for determining a position of the eyewear device within an environment;
a processor configured to determine a status of the VIOS, select a sampling rate for one of the plurality of sensors based on the determined status, sample the one of the plurality of sensors at the selected sampling rate, adjust the plurality of sensors based on the determined status, and determine the position of the eyewear device within the environment using the adjusted plurality of sensors; and
a frame supporting the VIOS and the processor, the frame configured to be worn on the head of a user,
wherein the VIOS is configured to capture images with the first camera, and the processor is configured to identify a physical environment of the eyewear device and determine the status of the VIOS based on the identified physical environment.

11. The device of claim 10 , wherein the plurality of sensors include an inertial measurement unit (IMU) and a first camera.

12. The device of claim 11 , wherein the VIOS is configured to capture images with the first camera, and the processor is configured to identify a physical environment of the eyewear device and determine the status of the VIOS based on the identified physical environment.

13. The device of claim 10 , wherein the first camera is a first visible light camera, and wherein the plurality of sensors further includes one or more of a second visible light camera, a first depth camera, a second depth camera, another IMU, a radar system, or a GPS.

14. The device of claim 10 , wherein the processor is configured to:
capture images with the first camera;
identify a physical environment of the eyewear device;
compare the identified physical environment to a prior physical environment;
identify new information in the identified physical environment; and
determine the status of the VIOS based on the new information in the identified physical environment.

15. The device of claim 10 , wherein the determining the status of the VIOS includes:
determining at least one of a motion parameter value or an uncertainty parameter value of the eyewear device; and
mapping the determined at least one of the motion parameter value or the uncertainty parameter value to one of a plurality of VIOS status configuration options.

16. The device of claim 15 , wherein the VIOS status configuration options include at least a low power level and a high power level and wherein the adjusting includes:
placing the VIOS in the low power level when the motion parameter value and the uncertainty parameter value are low; and
placing the VIOS in the high power level when the motion parameter values and the uncertainty parameter value are high.

17. A non-transitory computer-readable medium storing program code for visual-inertial tracking when executed by an eyewear device having a plurality of sensors, a processor, and a memory, the program code, when executed, is operative to cause an electronic processor to perform the steps of:
monitoring a plurality of sensors of a visual inertial odometry system (VIOS), wherein each of the plurality of sensors provide input for determining a position of the eyewear device within an environment;
determining a status of the visual-inertial tracking system;
selecting a sampling rate for one of the plurality of sensors based on the determined status; and
sampling the one of the plurality of sensors at the selected sampling rate,
wherein the determining the position of the eyewear device comprises determining the position of the eyewear device within the environment using the one of the plurality of sensors sampled at the selected sampling rate.

18. The non-transitory computer-readable medium of claim 17 , further comprising adjusting the plurality of sensors based on the determined status.

Assignees

• Snap Inc

Inventors

• Borys Olha
• Halmetschlager-Funek Georg
• Kalkgruber Matthias
• Wolf Daniel
• Zillner Jakob

Classifications

• G06F1/3206

  Monitoring of events, devices or parameters that trigger a change in power modality · CPC title

• G06F1/163

  Wearable computers, e.g. on a belt · CPC title

• G02B2027/0178

  Eyeglass type (eyeglass details G02C) · CPC title

• G06T7/97

  Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85) · CPC title

• G06T7/50

  Depth or shape recovery · CPC title