Visual-inertial tracking using rolling shutter cameras

Patent metadata

Field	Value
Publication number	US-12356095-B2
Application number	US-202418677296-A
Country	US
Kind code	B2
Filing date	May 29, 2024
Priority date	Jun 29, 2020
Publication date	Jul 8, 2025
Grant date	Jul 8, 2025

Abstract

Visual-inertial tracking of an eyewear device using a rolling shutter camera(s). The device includes a position determining system. Visual-inertial tracking is implemented by sensing motion of the device. An initial pose is obtained for a rolling shutter camera and an image of an environment is captured. The image includes feature points captured at a particular capture time. A number of poses for the rolling shutter camera is computed based on the initial pose and sensed movement of the device. The number of computed poses is responsive to the sensed movement of the mobile device. A computed pose is selected for each feature point in the image by matching the particular capture time for the feature point to the particular computed time for the computed pose. The position of the mobile device is determined within the environment using the feature points and the selected computed poses for the feature points.

First claim

What is claimed is:

1. A method for visual-inertial tracking with a mobile device having a motion detector and a camera system including a first rolling shutter camera, the method comprising:
obtaining an initial pose for the first rolling shutter camera;
capturing, during a capture period, an image of an environment using the camera system, the image including feature points, each feature point captured at a capture time during the capture period;
sensing, during the capture period, movement of the mobile device with the motion detector;
computing a number of poses for the first rolling shutter camera based on the initial pose and the sensed movement, each computed pose corresponding to a computed time within the capture period, wherein the number of computed poses is responsive to the sensed movement of the mobile device;
storing each of the computed number of poses in a lookup table;
selecting, for each feature point in the image, one of the number of computed poses by matching the capture time for the feature point to the computed time for the computed pose, wherein the selecting includes retrieving, for feature points in the image, the one of the number of computed poses from the lookup table with the corresponding capture time that is prior to the corresponding computed time; and
determining a position of the mobile device within the environment using the feature points and the selected computed poses for the feature points.

2. The method of claim 1 , further comprising:
wherein the selecting includes retrieving, for each feature point in the image, the one of the number of computed poses from the lookup table with the corresponding capture time that is immediately prior to the corresponding computed time.

3. The method of claim 1 , wherein the camera system is a multi-camera camera system further comprising at least a second rolling shutter camera.

4. The method of claim 3 , further comprising:
obtaining another initial pose for the second rolling shutter camera;
capturing, during another capture period, a second image of the environment using the second rolling shutter camera, the image including feature points, each feature point captured at a capture time during the other capture period;
sensing, at a plurality of times during the other capture period, movement of the mobile device with the motion detector;
computing a number of poses for the second camera based on the initial pose and the sensed movement, each second camera computed pose corresponding to a computed time within the other capture period, wherein the number of second camera computed poses is responsive to the sensed movement of the mobile device during the other capture period;
selecting, for feature points in the second image, one of the number of second camera computed poses by matching the capture time for the feature points to the computed time for the computed pose; and
determining the position of the mobile device within the environment using the feature points for the first image, the feature points for the second image, and the selected computed poses for the feature points.

5. The method of claim 4 , the method further comprising the steps of:
storing each of the second camera computed number of poses in the lookup table;
wherein the selecting includes retrieving, for feature points in the image, the one of the number of second camera computed poses from the lookup table with the corresponding capture time that is prior to the corresponding computed time.

6. The method of claim 4 , wherein the determining step comprises:
storing each of the second camera computed number of poses in the lookup table;
wherein the selecting includes retrieving, for each feature point in the image, the one of the number of second camera computed poses from the lookup table with the corresponding capture time that is immediately prior to the corresponding computed time.

7. The method of claim 1 , wherein the determining the position comprises applying a simultaneous localization and mapping (SLAM) algorithm using the feature points and the selected computed poses for the feature points.

8. The method of claim 1 , wherein the sensing movement comprises at least one of:
sensing motion with an inertial measurement unit (IMU); or
sensing a rate of change in position of the mobile device responsive to coordinates from a global positioning system (GPS).

9. The method of claim 1 , wherein the number of computed poses is less than the number of feature points.

10. A visual-inertial tracking system for determining a position of an eyewear device within an environment, the system comprising:
a motion detector;
a first rolling shutter camera;
an eyewear device comprising the motion detector, the first rolling shutter camera, a processor, and a memory; and
programming in the memory, wherein execution of the programming by the processor configures the eyewear device to obtain an initial pose for the first rolling shutter camera, capture, during a capture period, an image of an environment using the camera system, the image including feature points, each feature point captured at a capture time during the capture period, sense, during the capture period, movement of the eyewear device with the motion detector, compute a number of poses for the first rolling shutter camera based on the initial pose and the sensed movement, each computed pose corresponding to a computed time within the capture period, wherein the number of computed poses is responsive to the sensed movement of the eyewear device, store each of the number of computed poses in a lookup table stored in memory, select, for feature points in the image, one of the number of computed poses by matching the capture time for the feature points to the computed time for the computed pose, retrieve, from memory, for feature points in the image, the one of the number of computed poses from the lookup table with the corresponding capture time that is prior to the corresponding computed time, and determine a position of the eyewear device within the environment using the feature points and the selected computed poses for the feature points.

11. The system of claim 10 , further comprising:
retrieve, from memory, for each feature point in the image, the one of the number of computed poses from the lookup table with the corresponding capture time that is immediately prior to the corresponding computed time.

12. The system of claim 10 , wherein the number of computed poses is less than the number of feature points.

13. The system of claim 10 , wherein movement is sensed by at least one of:
an inertial measurement unit (IMU); or
a rate of change in position of the eyewear device responsive to coordinates from a global positioning system (GPS).

14. The system of claim 10 , wherein the position is determined by applying a simultaneous localization and mapping (SLAM) algorithm using the feature points and the selected computed poses for the feature points.

15. The system of claim 10 , wherein the system further comprises a multi-camera camera system comprising a second rolling shutter camera.

16. The system of claim 15 , wherein the programming in the memory, when executed by the processor further configures the eyewear device to obtain another initial pose for the second rolling shutter camera, capture, during another capture period, a second image of the environment using the second rolling shutter camera, the image including feature points, each feature point captured at a capture time during the other capture period, sense, at a plurality of times during the other capture

Assignees

• Snap Inc

Inventors

• Kalkgruber Matthias
• Mendez Erick Mendez
• Wagner Daniel
• Wolf Daniel
• Zhou Kai

Classifications

• H04N23/51

  Housings · CPC title

• G02B2027/0178

  Eyeglass type (eyeglass details G02C) · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title

• G02B2027/014

  comprising information/image processing systems · CPC title

• G02B27/0172

  characterised by optical features · CPC title