Visual-inertial tracking using rolling shutter cameras

Patent metadata

Field	Value
Publication number	US-12028626-B2
Application number	US-202318098939-A
Country	US
Kind code	B2
Filing date	Jan 19, 2023
Priority date	Jun 29, 2020
Publication date	Jul 2, 2024
Grant date	Jul 2, 2024

Abstract

Visual-inertial tracking of an eyewear device using a rolling shutter camera(s). The eyewear device includes a position determining system. Visual-inertial tracking is implemented by sensing motion of the eyewear device. An initial pose is obtained for a rolling shutter camera and an image of an environment is captured. The image includes feature points captured at a particular capture time. A number of poses for the rolling shutter camera is computed based on the initial pose and sensed movement of the device. The number of computed poses is responsive to the sensed movement of the mobile device. A computed pose is selected for each feature point in the image by matching the particular capture time for the feature point to the particular computed time for the computed pose. The position of the mobile device is determined within the environment using the feature points and the selected computed poses for the feature points.

First claim

What is claimed is:

1. A method for visual-inertial tracking with a mobile device having a motion detector and a camera system including a first rolling shutter camera, the method comprising:
obtaining an initial pose for the first rolling shutter camera;
capturing, during a capture period, an image of an environment using the camera system, the image including feature points, each feature point captured at a particular capture time during the capture period;
sensing, during the capture period, movement of the mobile device with the motion detector;
computing a number of poses for the first rolling shutter camera based on the initial pose and the sensed movement, each computed pose corresponding to a particular computed time within the capture period, wherein the number of computed poses is responsive to the sensed movement of the mobile device, wherein the number of poses computed by the mobile device varies with the sensed movement of the mobile device;
selecting, for each feature point in the image, one of the number of computed poses by matching the particular capture time for the feature point to the particular computed time for the computed pose, wherein the number of poses is selected based on predefined thresholds; and
determining a position of the mobile device within the environment using the feature points and the selected computed poses for the feature points.

2. The method of claim 1 , further comprising:
storing each of the computed number of poses in a lookup table;
wherein the selecting includes retrieving, for each feature point in the image, the one of the number of computed poses from the lookup table with the corresponding particular capture time that is closest to the corresponding particular computed time.

3. The method of claim 1 , further comprising:
storing each of the computed number of poses in a lookup table;
wherein the selecting includes retrieving, for each feature point in the image, the one of the number of computed poses from the lookup table with the corresponding particular capture time that is immediately prior to the corresponding particular computed time.

4. The method of claim 1 , wherein the camera system is a multi-camera camera system further comprising at least a second rolling shutter camera.

5. The method of claim 4 , further comprising:
obtaining another initial pose for the second rolling shutter camera;
capturing, during another capture period, a second image of the environment using the second rolling shutter camera, the image including feature points, each feature point captured at a particular capture time during the other capture period;
sensing, at a plurality of times during the other capture period, movement of the mobile device with the motion detector;
computing a number of poses for the second camera based on the initial pose and the sensed movement, each second camera computed pose corresponding to a particular computed time within the other capture period, wherein the number of second camera computed poses is responsive to the sensed movement of the mobile device during the other capture period;
selecting, for each feature point in the second image, one of the number of second camera computed poses by matching the particular capture time for the feature point to the particular computed time for the computed pose; and
determining the position of the mobile device within the environment using the feature points for the first image, the feature points for the second image, and the selected computed poses for the feature points.

6. The method of claim 5 , the method further comprising:
storing each of the second camera computed number of poses in a lookup table;
wherein the selecting includes retrieving, for each feature point in the image, the one of the number of second camera computed poses from the lookup table with the corresponding particular capture time that is closest to the corresponding particular computed time.

7. The method of claim 5 , wherein the determining comprises:
storing each of the second camera computed number of poses in a lookup table;
wherein the selecting includes retrieving, for each feature point in the image, the one of the number of second camera computed poses from the lookup table with the corresponding particular capture time that is immediately prior to the corresponding particular computed time.

8. The method of claim 1 , wherein the determining the position comprises applying a simultaneous localization and mapping (SLAM) algorithm using the feature points and the selected computed poses for the feature points.

9. The method of claim 1 , wherein the sensing movement comprises at least one of:
sensing motion with an inertial measurement unit (IMU); or
sensing a rate of change in position of the mobile device responsive to coordinates from a global positioning system (GPS).

10. The method of claim 1 , wherein the number of computed poses is less than the number of feature points.

11. A visual-inertial tracking system for determining a position of an eyewear device within an environment, the system comprising:
a motion detector;
a first rolling shutter camera;
an eyewear device comprising the motion detector, the first rolling shutter camera, a processor, and a memory; and
programming in the memory, wherein execution of the programming by the processor configures the eyewear device to obtain an initial pose for the first rolling shutter camera, capture, during a capture period, an image of an environment using the camera system, the image including feature points, each feature point captured at a particular capture time during the capture period, sense, during the capture period, movement of the eyewear device with the motion detector, compute a number of poses for the first rolling shutter camera based on the initial pose and the sensed movement, each computed pose corresponding to a particular computed time within the capture period, wherein the number of computed poses is responsive to the sensed movement of the eyewear device and the number of poses computed by the eyewear device varies with the sensed movement of the eyewear device, select, for each feature point in the image, one of the number of computed poses by matching the particular capture time for the feature point to the particular computed time for the computed pose, wherein the number of poses is selected based on predefined thresholds; and determine a position of the eyewear device within the environment using the feature points and the selected computed poses for the feature points.

12. The system of claim 11 , further comprising:
a lookup table stored in memory, wherein each of the computed number of poses is stored in the lookup table.

13. The system of claim 12 , further comprising:
store, in memory, each of the computed number of poses in a lookup table; and
retrieve, from memory, for each feature point in the image, the one of the number of computed poses from the lookup table with the corresponding particular capture time that is closest to the corresponding particular computed time.

14. The system of claim 12 , further comprising:
store, in memory, each of the computed number of poses in a lookup table; and
retrieve, from memory, for each feature point in the image, the one of the number of computed poses from the lookup table with the corresponding particular capture time that is immediately prior to the corresponding particular computed time.

15. The system of claim 11 , wherein the system further comprises a multi-camera camera system comprising a second rolling shutter camera.

16. The sys

Assignees

• Snap Inc

Inventors

• Kalkgruber Matthias
• Mendez Erick Mendez
• Wagner Daniel
• Wolf Daniel
• Zhou Kai

Classifications

• G01C21/1656

  with passive imaging devices, e.g. cameras · CPC title

• H04N23/51

  Housings · CPC title

• G02B2027/0178

  Eyeglass type (eyeglass details G02C) · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title

• G02B2027/014

  comprising information/image processing systems · CPC title