Pose tracking for rolling shutter camera

Patent metadata

Field	Value
Publication number	US-12101557-B2
Application number	US-202318375813-A
Country	US
Kind code	B2
Filing date	Oct 2, 2023
Priority date	Sep 30, 2020
Publication date	Sep 24, 2024
Grant date	Sep 24, 2024

Abstract

A method and apparatus of tracking poses of a rolling-shutter camera in an augmented reality (AR) system is provided. The method and apparatus use camera information and inertial sensor readings from Inertial Measurement Unit (IMU) to estimate the pose of the camera at a reference line. Thereafter, relative pose changes at scanlines may be calculated using the inertial sensor data. The estimated reference pose of the camera is then further refined based on the visual information from the camera, the relative pose changes and the optimized reference line pose of a previous image. Thereafter, the estimate of the scanline poses may be updated using the relative pose changes obtained in the earlier steps.

First claim

What is claimed is:

1. An eyewear device comprising:
an inertial measurement unit (IMU) configured to capture IMU data;
a rolling shutter camera configured to capture image frames, each image frame comprising a plurality of image scan lines;
a first pipeline configured to process IMU data;
a second pipeline configured to process the IMU data and the image frames, the second pipeline comprising:
a selection module configured to select features within the image frames for tracking;
a feature tracking module configured to track the features from image frame to image frame;
a triangulation module configured to obtain three-dimensional (3D) locations corresponding to the features being tracked by computing spatial locations of the tracked features; and
an extended Kalman filter (EKF) module configured to calculate poses of the eyewear device using the 3D locations and the IMU data; and

a visual-inertial tracking system (VITS) core coupled to the first pipeline and the second pipeline, the VITS core configured to:
estimate a reference pose of the rolling shutter camera at a reference line, the reference line being one of the plurality of the image scan lines;
estimate relative poses of the rolling shutter camera to the reference pose of the rolling shutter camera using IMU data; and
refine the reference pose of the rolling shutter camera based on the relative poses of the rolling shutter camera in combination with information contained in the image frames.

2. The eyewear device of claim 1 , wherein the image frames are captured by the rolling shutter camera at a relatively low rate and the IMU data is captured by the IMU at a relatively high rate.

3. The eyewear device of claim 2 , wherein the relatively low rate is 30 cycles per second and the relatively high rate is 800 cycles per second.

4. The eyewear device of claim 3 , wherein the capture of image frames and the capture or IMU data are tightly coupled.

5. The eyewear device of claim 1 , wherein the first pipeline comprises a strap down integration update module configured to process IMU data.

6. The eyewear device of claim 1 , wherein the reference pose is estimated based on a refined reference pose of a previous image frame and the IMU data.

7. The eyewear device of claim 1 , wherein the selected features comprise at least one corner and at least one edge.

8. The eyewear device of claim 1 , wherein the IMU is next to the rolling shutter camera.

9. The eyewear device of claim 8 , further comprising:
a front façade supporting the IMU and the rolling shutter camera; and
a pair of arms extending from the front façade.

10. A visual-inertial tracking method for use with an eyewear device, the method comprising:
capturing inertial measurement unit (IMU) data with an IMU;
capturing image frames with a rolling shutter camera, each image frame comprising a plurality of image scan lines;
processing IMU data in a first pipeline;
processing the IMU data and the image frames in a second pipeline, the second pipeline comprising a selection module configured to select features within the image frames for tracking, a feature tracking module configured to track the features from image frame to image frame, a triangulation module configured to obtain three-dimensional (3D) locations corresponding to the features being tracked by computing spatial locations of the tracked features, and an extended Kalman filter (EKF) module configured to calculate poses of the eyewear device using the 3D locations and the IMU data;
estimating a reference pose of the rolling shutter camera at a reference line using a visual-inertial tracking system (VITS) core coupled to the first pipeline and the second pipeline, the reference line being one of the plurality of the image scan lines;
estimating, by the VITS core, relative poses of the rolling shutter camera to the reference pose of the rolling shutter camera using IMU data; and
refining, by the VITS core, the reference pose of the rolling shutter camera based on the relative poses of the rolling shutter camera in combination with information contained in the image frames.

11. The method of claim 10 , wherein the image frames are captured by the rolling shutter camera at a relatively low rate and the IMU data is captured by the IMU at a relatively high rate.

12. The method of claim 11 , wherein the relatively low rate is 30 cycles per second and the relatively high rate is 800 cycles per second.

13. The method of claim 12 , wherein the capture of image frames and the capture or IMU data are tightly coupled.

14. The method of claim 10 , wherein the first pipeline comprises a strap down integration update module configured to process IMU data.

15. The method of claim 10 , wherein the reference pose is estimated based on a refined reference pose of a previous image frame and the IMU data.

16. A non-transitory computer-readable medium with instructions stored thereon, wherein the medium is configured to be incorporated in an augmented reality system of an eyewear device configured to track poses of a rolling shutter camera, the instructions, when executed by a processor, configuring the eyewear device to perform steps comprising:
capturing inertial measurement unit (IMU) data with an IMU;
capturing image frames with the rolling shutter camera, each image frame comprising a plurality of image scan lines;
processing IMU data in a first pipeline;
processing the IMU data and the image frames in a second pipeline, the second pipeline comprising a selection module configured to select features within the image frames for tracking, a feature tracking module configured to track the features from image frame to image frame, a triangulation module configured to obtain three-dimensional (3D) locations corresponding to the features being tracked by computing spatial locations of the tracked features, and an extended Kalman filter (EKF) module configured to calculate poses of the eyewear device using the 3D locations and the IMU data;
estimating a reference pose of the rolling shutter camera at a reference line using a visual-inertial tracking system (VITS) core coupled to the first pipeline and the second pipeline, the reference line being one of the plurality of the image scan lines;
estimating, by the VITS core, relative poses of the rolling shutter camera to the reference pose of the rolling shutter camera using IMU data; and

refining, by the VITS core, the reference pose of the rolling shutter camera based on the relative poses of the rolling shutter camera in combination with information contained in the image frames.

17. The non-transitory computer-readable medium of claim 16 , wherein the image frames are captured by the rolling shutter camera at a relatively low rate and the IMU data is captured by the IMU at a relatively high rate.

18. The non-transitory computer-readable medium of claim 17 , wherein the relatively low rate is 30 cycles per second and the relatively high rate is 800 cycles per second.

19. The non-transitory computer-readable medium of claim 16 , wherein the first pipeline comprises a strap down integration update module configured to process IMU data.

20. The non-transitory computer-readable medium of claim 16 , wherein the reference pose is estimated based on a refined reference pose of a previous image frame and the IMU data.

Assignees

• Halmetschlager Funek Georg
• Hol Jeroen
• Kalkgruber Matthias
• Kuntner Nikolaj
• Wolf Daniel
• Snap Inc

Inventors

• Halmetschlager-Funek Georg
• Hol Jeroen
• Kalkgruber Matthias
• Kuntner Nikolaj
• Wolf Daniel

Classifications

• H04N25/531

  by controlling rolling shutters in CMOS SSIS · CPC title

• G06T7/73 Primary

  using feature-based methods · CPC title

• G06T2207/30244

  Camera pose · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title