Calibration of stereo cameras and handheld object

Patent metadata

Field	Value
Publication number	US-10503247-B2
Application number	US-201715826575-A
Country	US
Kind code	B2
Filing date	Nov 29, 2017
Priority date	May 9, 2017
Publication date	Dec 10, 2019
Grant date	Dec 10, 2019

Abstract

One disclosed example provides a head-mounted device including a stereo camera arrangement, a logic device configured to execute instructions, and a storage device storing instructions executable by the logic device to, for each camera in the stereo camera arrangement, receive image data of a field of view of the camera, detect light sources of a handheld object in the image data, and based upon the light sources detected, determine a pose of the handheld object. The instructions are executable to, based upon the pose of the handheld object determined for each camera in the stereo camera arrangement, calibrate the stereo camera arrangement.

First claim

The invention claimed is:

1. A head-mounted device, comprising:
a stereo camera arrangement;
a logic device configured to execute instructions; and
a storage device comprising instructions executable by the logic device to
for each camera in the stereo camera arrangement,
receive image data of a field of view of the camera,
detect a plurality of light sources of a handheld object in the image data, the plurality of light sources comprising a subset of light sources of the handheld object, and
based upon the plurality of light sources detected, determine a pose of the handheld object; and

based upon the pose of the handheld object determined for each camera in the stereo camera arrangement, calibrate the stereo camera arrangement.

2. The head-mounted device of claim 1 , wherein the instructions are executable to calibrate the stereo camera arrangement during one or more of a motion of the head-mounted device and a motion of the handheld object.

3. The head-mounted device of claim 1 , wherein the instructions are executable to calibrate intrinsic characteristics for each camera of the stereo camera arrangement, the intrinsic characteristics comprising one or more of lens distortion, focal length, and principal point.

4. The head-mounted device of claim 1 , wherein the instructions are executable to calibrate extrinsic characteristics for each camera of the stereo camera arrangement, the extrinsic characteristics comprising one or more of a rotation and a translation.

5. The head-mounted device of claim 1 , wherein the instructions are executable to determine the pose by fitting an observed light pattern to a model light pattern via a rigid body transformation.

6. The head-mounted device of claim 1 , wherein the instructions are executable to determine the pose by, for each light source detected in the image data received from a first camera, searching along an epipolar line for a companion pixel in the image data received from a second camera, the companion pixel corresponding to the light source detected in the image data received from the first camera.

7. The head-mounted device of claim 1 , wherein the instructions are executable to calibrate the stereo camera arrangement utilizing one or more of a Kalman filter, a sliding window bundle adjustment, and a recursive Gauss-Newton filter.

8. The head-mounted device of claim 1 , wherein the instructions are executable to calibrate the stereo camera arrangement between 6 and 10 times per minute.

9. A method for determining a calibration for a handheld object via a head-mounted device comprising a stereo camera arrangement, the method comprising:
receiving, from a first camera of the stereo camera arrangement, first image data from a perspective of the first camera;
receiving, from a second camera of the stereo camera arrangement, second image data from a perspective of the second camera;
detecting, in the first image data and the second image data, a plurality of light sources of the handheld object;
determining a three-dimensional position of each light source of the plurality of light sources detected; and
determining a calibration for the handheld object based upon each three-dimensional position determined.

10. The method of claim 9 , further comprising
receiving from an inertial measurement unit (IMU) of the handheld object a sensed orientation of the handheld object, and
determining a calibration for the IMU based upon the sensed orientation.

11. The method of claim 10 , wherein determining the calibration for the IMU comprises determining a calibration for a position and a rotation of the IMU.

12. The method of claim 9 , wherein determining the calibration of the handheld object comprises determining calibration parameters via one or more of a Kalman filter, a sliding window bundle adjustment, and a recursive Gauss-Newton filter.

13. The method of claim 9 , wherein determining the three-dimensional position of each light source comprises fitting an observed light pattern to a model light pattern via a rigid body transformation.

14. A head-mounted device, comprising:
a stereo camera arrangement;
a logic device configured to execute instructions; and
a storage device comprising instructions executable by the logic device to
for each camera in the stereo camera arrangement,
receive image data of a surrounding environment from a perspective of the camera,
detect via the image data a plurality of feature points of a handheld object in a field of view of the camera,
determine a location of each feature point detected, and
determine a pose of the handheld object from the perspective of the camera based on the location of each feature point determined;

update one or more calibration parameters for the stereo camera arrangement based on the pose of the handheld object determined for each camera;
determine a three-dimensional position of each feature point based on the location of each feature point determined for each camera of the stereo camera arrangement; and
update one or more calibration parameters for the handheld object based on the three-dimensional position of each feature point determined.

15. The head-mounted device of claim 14 , wherein the instructions are executable to receive the image data as a background process during use of the head-mounted device and the handheld object.

16. The head-mounted device of claim 14 , wherein the instructions are further executable to receive from an inertial measurement unit (IMU) of the handheld object a sensed orientation of the handheld object, and to determine a calibration for one or more of a translation and a rotation of the IMU based upon the sensed orientation.

17. The head-mounted device of claim 14 , wherein the one or more calibration parameters for the stereo camera arrangement comprise one or more of an intrinsic camera characteristic and an extrinsic camera characteristic.

18. The head-mounted device of claim 14 , wherein the instructions are executable to determine the pose of the handheld object by one or more of
fitting an observed feature point pattern to a model feature point pattern via a rigid body transformation; and
for each feature point detected in the image data received from a first camera of the stereo camera arrangement, searching along an epipolar line for a companion feature point in the image data received from a second camera of the stereo camera arrangement, the companion feature point corresponding to the feature point detected in the image data received from the first camera.

19. The head-mounted device of claim 14 , wherein the instructions are executable to update the one or more calibration parameters for the stereo camera arrangement and update the one or more calibration parameters for the handheld object simultaneously.

20. The head-mounted device of claim 14 , wherein the instructions are executable to update one or more calibration parameters for one or more of the stereo camera arrangement and the handheld object via one or more of a Kalman filter, a sliding window bundle adjustment, and a recursive Gauss-Newton filter.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Samples Michael Edward
• Balan Alexandru Octavian
• Sirtkaya Salim
• Jovanovic Vuk
• Panjevic Filip
• Khapko Taras
• Rivera Iii Ruven Jaime

Classifications

• G06T7/74

  involving reference images or patches · CPC title

• G06T7/85

  Stereo camera calibration · CPC title

• H05B45/10

  Controlling the intensity of the light · CPC title

• H05B47/11

  by determining the brightness or colour temperature of ambient light · CPC title

• H05B47/19

  via wireless transmission · CPC title