Efficient augmented reality display calibration

Patent metadata

Field	Value
Publication number	US-10334240-B2
Application number	US-201615338049-A
Country	US
Kind code	B2
Filing date	Oct 28, 2016
Priority date	Oct 28, 2016
Publication date	Jun 25, 2019
Grant date	Jun 25, 2019

Abstract

Techniques of augmented reality device calibration are disclosed. In some example embodiments, a system calibrates a visual inertial navigation (VIN) camera of a head mounted display (HMD) device with at least one eye camera of the HMD device to generate multi-camera calibration parameters, calibrating the at eye camera(s) with a display module of the HMD to generate display calibration parameters, calibrating the IMU with the VIN camera to generate VIN calibration parameters, and calibrating the IMU to the display module using the multi-camera calibration parameters, the display calibration parameters, and the VIN calibration parameters.

First claim

What is claimed is:

1. A computer-implemented method comprising:
calibrating multiple cameras of a head mounted display (HMD) device with respect to one another, the multiple cameras comprising at least one eye camera, the at least one eye camera simulating the positioning of at least one human eye within an interior space of the HMD device looking through a display screen of the HMD device to an exterior space of the HMD, the multiple cameras further comprising a visual inertial navigation (VIN) camera, the VIN camera being configured to be used with an inertial measurement unit (IMU) of the HMD device to determine a VIN state, the calibrating of the multiple cameras comprising calculating multi-camera calibration parameters;
calibrating the at least one eye camera with a display module of the HMD, the display module being configured to display AR content on the display screen, the calibrating of the at least one eye camera with the display module comprising calculating display calibration parameters;
calibrating the IMU to the VIN camera, the calibrating of the IMU to the VIN camera comprising calculating VIN calibration parameters;
calibrating, by a machine having a memory and at least one processor, one of a plurality of components of the HMD device with another one of the plurality of components of the HMD device using the multi-camera calibration parameters and the display calibration parameters, the plurality of components comprising the display module and at least one of the multiple cameras, the calibrating of the one of a plurality of components of the HMD device with another one of the plurality of components of the HMD device comprising calibrating the IMU to the display module using the multi-camera calibration parameters, the display calibration parameters, and the VIN calibration parameters;
performing a loop closure process using the HMD device using the VIN calibration parameters; and
validating the VIN calibration parameters based on the loop closure process and ground-truth data provided by a trajectory of a calibration robot.

2. The computer-implemented method of claim 1 , wherein the multiple cameras comprise at least one additional camera other than the VIN camera and the at least one eye camera.

3. The computer-implemented method of claim 1 , wherein the calibrating of the multiple cameras comprises receiving VIN data and eye camera data, the VIN data having been captured by the VIN camera and the eye camera data having been captured by the at least one eye camera while the VIN camera, the at least one eye camera, and the HMD device were physically coupled to a calibration robot, and the calculating of the multi-camera calibration parameters comprises calculating the multi-camera calibration parameters using the VIN data and the eye camera data while the HMD device is not physically coupled to the calibration robot.

4. The computer-implemented method of claim 1 , wherein the calibrating of the at least one eye camera to the display module comprises receiving eye camera data, the eye camera data having been captured by the at least one eye camera while the VIN camera, the at least one eye camera, and the HMD device were physically coupled to a calibration robot, and the calculating of the display calibration parameters comprises calculating the display calibration parameters using the eye camera data while the HMD device is not physically coupled to the calibration robot.

5. The computer-implemented method of claim 1 , wherein the calibrating of the IMU to the VIN camera comprises receiving VIN data, the VIN data having been captured by the VIN camera while the VIN camera and the HMD device were physically coupled to a calibration robot, the calculating of the VIN calibration parameters comprising calculating the VIN calibration parameters using the VIN data while the HMD device is not coupled to the calibration robot.

6. The computer-implemented method of claim 1 , wherein the HMD device comprises a helmet.

7. The computer-implemented method of claim 1 , wherein the display screen is transparent.

8. The computer-implemented method of claim 1 , wherein the display module comprises at least one projector configured to project the AR content onto the display screen.

9. A system comprising:
at least one processor; and
a non-transitory computer-readable medium storing executable instructions that, when executed, cause the at least one processor to perform operations comprising:
calibrating a visual inertial navigation (VIN) camera of a head mounted display (HMD) device to at least one eye camera of the HMD device, the at least one eye camera simulating the positioning of at least one human eye within an interior space of the HMD device looking through a display screen of the HMD device to an exterior space of the HMD, the VIN camera being configured to be used with an inertial measurement unit (IMU) of the HMD device to determine a VIN state, the calibrating of the VIN camera to the at least one eye camera comprising calculating multi-camera calibration parameters;
calibrating the at least one eye camera to a display module of the HMD, the display module being configured to display AR content on the display screen, the calibrating of the at least one eye camera to the display module comprising calculating display calibration parameters;
calibrating the IMU to the VIN camera, the calibrating of the IMU to the VIN camera comprising calculating VIN calibration parameters;
calibrating the IMU to the display module using the multi-camera calibration parameters, the display calibration parameters, and the VIN calibration parameters;
performing a loop closure process using the HMD device using the VIN calibration parameters; and
validating the VIN calibration parameters based on the loop closure process and ground-truth data provided by a trajectory of a calibration robot.

10. The system of claim 9 , wherein the HMD device comprises a helmet.

11. The system of claim 9 , wherein the calibrating of the VIN camera to the at least one eye camera comprises receiving VIN data and eye camera data, the VIN data having been captured by the VIN camera and the eye camera data having been captured by the at least one eye camera while the VIN camera, the at least one eye camera, and the HMD device were physically coupled to a calibration robot, and the calculating of the multi-camera calibration parameters comprises calculating the multi-camera calibration parameters using the VIN data and the eye camera data while the HMD device is not physically coupled to the calibration robot.

12. The system of claim 9 , wherein the calibrating of the at least one eye camera to the display module comprises receiving eye camera data, the eye camera data having been captured by the at least one eye camera while the VIN camera, the at least one eye camera, and the HMD device were physically coupled to a calibration robot, and the calculating of the display calibration parameters comprises calculating the display calibration parameters using the eye camera data while the HMD device is not physically coupled to the calibration robot.

13. The system of claim 9 , wherein the calibrating of the IMU to the VIN camera comprises receiving VIN data, the VIN data having been captured by the VIN camera while the VIN camera and the HMD device were physically coupled to a calibration robot, the calculating of the VIN calibration parameters comprising calculating the VIN calibration parameters using the VIN data while the HMD device is not coupled to the calibration robot.

14. The system of claim 9 , wherein the display screen is transparent.

15. Th

Assignees

• Daqri Llc

Inventors

• Zhao Wenyi
• Broaddus Christopher
• Murarka Aniket
• Nasery Varun
• Drew Brendan
• Chu Chen-Chi
• Gunn Lawrence

Classifications

• H04N13/327

  Calibration thereof · CPC title

• H04N13/246

  Calibration of cameras · CPC title

• H04N13/243

  using three or more two-dimensional [2D] image sensors · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title

• G02B2027/014

  comprising information/image processing systems · CPC title