Natural user interface camera calibration

Patent metadata

Field	Value
Publication number	US-10088971-B2
Application number	US-201414566410-A
Country	US
Kind code	B2
Filing date	Dec 10, 2014
Priority date	Dec 10, 2014
Publication date	Oct 2, 2018
Grant date	Oct 2, 2018

Abstract

A method of calibrating a six-degree-of-freedom (6DoF) pose of a natural user interface (NUI) camera relative to a display is provided. Calibration video imaging an environment from a calibration perspective, which sites the display and one or more features, is received from the NUI camera or a calibration camera. A three-dimensional map of the environment, which defines a 6DoF pose of the display and a three-dimensional location of each of the one or more features, is modeled from the calibration video. Primary video imaging the environment from an operation perspective, which sites the one or more features, is received from the NUI camera. A 6DoF pose of the NUI camera is found within the three-dimensional map of the environment based on the operation perspective view of the one or more features.

First claim

The invention claimed is:

1. A method of calibrating a six-degrees-of-freedom (6DoF) pose of a natural user interface (NUI) camera relative to a display:
receiving, from a calibration camera, calibration video imaging an environment from a calibration perspective that sites the display and one or more features;
three-dimensionally modeling, from the calibration video, a three-dimensional map of the environment, the three-dimensional map of the environment defining a 6DoF pose of the display within the environment and a three-dimensional location of each of the one or more features within the environment;
receiving, from the NUI camera, primary video imaging the environment from an operation perspective that sites the one or more features; and
finding a 6DoF pose of the NUI camera within the three-dimensional map of the environment based on the operation perspective view of the one or more features.

2. The method of claim 1 , further comprising:
receiving, from the NUI camera, primary video imaging a user in the environment;
identifying three-dimensional locations of a first body part of the user and a second body part of the user relative to the display; and
visually indicating, on the display, an intersection point of the display and a ray extending from the first body part through the second body part.

3. The method of claim 1 , further comprising:
translating movement of a second body part of a user towards the display as an input command directed to an object displayed at an intersection point of the display and a ray extending from a first body part of the user through the second body part.

4. The method of claim 3 , wherein movement of the second body part towards the display is determined based on a distance between the second body part and a body of the user.

5. The method of claim 3 , wherein movement of the second body part towards the display is determined based on a distance between the second body part and the first body part.

6. The method of claim 3 , further comprising:
presenting a display object on the display; and
selecting the display object responsive to movement of the second body part towards the display when the intersection point is coincident with the display object.

7. The method of claim 2 , wherein the first body part is an eye of the user and the second body part is a finger of the user.

8. The method of claim 2 , wherein the first body part is an elbow of the user and the second body part is a finger of the user.

9. The method of claim 1 , wherein the primary video sites the calibration camera.

10. The method of claim 9 , wherein the calibration camera includes two or more reference markings visible to the NUI camera.

11. The method of claim 9 , wherein the calibration camera includes a display screen configured to present a reference image visible to the NUI camera.

12. The method of claim 1 , wherein the NUI camera is a depth camera.

13. The method of claim 1 , wherein the display is configured to present a calibration image visible to the calibration camera.

14. The method of claim 1 , wherein the calibration camera is included in a head-mounted augmented reality device wearable by a user.

15. A method of calibrating a six-degrees-of-freedom (6DoF) pose of a natural user interface (NUI) camera relative to a display:
receiving, from the NUI camera, calibration video imaging an environment from a calibration perspective that sites the display and one or more features;
three-dimensionally modeling, from the calibration video, a three-dimensional map of the environment, the three-dimensional map of the environment defining a 6DoF pose of the display within the environment and a three-dimensional location of each of the one or more features within the environment;
receiving, from the NUI camera, primary video imaging the environment, from an operation perspective that sites the one or more features; and
finding a 6DoF pose of the NUI camera within the three-dimensional map of the environment based on the operation perspective view of the one or more features.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Bastien David
• Kennedy Kevin Alan

Classifications

• G06T2207/30196

  Human being; Person · CPC title

• H04N13/271

  wherein the generated image signals comprise depth maps or disparity maps · CPC title

• G06T2207/30244

  Camera pose · CPC title

• G06T7/80

  Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration · CPC title

• H04N13/207

  using a single two-dimensional [2D] image sensor · CPC title