Natural user interface camera calibration

Patent metadata

Field	Value
Publication number	US-2016170603-A1
Application number	US-201414566410-A
Country	US
Kind code	A1
Filing date	Dec 10, 2014
Priority date	Dec 10, 2014
Publication date	Jun 16, 2016
Grant date	—

Abstract

A method of calibrating a six-degree-of-freedom (6DoF) pose of a natural user interface (NUI) camera relative to a display is provided. Calibration video imaging an environment from a calibration perspective, which sites the display and one or more features, is received from the NUI camera or a calibration camera. A three-dimensional map of the environment, which defines a 6DoF pose of the display and a three-dimensional location of each of the one or more features, is modeled from the calibration video. Primary video imaging the environment from an operation perspective, which sites the one or more features, is received from the NUI camera. A 6DoF pose of the NUI camera is found within the three-dimensional map of the environment based on the operation perspective view of the one or more features.

First claim

1 . A method of calibrating a six-degrees-of-freedom (6DoF) pose of a natural user interface (NUI) camera relative to a display:
receiving, from a calibration camera, calibration video imaging an environment from a calibration perspective that sites the display and one or more features; three-dimensionally modeling, from the calibration video, a three-dimensional map of the environment, the three-dimensional map of the environment defining a 6DoF pose of the display within the environment and a three-dimensional location of each of the one or more features within the environment; receiving, from the NUI camera, primary video imaging the environment from an operation perspective that sites the one or more features; and finding a 6DoF pose of the NUI camera within the three-dimensional map of the environment based on the operation perspective view of the one or more features.

2 . The method of claim 1 , further comprising:
receiving, from the NUI camera, primary video imaging a user in the environment; identifying three-dimensional locations of a first body part of the user and a second body part of the user relative to the display; and visually indicating, on the display, an intersection point of the display and a ray extending from the first body part through the second body part.

3 . The method of claim 1 , further comprising:
translating movement of a second body part of a user towards the display as an input command directed to an object displayed at an intersection point of the display and a ray extending from a first body part of the user through the second body part.

4 . The method of claim 3 , wherein movement of the second body part towards the display is determined based on a distance between the second body part and a body of the user.

5 . The method of claim 3 , wherein movement of the second body part towards the display is determined based on a distance between the second body part and the first body part.

6 . The method of claim 3 , further comprising:
presenting a display object on the display; and selecting the display object responsive to movement of the second body part towards the display when the intersection point is coincident with the display object.

7 . The method of claim 2 , wherein the first body part is an eye of the user and the second body part is a finger of the user.

8 . The method of claim 2 , wherein the first body part is an elbow of the user and the second body part is a finger of the user.

9 . The method of claim 1 , wherein the primary video sites the calibration camera.

10 . The method of claim 9 , wherein the calibration camera includes two or more reference markings visible to the NUI camera.

11 . The method of claim 9 , wherein the calibration camera includes a display screen configured to present a reference image visible to the NUI camera.

12 . The method of claim 1 , wherein the NUI camera is a depth camera.

13 . The method of claim 1 , wherein the display is configured to present a calibration image visible to the calibration camera.

14 . The method of claim 1 , wherein the calibration camera is included in a head-mounted augmented reality device wearable by a user.

15 . A method of using a depth camera to control a computing system, comprising:
receiving, at the computing system, a three-dimensional model defining a 6DoF pose of a display relative to a 6DoF pose of the depth camera within an environment; receiving, at the computing system, a machine-readable body model of a user within the environment, the machine-readable body model including a three-dimensional location of a first body part of the user and a three-dimensional location of a second body part of the user; during a first condition, translating a position of the first body part of the user and a position of the second body part of the user into a physical interaction zone gesture that controls the computing system; and during a second condition, translating a position of the first body part of the user and a position of the second body part of the user relative to the display into a pointing gesture that controls the computing system.

16 . The method of claim 15 , further comprising:
transitioning from the first condition to the second condition responsive to receiving a command from the user.

17 . The method of claim 15 , further comprising:
transitioning from the first condition to the second condition responsive to receiving instructions from an application executing on the computing system.

18 . The method of claim 15 , further comprising:
transitioning from the first condition to the second condition responsive to recognizing that a distance of the second body part relative to the display has decreased below a threshold.

19 . The method of claim 15 , further comprising:
during the second condition, visually indicating, on the display, an intersection point of the display and a ray extending from the first body part through the second body part.

20 . A method of calibrating a six-degrees-of-freedom (6DoF) pose of a natural user interface (NUI) camera relative to a display:
receiving, from the NUI camera, calibration video imaging an environment from a calibration perspective that sites the display and one or more features; three-dimensionally modeling, from the calibration video, a three-dimensional map of the environment, the three-dimensional map of the environment defining a 6DoF pose of the display within the environment and a three-dimensional location of each of the one or more features within the environment; receiving, from the NUI camera, primary video imaging the environment, from an operation perspective that sites the one or more features; and finding a 6DoF pose of the NUI camera within the three-dimensional map of the environment based on the operation perspective view of the one or more features.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Bastien David
• Kennedy Kevin Alan

Classifications

• G06T15/55

  Radiosity · CPC title

• G06T2207/30244

  Camera pose · CPC title

• H04N13/0253

  Electricity · mapped topic

• H04N13/0246

  Electricity · mapped topic

• H04N13/0271

  Electricity · mapped topic