Determining a pose of a handheld object

Patent metadata

Field	Value
Publication number	US-10679376-B2
Application number	US-201815961539-A
Country	US
Kind code	B2
Filing date	Apr 24, 2018
Priority date	Apr 24, 2018
Publication date	Jun 9, 2020
Grant date	Jun 9, 2020

Abstract

Examples are disclosed herein that relate to determining a pose of a handheld object. One example provides a computing system configured to determine a pose of a handheld object comprising a plurality of light sources by acquiring image data of a surrounding environment, detecting a subset of light sources of the plurality of light sources of the handheld object in the image data, and performing a search, without using previous pose data, to determine the pose of the handheld object relative to the computing system. The computing system is further configured to use the pose determined to perform a later search for an updated pose of the handheld object, and if the later search fails to find the updated pose, determine the updated pose by again performing the search without using previous pose data.

First claim

The invention claimed is:

1. A computing system, comprising
a logic subsystem; and
a storage subsystem comprising instructions executable by the logic subsystem to determine a pose of a handheld object comprising a plurality of light sources by
acquiring image data of a surrounding environment,
detecting a subset of light sources of the plurality of light sources of the handheld object in the image data,
performing a search, without using previous pose data, to determine the pose of the handheld object relative to the computing system by
determining a plurality of sets of 2D-3D point correspondences between the subset of light sources in the image data and light sources of the handheld object based upon comparing a 3D model of the handheld object to the image data;
filtering the plurality of sets of 2D-3D point correspondences based upon geometric properties of each set of 2D-3D point correspondences to form a filtered plurality of sets of 2D-3D point correspondences;
determining one or more candidate poses from the filtered plurality of sets of 2D-3D point correspondences via a perspective-n-point process; and
determining the pose of the handheld object from the one or more candidate poses;

taking an action on the computing system based on the pose determined,
using the pose determined to perform a later search for an updated pose of the handheld object, and
if the later search fails to find the updated pose, determining the updated pose by again performing the search without using previous pose data.

2. The computing system of claim 1 , wherein the instructions are executable to determine each set of 2D-3D point correspondences by
locating a cluster of at least four light sources within the image data,
selecting a selected light source in the cluster to be an anchor point,
determining a 2D-3D point correspondence for the anchor point in the 3D model of the handheld object,
determining a second 2D-3D point correspondence for a second light source in the cluster of at least four light sources based upon a proximity of the second light source to the anchor point, and
determining a third 2D-3D point correspondence for a third light source in the cluster of at least four light sources based upon a proximity of the third light source to the anchor point.

3. The computing system of claim 1 , wherein the instructions are executable to filter the plurality of sets of 2D-3D point correspondences by, for one or more sets of 2D-3D point correspondences, determining a triangle formed by three 2D-3D point correspondences in the image data, and rejecting the set of 2D-3D point correspondences based on one or more of a shape of the triangle and an orientation of the triangle.

4. The computing system of claim 1 , wherein the instructions are further executable to, for each of the one or more candidate poses, determine additional 2D-3D point correspondences for each remaining light source in the cluster of at least four light sources based on the 3D model of the handheld object.

5. The computing system of claim 4 , wherein the instructions are executable to, for each of the one or more candidate poses, validate the candidate pose based upon a reprojection error by reprojecting remaining 3D points of the candidate pose that correspond to remaining light sources in the image data, determining the reprojection error for the reprojected 3D points, and validating the candidate pose if the reprojection error is below a threshold error.

6. The computing system of claim 4 , wherein the instructions are executable to, for each of the one or more candidate poses, validate the candidate pose based upon an intensity error by controlling each light source of the cluster of at least four light sources on the handheld object, determining the intensity error using a predicted intensity of each light source based upon the candidate pose compared to a detected intensity for each light source, and validating the candidate pose if the intensity error is below a threshold error.

7. The computing system of claim 4 , wherein the instructions are executable to, for a selected candidate pose, reject the selected candidate pose if additional 2D-3D point correspondences cannot be determined for any remaining light source for the selected candidate pose.

8. The computing system of claim 4 , wherein the instructions are further executable to change the position of the 3D model of the handheld object when determining one or more of the additional 2D-3D point correspondences.

9. The computing system of claim 1 , wherein the instructions are further executable to use the pose determined to perform the later search for the updated pose of the handheld object by using the pose determined in combination with motion data regarding the handheld object.

10. The computing system of claim 1 , wherein the instructions are further executable to use the pose determined to perform the later search for the updated pose of the handheld object by translating the pose determined.

11. The computing system of claim 1 , wherein the pose determined comprises a candidate pose for which a target number of 2D-3D point correspondences are determined.

12. A computing system, comprising
a logic subsystem; and
a storage subsystem configured to store instructions executable by the logic subsystem to determine a pose of a handheld object comprising a plurality of light sources by
acquiring image data of a surrounding environment,
detecting, in the image data, a subset of light sources of the plurality of light sources of the handheld object,
pre-filtering the detected plurality of light sources in the image data for a cluster of a threshold number of light sources,
determining a plurality of sets of 2D-3D point correspondences between the subset of light sources in the image data and light sources of the handheld object based upon comparing a 3D model of the handheld object to the image data,
filtering the plurality of sets of 2D-3D point correspondences based upon geometric properties of each set of 2D-3D point correspondences to form a filtered plurality of sets of 2D-3D point correspondences,
determining one or more candidate poses from the filtered plurality of sets of 2D-3D point correspondences via a perspective-n-point process,
for each of the one or more candidate poses, determining additional 2D-3D point correspondences for remaining light sources in the subset of light sources,
determining the pose of the handheld object from the one or more candidate poses, and
taking an action on the computing system based on the pose determined.

13. The computing system of claim 12 , wherein the instructions are executable to filter the plurality of sets of 2D-3D point correspondences by, for one or more sets of 2D-3D point correspondences, determining a triangle formed by three 2D-3D point correspondences in the image data, and rejecting the set of 2D-3D point correspondences based on one or more of a shape of the triangle and an orientation of the triangle.

14. The computing system of claim 12 , wherein the instructions are executable to, for each of the one or more candidate poses, validate the candidate pose based upon a reprojection error by reprojecting remaining 3D points of the candidate pose that correspond to remaining light sources in the image data, determining the reprojection error for the reprojected 3D points, and validating the candidate pose if the reprojection error is below a threshold error.

15. The computing system of claim 12 , wherein the instructions are executable to, for each of the one or more candidate poses, validate the candidate pose

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Balan Alexandru Octavian
• Boskovic Ronald
• Panjevic Filip
• Razumenic Ivan
• Jovanovic Vuk

Classifications

• G06T7/60

  Analysis of geometric attributes · CPC title

• G06T7/70 Primary

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G02B27/0172

  characterised by optical features · CPC title