Hybrid corner and edge-based tracking

Patent metadata

Field	Value
Publication number	US-10380763-B2
Application number	US-201715814924-A
Country	US
Kind code	B2
Filing date	Nov 16, 2017
Priority date	Nov 16, 2016
Publication date	Aug 13, 2019
Grant date	Aug 13, 2019

Abstract

A method includes acquiring, from a camera, an image frame including a representation of an object, and retrieving from a memory, data containing a template of a first pose of the object. A processor compares the first template to the image frame. A plurality of candidate locations in the image frame having a correlation with the template exceeding a predetermined threshold is determined. Edge registration on at least one candidate location of the plurality of candidate locations is performed to derive a refined pose of the object. Based at least in part on the performed edge registration, an initial pose of the object is determined, and a display image is output for display on a display device. The position at which the display image is displayed and/or the content of the display image is based at least in part on the determined initial pose of the object.

First claim

What is claimed is:

1. A non-transitory computer readable medium that embodies instructions that cause one or more processors to perform a method comprising:
acquiring, from a camera, an image frame representing an object in a real scene;
detecting, from the image frame, image first features using a first feature detection algorithm;
retrieving, from one or more memories, data containing (a) a preceding pose of the object, (b) preceding image first features detected in a preceding image frame prior to the image frame and corresponding to the preceding pose of the object, and (c) first 3D points corresponding to the preceding image first features;
matching the image first features and the preceding image first features to establish first correspondences between the image first features and the first 3D points;
deriving a candidate pose of the object corresponding to the image frame based on the preceding pose of the object and the first correspondences;
deriving (a) candidate pose second features using the candidate pose and (b) second 3D points corresponding to the candidate pose second features;
detecting, from the image frame, image second features using a second feature detection algorithm;
matching the image second features and the candidate pose second features to establish second correspondences between the image second features and the second 3D points; and
deriving a pose of the object in the image frame based on (a) the candidate pose, (b) the first correspondences, and (c) the second correspondences.

2. The non-transitory computer readable medium of claim 1 , wherein detecting first image features by a first feature detection algorithm comprises detecting corners of a representation of the object in the image frame using a corner detection algorithm.

3. The non-transitory computer readable medium of claim 2 , wherein detecting second image features by a second feature detection algorithm comprises detecting a plurality of contour points of the representation of the object using an edge detection algorithm.

4. The non-transitory computer readable medium of claim 1 , further comprising:
retrieving, from one or more memories, keyframe data containing (a) keyframe first features detected in a keyframe image frame prior to the image frame and corresponding to the keyframe pose of the object and (b) keyframe 3D points corresponding to the keyframe first features; and
matching the image first features and the keyframe first features to establish keyframe correspondences between the image first features and the keyframe 3D points, wherein deriving a candidate pose of the object corresponding to the image frame based on the preceding pose of the object and the first correspondences comprises deriving a candidate pose of the object corresponding to the image frame based on the preceding pose of the object, the first correspondences, and the keyframe correspondences.

5. The non-transitory computer readable medium of claim 4 , further comprising:
selectively updating the keyframe data based on the detected image first features and candidate pose.

6. The non-transitory computer readable medium of claim 1 , further comprising:
performing a hybrid refinement M times after deriving the pose of the object in the image frame, where M is an integer equal to or greater than 1, the hybrid refinement comprising:
deriving (a) second features using an N−1th pose and (b) second 3D points corresponding to the N−1th pose second features, where N is an integer value of a current performance of the hybrid refinement and the 0th pose is the pose;
detecting, from the image frame, image second features using the second feature detection algorithm;
matching the image second features and the N−1th pose second features to establish N−1th pose second correspondences between the image second features and the N−1th pose second 3D points; and
deriving an Nth pose of the object in the image frame based on (a) the candidate pose, (b) the first correspondences, and (c) the N−1th pose second correspondences.

7. The non-transitory computer readable medium of claim 6 , wherein M is three, and further comprising:
selecting the 3 rd pose of the object in the image frame as a final pose of the object in the image frame.

8. A method of detecting an initial pose of an object in an image frame, the method comprising:
acquiring, from a camera, an image frame representing an object in a real scene;
detecting, from the image frame, image first features using a first feature detection algorithm;
retrieving, from one or more memories, data containing (a) a preceding pose of the object, (b) preceding image first features detected in a preceding image frame prior to the image frame and corresponding to the preceding pose of the object, and (c) first 3D points corresponding to the preceding image first features;
matching the image first features and the preceding image first features to establish first correspondences between the image first features and the first 3D points;
deriving a candidate pose of the object corresponding to the image frame based on the preceding pose of the object and the first correspondences;
deriving (a) candidate pose second features using the candidate pose and (b) second 3D points corresponding to the candidate pose second features;
detecting, from the image frame, image second features using a second feature detection algorithm;
matching the image second features and the candidate pose second features to establish second correspondences between the image second features and the second 3D points; and
deriving a pose of the object in the image frame based on (a) the candidate pose, (b) the first correspondences, and (c) the second correspondences.

9. The method of claim 8 , wherein detecting first image features by a first feature detection algorithm comprises detecting corners of a representation of the object in the image frame using a corner detection algorithm.

10. The method of claim 9 , wherein detecting second image features by a second feature detection algorithm comprises detecting a plurality of contour points of the representation of the object using an edge detection algorithm.

11. The method of claim 8 , further comprising:
retrieving, from one or more memories, keyframe data containing (a) keyframe first features detected in a keyframe image frame prior to the image frame and corresponding to the keyframe pose of the object and (b) keyframe 3D points corresponding to the keyframe first features; and
matching the image first features and the keyframe first features to establish keyframe correspondences between the image first features and the keyframe 3D points, wherein deriving a candidate pose of the object corresponding to the image frame based on the preceding pose of the object and the first correspondences comprises deriving a candidate pose of the object corresponding to the image frame based on the preceding pose of the object, the first correspondences, and the keyframe correspondences.

12. The method of claim 11 , further comprising:
selectively updating the keyframe data based on the detected image first features and candidate pose.

13. The method of claim 8 , further comprising:
performing a hybrid refinement M times after deriving the pose of the object in the image frame, where M is an integer equal to or greater than 1, the hybrid refinement comprising:
deriving (a) second features using an N−1th pose and (b) second 3D points corresponding to the N−1th pose second features, where N is an integer value of a current performance of the hybrid refinement and the 0th pose is the pose;
detecting, from the image fr

Assignees

• Seiko Epson Corp

Inventors

• Levinshtein Alex
• Baig Qadeer
• Rotenstein Andrei Mark
• Zhao Yan

Classifications

• G06T9/20

  Contour coding, e.g. using detection of edges · CPC title

• G02B2027/0187

  slaved to motion of at least a part of the body of the user, e.g. head, eye · CPC title

• G06T2200/04

  involving 3D image data · CPC title

• G06T7/32

  using correlation-based methods · CPC title

• G06T7/75 Primary

  involving models · CPC title