Dynamic extension of map data for object detection and tracking

Patent metadata

Field	Value
Publication number	US-9811731-B2
Application number	US-201414505345-A
Country	US
Kind code	B2
Filing date	Oct 2, 2014
Priority date	Oct 4, 2013
Publication date	Nov 7, 2017
Grant date	Nov 7, 2017

Abstract

A computer-implemented method of tracking a target object in an object recognition system includes acquiring a plurality of images with a camera and simultaneously tracking the target object and dynamically building online map data from the plurality of images. Tracking of the target object is based on the online map data and the offline map data. In one aspect, tracking the target object includes enabling only one of the online map data and offline map data for tracking based on whether tracking is successful. In another aspect, tracking the target object includes fusing the online map data with the offline map data to generate a fused online model.

First claim

What is claimed is:

1. A computer-implemented method of tracking a target object in an object recognition system, the method comprising:
acquiring a plurality of images with a camera;
acquiring offline map data of the target object; and
simultaneously tracking the target object and dynamically building online map data from the plurality of images, wherein tracking the target object includes selecting between the online map data and the offline map data for tracking the target object based at least in part on a size of the target object relative to an image in which the target object is being tracked.

2. The computer-implemented method of claim 1 , wherein tracking the target object includes:
estimating a target pose of the target object with respect to the camera, wherein the target pose estimation may be successful or unsuccessful;
determining whether the target pose estimation is successful; and if so,
enabling only one of the online map data and offline map data for tracking of the target object in a subsequent image.

3. The computer-implemented method of claim 2 , further comprising enabling both online map data and offline map data, if the tracking of the target object is not successful, for tracking of the target object in the subsequent image.

4. The computer-implemented method of claim 2 , wherein enabling only one of the online map data and the offline map data includes:
computing a region size of the target object;
enabling the offline map data and disabling the online map data if the region size of the target object is greater than a region threshold; and
enabling the online map data and disabling the offline map data if the region size of the target object is not greater than the region threshold.

5. The computer-implemented method of claim 4 , wherein computing the region size of the target object includes projecting a bounding box of offline map data features onto an acquired image.

6. The computer-implemented method of claim 4 , wherein the region threshold corresponds to the target object assuming at least half of an image.

7. The computer-implemented method of claim 1 , wherein the online map data is based on one or more keyframes corresponding with the plurality of images acquired with the camera.

8. The computer-implemented method of claim 1 , wherein building the online map data includes keyframe Simultaneous Localization and Mapping (SLAM).

9. The computer-implemented method of claim 1 , wherein building the online map data includes adding a candidate keyframe to one or more keyframes by computing a pose distance between the candidate keyframe and other online keyframes corresponding to images acquired by the camera.

10. The computer-implemented method of claim 1 , further comprising fusing the online map data with the offline map data to generate a fused online model of the target object and wherein tracking of the target object uses the fused online model when online map data is selected for tracking the target object.

11. The computer-implemented method of claim 10 , further comprising fusing the online map data with the offline map data including:
extracting one or more online features from at least one of the plurality of acquired images;
comparing an online feature with offline features included in the offline map data;
updating a descriptor of an offline feature and adding the updated offline feature to the fused online model if both the online feature and the offline feature correspond to a same 3D point of the target object; and
adding the online feature to the fused online model if the online feature corresponds to a new 3D point on the target object, where the new 3D point does not correspond to any offline feature.

12. A non-transitory computer-readable medium including program code stored thereon for tracking a target object in an object recognition system, the program code comprising instructions to:
acquire a plurality of images with a camera;
acquire offline map data of the target object; and
simultaneously track the target object and dynamically building online map data from the plurality of images, wherein the instruction to track the target object includes instruction to select between the online map data and the offline map data for tracking the target object based at least in part on a size of the target object relative to an image in which the target object is being tracked.

13. The non-transitory computer-readable medium of claim 12 , wherein the instructions to track the target object includes instructions to:
estimate a target pose of the target object with respect to the camera, wherein the target pose estimation may be successful or unsuccessful;
determine whether the target pose estimation is successful; and if so,
enable only one of the online map data and offline map data for tracking of the target object in a subsequent image.

14. The non-transitory computer-readable medium of claim 13 , further comprising instructions to enable both online map data and offline map data if the tracking of the target object is not successful for tracking of the target object in the subsequent image.

15. The non-transitory computer-readable medium of claim 13 , wherein enabling only one of the online map data and the offline map data includes instructions to:
compute a region size of the target object;
enable the offline map data and disabling the online map data if the region size of the target object is greater than a region threshold; and
enable the online map data and disable the offline map data if the region size of the target object is not greater than the region threshold.

16. The non-transitory computer-readable medium of claim 15 , wherein the instructions to compute the region size of the target object includes instructions to project a bounding box of offline map data features onto an acquired image.

17. The non-transitory computer-readable medium of claim 15 , wherein the region threshold corresponds to the target object assuming at least half of an image.

18. The non-transitory computer-readable medium of claim 12 , wherein the online map data is based on one or more keyframes corresponding with the plurality of images acquired with the camera.

19. The non-transitory computer-readable medium of claim 12 , wherein the instructions to build the online map data includes keyframe Simultaneous Localization and Mapping (SLAM).

20. The non-transitory computer-readable medium of claim 12 , wherein the instructions to build the online map data includes instructions to add a candidate keyframe to one or more keyframes by computing a pose distance between the candidate keyframe and other online keyframes corresponding to images acquired by the camera.

21. The non-transitory computer-readable medium of claim 12 , further comprising instructions to fuse the online map data with the offline map data to generate a fused online model of the target object and wherein tracking of the target object uses the fused online model when online map data is selected for tracking the target object.

22. The non-transitory computer-readable medium of claim 21 , further comprising instructions to fuse the online map data with the offline map data including instructions to:
extract one or more online features from at least one of the plurality of acquired images;
compare the one or more online features with offline features included in the offline map data;
update a descriptor of an offline feature and add the updated offline feature

Assignees

• Qualcomm Inc

Inventors

• Lee Wonwoo
• Pan Qi
• Maggio Emilio
• Tallonneau Romain

Classifications

• G06V20/20 Primary

  in augmented reality scenes · CPC title

• G06K9/00624 Primary

  Physics · mapped topic

• G06K9/6202

  Physics · mapped topic

• G06K9/4604

  Physics · mapped topic

• G06K9/00671

  Physics · mapped topic