In situ creation of planar natural feature targets

Patent metadata

Field	Value
Publication number	US-11481982-B2
Application number	US-202016918277-A
Country	US
Kind code	B2
Filing date	Jul 1, 2020
Priority date	Mar 14, 2013
Publication date	Oct 25, 2022
Grant date	Oct 25, 2022

Abstract

Disclosed are a system, apparatus, and method for in-situ creation of planar natural feature targets. In one embodiment, a planar target is initialized from a single first reference image one or more subsequent images are processed. In one embodiment, the planar target is tracked in six degrees of freedom upon the processing of the one or more subsequent images and a second reference image is selected from the processed one or more subsequent images. In one embodiment, upon selecting the second reference image the planar target is refined to a more accurate planar target.

First claim

What is claimed is:

1. A device comprising:
a memory; and
a processor coupled to the memory, the processor configured to:
determine a first pose of a camera;
receive a first image associated with the first pose of the camera;
generate an initial virtual representation of a planar target based on the first image;
track the planar target as the camera moves from the first pose to a second pose, wherein the second pose is different than the first pose;
display the initial virtual representation of the planar target during at least a portion of the tracking of the planar target and until an updated virtual representation of the planar target is generated;
determine, based on tracking the planar target, a second image associated with the second pose of the camera; and
generate the updated virtual representation of the planar target based on the second image, wherein the updated virtual representation of the planar target is more accurate than the initial virtual representation of the planar target.

2. The device of claim 1 , wherein the processor is further configured to:
display the initial virtual representation of the planar target at a first time.

3. The device of claim 2 , wherein the processor is further configured to:
display, at a second time subsequent to the first time, the updated virtual representation of the planar target.

4. The device of claim 1 , wherein:
the processor is configured to generate a distorted virtual representation of the planar target as the camera moves from the first pose of the camera; and
to display the initial virtual representation of the planar target, the processor is configured to display the distorted virtual representation of the planar target.

5. The device of claim 4 , wherein the processor is configured to generate the distorted virtual representation prior to processing the second image.

6. The device of claim 1 , wherein the processor is further configured to:
assume a fronto-parallel view of the planar target.

7. The device of claim 6 , wherein the processor is further configured to:
generate a rectified view of the planar target from the first image or the second image if neither the first image nor the second image are fronto-parallel views of the planar target.

8. The device of claim 1 , wherein the processor is further configured to:
assign the entire first image as the planar target.

9. The device of claim 1 , wherein the processor is further configured to:
determine, for one or more features of the planar target in the first image, a respective reference feature location within the planar target.

10. The device of claim 1 , wherein, to generate the updated virtual representation of the planar target, the processor is configured to:
extract one or more reference features from the planar target in the first image;
extract one or more reference features from the second image;
match one or more reference features from the second image to one or more reference features from the planar target in the first image;
triangulate matching reference features from the second image and from the planar target in the first image; and
generate the updated virtual representation of the planar target based on the triangulation.

11. The device of claim 1 , wherein the planar target is one or more of: partially occluded in the first image, of any planar shape, or in any orientation relative to gravity.

12. The device of claim 1 , wherein the processor is configured to generate the updated virtual representation of the planar target based on tracking the planar target in six degrees of freedom.

13. A method comprising:
determining a first pose of a camera;
receiving a first image associated with the first pose of the camera;
generating an initial virtual representation of a planar target based on the first image;
tracking the planar target as the camera moves from the first pose to a second pose, wherein the second pose is different than the first pose;
displaying the initial virtual representation of the planar target during at least a portion of the tracking of the planar target;
determining, based on tracking the planar target, a second image associated with the second pose of the camera; and
generating an updated virtual representation of the planar target based on the second image, wherein the initial virtual representation of the planar target is displayed until the updated virtual representation of the planar target is generated, and wherein the updated virtual representation of the planar target is more accurate than the initial virtual representation of the planar target.

14. The method of claim 13 , further comprising:
displaying the initial virtual representation of the planar target at a first time.

15. The method of claim 14 , further comprising:
displaying, at a second time subsequent to the first time, the updated virtual representation of the planar target.

16. The method of claim 13 , further comprising:
generating a distorted virtual representation of the planar target as the camera moves from the first pose of the camera;
wherein displaying the initial virtual representation of the planar target includes displaying the distorted virtual representation of the planar target.

17. The method of claim 16 , wherein generating the distorted virtual representation includes generating the distorted virtual representation prior to processing the second image.

18. The method of claim 13 , further comprising:
assuming a fronto-parallel view of the planar target.

19. The method of claim 18 , further comprising:
generating a rectified view of the planar target from the first image or the second image if neither the first image nor the second image are fronto-parallel views of the planar target.

20. The method of claim 13 , further comprising:
assigning the entire first image as the planar target.

21. The method of claim 13 , further comprising:
determining, for one or more features of the planar target in the first image, a respective reference feature location within the planar target.

22. The method of claim 13 , wherein generating the updated virtual representation of the planar target includes:
extracting one or more reference features from the planar target in the first image;
extracting one or more reference features from the second image;
matching one or more reference features from the second image to one or more reference features from the planar target in the first image;
triangulating matching reference features from the second image and from the planar target in the first image; and
generating the updated virtual representation of the planar target based on the triangulation.

23. The method of claim 13 , wherein the planar target is one or more of: partially occluded in the first image, of any planar shape, or in any orientation relative to gravity.

24. The method of claim 13 , wherein generating the updated virtual representation of the planar target includes generating the updated virtual representation of the planar target based on tracking the planar target in six degrees of freedom.

25. A non-transitory computer-readable storage medium storing instructions that, when executed, cause one or more processors of a device to:
determine a first pose of a camera;
receive a first image associated with the first pose of the camera;
generate an initial virtual representation of a planar target based

Assignees

• Qualcomm Inc

Inventors

• Wagner Daniel
• Pan Qi

Classifications

• G06T2207/30244

  Camera pose · CPC title

• G06T7/246

  using feature-based methods, e.g. the tracking of corners or segments · CPC title

• G06T7/74 Primary

  involving reference images or patches · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title