Method of determining a position and orientation of a device associated with a capturing device for capturing at least one image

Patent metadata

Field	Value
Publication number	US-9990726-B2
Application number	US-201214430170-A
Country	US
Kind code	B2
Filing date	Sep 27, 2012
Priority date	Sep 27, 2012
Publication date	Jun 5, 2018
Grant date	Jun 5, 2018

Abstract

A method of determining a position and orientation of a device is provided. The position and orientation of the device is determined based on multiple degrees of freedom (DoF) and the device is associated with a capturing device for capturing at least one image is provided. The method includes: capturing at least one image of a real object with the capturing device, and providing a coordinate system in relation to the object; providing an estimation of intrinsic parameters of the capturing device; providing pose data to compute first and second DoFs in the coordinate system, with each DoF having a confidence degree; determining an initial pose of the device; performing a pose estimation process, and calculating in the pose estimation process an estimation of the DoFs having a second confidence degree; and determining a position and orientation of the device.

First claim

What is claimed is:

1. A method of determining a position and orientation of a capturing device, the method comprising:
receiving an image depicting a real object from the capturing device;
determining a coordinate system in relation to the real object;
determining a first value of a first degree of freedom of the capturing device in the coordinate system based on first pose data, wherein the first value is associated with a first confidence degree;
determining a second value of a second degree of freedom of the capturing device in the coordinate system based on second pose data, wherein the second value is associated with a second confidence degree different from the first confidence degree;
determining an initial pose of the capturing device with respect to the real object based on the first value of the first degree of freedom and the second value of the second degree of freedom;
determining an updated value of the second degree of freedom of the capturing device based on the second confidence degree, the initial pose and the image; and
determining a position and an orientation of the capturing device in the coordinate system based on the first value of the first degree of freedom and the updated value of the second degree of freedom.

2. The method according to claim 1 , wherein the first pose data includes first sensor data from a first sensor and the second pose data includes second sensor data from a second sensor.

3. The method according to claim 2 , wherein the first confidence degree is associated with the first sensor and the second confidence degree is associated with the second sensor.

4. The method according to claim 2 , wherein the first sensor includes an accelerometer, a gyroscope, a global position sensor, an inertial sensor, a magnetometer, an odometer, a mechanical sensor, a rotary encoder, a tracking sensor, a measuring arm, a laser tracker, or a combination thereof.

5. The method according to claim 1 , wherein the first pose data indicates a gravity vector of the capturing device.

6. The method according to claim 5 , wherein the first pose data is provided from a gravity sensor or combination of sensors indicative of the gravity vector.

7. The method according to claim 1 , further comprising:
projecting a digital representation of the real object into the image based on the initial pose; and
determining the updated value of the second degree of freedom based on the digital representation of the real object.

8. The method according to claim 7 , wherein determining the updated value includes:
matching a first feature of the real object depicted in the image with a second feature of the digital representation to identify a feature correspondence; and
determining the updated value based on the feature correspondence.

9. The method according to claim 1 , wherein the first confidence degree is greater than the second confidence degree.

10. The method according to claim 1 , further including receiving user input that comprises interaction with the capturing device, the image, a digital representation of the real object, or a projection of the digital representation of the real object into the image.

11. The method according to claim 10 , wherein the interaction includes at least one of: moving the capturing device with respect to the real object or inputting a command to translate, scale or rotate the digital representation.

12. The method according to claim 10 , wherein the interaction includes performing a selection, segmentation, or drawing on a view of the real object in the image.

13. The method according to claim 10 , wherein the second value of the second degree of freedom is determined based further on the user input.

14. The method according to claim 10 , wherein the interaction includes initiating a transformation of the projection of the digital representation into a second projection of the digital representation, wherein the second value of the second confidence degree is determined based further on the transformation.

15. The method according to claim 8 , wherein the feature correspondence has approximately the same orientation as the projection of the digital representation.

16. The method according to claim 8 , wherein at least one of search direction or search range for determining the feature correspondence is constrained by the first value of the first degree of freedom based on the first confidence degree.

17. The method according to claim 1 , further comprising determining a score indicating a level of success of an object localization by:
projecting a digital representation of the real object into the image based on the determined position and orientation of the capturing device; and
comparing an orientation of the digital representation to an orientation of the real object in the image.

18. The method according to claim 1 , further comprising initializing an image-to-image tracking process based on the determined position and orientation of the capturing device.

19. The method according to claim 18 , wherein the image-to-image tracking process is configured to disregard the second degree of freedom based on the second confidence degree.

20. The method according to claim 18 , wherein the image-to-image tracking process includes detecting correspondences between another image and a projected digital representation of the real object in the image.

21. A non-transitory computer readable medium comprising software code executable to perform a method of determining a position and orientation of a device, the method comprising:
receiving an image depicting a real object from a capturing device;
determining a coordinate system in relation to the real object;
determining a first value of a first degree of freedom of the capturing device in the coordinate system based on first pose data, wherein the first value is associated with a first confidence degree;
determining a second value of a second degree of freedom of the capturing device in the coordinate system based on second pose data, wherein the second value is associated with a second confidence degree different from the first confidence degree;
determining an initial pose of the capturing device with respect to the real object based on the first value of the first degree of freedom and the second value of the second degree of freedom;
determining an updated value of the second degree of freedom of the capturing device based on the second confidence degree, the initial pose and the image; and
determining a position and an orientation of the capturing device in the coordinate system based on the first value of the first degree of freedom and the updated value of the second degree of freedom.

22. The method according to claim 11 , wherein the interaction includes moving, pinching, or rotation gestures detected by a touch screen or a gesture recognition system.

23. The method of claim 1 , wherein the updated value of the second degree of freedom is determined in response to the second confidence degree satisfying a confidence threshold.

24. The method of claim 1 , further comprising determining an intrinsic parameter of the capturing device, wherein the updated value of the second degree of freedom is determined based further on the intrinsic parameter of the capturing device.

25. An apparatus comprising:
one or more processors; and
a memory storing instructions that, when executed by the one or more processors, cause

Assignees

• Apple Inc

Inventors

• Langer Marion
• Meier Peter
• Benhimane Selim

Classifications

• G06T7/33

  using feature-based methods · CPC title

• G01S5/0247 Primary

  Determining attitude · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• H04N23/635

  Region indicators; Field of view indicators · CPC title

• H04N23/631

  Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters · CPC title