Method of image processing for an augmented reality application

Patent metadata

Field	Value
Publication number	US-9685004-B2
Application number	US-201214430162-A
Country	US
Kind code	B2
Filing date	Sep 28, 2012
Priority date	Sep 28, 2012
Publication date	Jun 20, 2017
Grant date	Jun 20, 2017

Abstract

An apparatus for and method of image processing in an augmented reality application is provided. The method includes the steps of: providing at least one image of a real environment; performing image processing in an augmented reality application with the at least one image employing visualization of overlaying digital information with visual impressions or the image of the real environment and employing vision-based processing or tracking; and adjusting at least one of a parameter and operating flow of the vision-based processing or tracking depending on at least one of the following: a usage of the image processing, a usage of the visualization, a visually perceivable property of the digital information or the real environment, a property of a display device employed in the visualization, or a manner in which a user is viewing the visualization.

First claim

What is claimed is:

1. A method of image processing in an augmented reality application, comprising:
receiving, by the augmented reality application, an image of a real environment, the application being executed by one or more processors;
performing image processing in the augmented reality application using the image, the image processing employing (i) a visualization of digital information overlaid onto one or more of the image and the real environment, and (ii) a vision-based processing or tracking technique, wherein the visualization is presented using a display device, and wherein a threshold of a re-projection error of the digital information is adjusted according to a pixels-per-inch (PPI) parameter of the display device; and
adjusting one or more of a parameter and an operating flow of the vision-based processing or tracking technique.

2. The method according to claim 1 , wherein the vision-based processing or tracking technique comprises vision-based tracking, detection, localization, or classification.

3. The method according to claim 1 , wherein the parameter of the vision-based processing or tracking technique includes one or more of:
a variable having a value,
a number of features in the image matched with a set of reference features, and
a number of features in the image to be used for a pose estimation of a camera capturing the image.

4. The method according to claim 1 , wherein the operating flow of the vision-based processing or tracking technique includes a plurality of components, wherein at least one combination of the components is employed in the vision-based processing or tracking technique, and wherein the plurality of components comprises one or more of:
one or more sensors measuring a spatial property of an object in the image or the real environment, and
one or more mathematical methods used in computer vision or image processing.

5. The method according to claim 4 , wherein, when the plurality of components comprises the one or more sensors, the object comprises a space in one or more of the image and the real environment and the spatial property is a property relating to the space.

6. The method according to claim 5 , wherein the property relating to the space comprises a geometry relating to the space, and the geometry describes one or more of:
a shape relating to the space,
a symmetry relating to the space,
a planarity relating to the space,
a geometrical size relating to the space, and
a density relating to the space.

7. The method according to claim 4 , wherein, when the plurality of components comprises the one or more mathematical methods, the one or more mathematical methods include one or more of:
a linear equation solver,
a cost function, and
an optimization method.

8. The method according to claim 1 , wherein the display device comprises a device displaying one or more of the digital information and the real environment to the user such that the user can visually perceive the real environment through the device.

9. The method according to claim 1 , further comprising:
quantifying a performance of the vision-based processing or tracking technique based on a similarity measure of sum-of-squared differences, wherein the similarity measure is defined, at least in part, by a similarity threshold value; and
adjusting the similarity threshold value.

10. The method according to claim 1 , wherein, when the digital information includes a spherical object, the vision-based processing or tracking technique includes optimizing at least one parameter describing a translation of the spherical object, and not considering at least one parameter describing a rotation of the spherical object.

11. The method according to claim 1 , wherein the visualization includes an animation of the digital information, and wherein, when the animation of the digital information includes a movement of the digital information, the vision-based processing or tracking technique does not include a stable pose estimation of a camera capturing the image.

12. The method according to claim 11 , wherein a number of feature correspondences employed to estimate the camera pose is decreased.

13. The method according to claim 1 , wherein adjusting one or more of a parameter and an operating flow of the vision-based processing or tracking technique is based on one or more of:
a usage of the image processing;
a usage of the visualization;
a visually perceivable property of one or more of the digital information and the real environment;
a property of the display device employed in the visualization; and
a manner in which a user is viewing the visualization.

14. The method according to claim 13 , wherein, when the adjustment of one or more of the parameter and the operating flow is based, at least in part, on the usage of the image processing, the usage of the image processing includes one or more of:
detecting at least one part of the real environment,
performing a visual search of at least one part of the image or the real environment,
detecting classification problems associated with the image or the real environment,
estimating a pose of a capturing device capturing the image, or
tracking at least one part of the real environment.

15. The method according to claim 13 , wherein, when the adjustment of one or more of the parameter and the operating flow is based, at least in part, on the usage of the visualization, the usage of the visualization includes one or more of:
delivering information to the user that is associated with at least one part of the real environment; and
enhancing the user's visual perception of one or more of the digital information and the real environment by superimposing the digital information onto the real environment in a correct perspective.

16. The method according to claim 13 , wherein, when the adjustment of one or more of the parameter and the operating flow is based, at least in part, on the visually perceivable property, the visually perceivable property includes one or more of:
a spatial property associated with one or more of the digital information and the real environment,
a symmetry associated with one or more of the digital information and the real environment,
a behavior associated with one or more of the digital information and the real environment,
a color associated with one or more of the digital information and the real environment,
a type of the digital information, and
a type of the real environment.

17. The method according to claim 16 , wherein, when the visually perceivable property includes the behavior, the behavior comprises data describing a position or a change in the position of at least one part of the digital information or the real environment, and wherein, when the visually perceivable property includes the type of digital information, the type of digital information includes one or more of a text, a drawing, and a video.

18. The method according to claim 13 , wherein the manner in which the user is viewing the visualization comprises one or more of eye gazing and eye focusing.

19. A non-transitory computer readable medium comprising instructions for image processing in an augmented reality application, which when executed by one or more processors, causes the one or more processors to:
receive an image of a real environment;
perform image processing in the augmented reality application using the image, the application being executed by the one or more processors, and the image processing employing (i) a visualization of digital in

Assignees

• Apple Inc

Inventors

• Wang Lejing
• Benhimane Selim

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T2207/30244

  Camera pose · CPC title

• G06T2207/20004

  Adaptive image processing · CPC title

• G06T11/00

  Two-dimensional [2D] image generation · CPC title