Adaptable framework for cloud assisted augmented reality

Patent metadata

Field	Value
Publication number	US-9633447-B2
Application number	US-201615179936-A
Country	US
Kind code	B2
Filing date	Jun 10, 2016
Priority date	Sep 20, 2010
Publication date	Apr 25, 2017
Grant date	Apr 25, 2017

Abstract

A mobile platform efficiently processes image data, using distributed processing in which latency sensitive operations are performed on the mobile platform, while latency insensitive, but computationally intensive operations are performed on a remote server. The mobile platform acquires image data, and determines whether there is a trigger event to transmit the image data to the server. The trigger event may be a change in the image data relative to previously acquired image data, e.g., a scene change in an image. When a change is present, the image data may be transmitted to the server for processing. The server processes the image data and returns information related to the image data, such as identification of an object in an image or a reference image or model. The mobile platform may then perform reference based tracking using the identified object or reference image or model.

First claim

What is claimed is:

1. A method comprising:
acquiring image data using a mobile platform, wherein the image data is from at least one captured image of an object;
tracking the object with visual based tracking using the at least one captured image of the object;
determining whether there is a trigger event comprising a change in the image data relative to previously acquired image data, wherein the trigger event comprises a scene change in which a different object appears in the at least one captured image with respect to a previous captured image;
transmitting the image data to a server when there is the trigger event while continuing to track the object with visual based tracking using the at least one captured image of the object;
receiving information related to the image data from the server, wherein the information related to the image data comprises at least one of the following: a two dimensional (2D) model of the different object, a three dimensional (3D) model of the different object, a three-dimensional coordinate estimation of points on the different object, augmentation information, saliency information about the different object, and information related to object matching; and
tracking the different object using the information related to the image data received from the server.

2. The method of claim 1 , wherein tracking the object further comprises using a reference image of the object received from the server.

3. The method of claim 1 , the method further comprising determining a quality of the at least one captured image before transmitting the image data to the server, wherein the image data is transmitted to the server only if the quality of the at least one captured image is better than a threshold.

4. The method of claim 3 , wherein determining the quality of the at least one captured image comprises at least one of analyzing a degree of sharpness of the at least one captured image, analyzing a number of detected corners in the at least one captured image; and using statistics derived from the at least one captured image with a learning classifier.

5. The method of claim 1 , further comprising rendering augmentation with respect to the different object based on the information related to the image data received from the server.

6. The method of claim 1 , wherein the information related to the image data comprises an identification of the different object.

7. The method of claim 1 , wherein the at least one captured image comprises a plurality of objects and the information related to the image data comprises an identification of the plurality of objects.

8. The method of claim 7 , further comprising:
obtaining poses for each of the plurality of objects with respect to the mobile platform; and
tracking each of the plurality of objects using the poses and the information related to the image data.

9. The method of claim 1 , further comprising:
obtaining a pose of the mobile platform with respect to the different object; and
wherein tracking the different object uses the pose and the information related to the image data.

10. The method of claim 9 , wherein the information related to the image data comprises a reference image of the different object, and wherein obtaining the pose comprises receiving from the server a first pose based on the at least one captured image and the reference image.

11. The method of claim 10 , wherein continuing to track the object with visual based tracking comprises performing reference-free tracking of the object until the first pose is received from the server.

12. The method of claim 10 , further comprising:
acquiring a second captured image of the different object when the first pose is received from the server;
tracking the different object between the at least one captured image and the second captured image to determine an incremental change; and
using the incremental change and the first pose to obtain the pose of the mobile platform with respect to the different object.

13. The method of claim 10 , further comprising:
acquiring a second captured image of the different object;
detecting the different object in the second captured image using the reference image;
using the different object detected in the second captured image and the reference image to obtain the pose of the mobile platform with respect to the different object; and
using the pose to initialize reference based tracking of the different object.

14. The method of claim 1 , wherein determining whether there is the scene change comprises:
determining a first change metric using the at least one captured image and the previous captured image;
determining a second change metric using the at least one captured image and a second previous captured image from a previous trigger event;
generating a histogram change metric for the at least one captured image; and
using the first change metric, the second change metric and the histogram change metric to determine the scene change.

15. The method of claim 1 , wherein the information related to the image data comprises an object identification, the method further comprising:
acquiring additional captured images of the different object;
identifying the different object in the additional captured images using the object identification;
generating a tracking mask for the additional captured images based on the object identification, the tracking mask indicating regions in the additional captured images where the different object is identified;
using the tracking mask with the additional captured images of the different object to identify remaining regions of the additional captured images; and
detecting trigger events comprising scene changes in the remaining regions of the additional captured images.

16. The method of claim 1 , further comprising acquiring sensor data comprising at least one of motion sensor data, position data, barcode recognition, text detection results, or contextual information, and transmitting the sensor data with the image data to the server.

17. The method of claim 16 , wherein the contextual information includes one or more of the following: user behavior, user preferences, location, information about a user, time of day, and lighting quality.

18. The method of claim 1 , wherein the image data is from a plurality of images of the object captured with a camera at different positions, the method further comprising determining a coarse estimate of a pose of the camera with respect to the object and transmitting the coarse estimate of the pose with the image data, and the information received from the server comprises at least one of a refinement of the pose and a three-dimensional model of the object.

19. The method of claim 1 , wherein the image data is from a plurality of images of the object captured with a camera at different positions, and the information received from the server further comprises a pose of the object relative to the camera.

20. A mobile platform comprising:
a sensor that acquires image data, wherein the sensor is a camera and the image data is from at least one captured image of an object;
a wireless transceiver; and
a processor coupled to the sensor and the wireless transceiver, the processor acquires the image data via the sensor, tracks the object with visual based tracking using the at least one captured image of the object, determines whether there is a trigger event comprising a change in the image data relative to previously acquired image data, wherein the trigger event c

Assignees

• Qualcomm Inc

Inventors

• Swaminathan Ashwin
• Sharma Piyush
• Jiang Bolan
• Chari Murali R
• Diaz Spindola Serafin
• Baheti Pawan Kumar
• Narayanan Vidya

Classifications

• G06T7/246 Primary

  using feature-based methods, e.g. the tracking of corners or segments · CPC title

• G06T2207/10004

  Still image; Photographic image · CPC title

• G06T7/269

  using gradient-based methods · CPC title

• G06T7/292

  Multi-camera tracking · CPC title

• G06T7/73

  using feature-based methods · CPC title