Auxiliary device as augmented reality platform

Patent metadata

Field	Value
Publication number	US-10026229-B1
Application number	US-201615019257-A
Country	US
Kind code	B1
Filing date	Feb 9, 2016
Priority date	Feb 9, 2016
Publication date	Jul 17, 2018
Grant date	Jul 17, 2018

Abstract

An auxiliary device can be used to display a fiducial that contains information useful in determining the physical size of the fiducial as displayed on the auxiliary device. A primary device can capture image data including a representation of the fiducial. The scale and orientation of the fiducial can be determined, such that a graphical overlay can be generated of an item of interest that corresponds to that scale and orientation. The overlay can then be displayed along with the captured image data, in order to provide an augmented reality experience wherein the image displayed on the primary device represents a scale-appropriate view of the item in a location of interest corresponding to the location of the auxiliary device. As the primary device is moved and the viewpoint of the camera changes, changes in relative scale and orientation to the fiducial are determined and the overlay is updated accordingly.

First claim

What is claimed is:

1. A method, comprising:
determining that an auxiliary computing device, executing a specified application, is stationary against a flat surface for at least a minimum period of time based on a first image data captured by the auxiliary computing device, the auxiliary computing device determined stationary when the first image data include dark frames for at least the minimum period of time;
causing the auxiliary computing device to display a fiducial on a display screen, the fiducial including fiducial data relating to the auxiliary computing device;
capturing a second image data using a camera of a primary computing device, the second image data including a representation of a fiducial;
analyzing the second image data to locate the fiducial and extract the fiducial data;
determining, based at least in part upon the second image data, a relative orientation of the fiducial with respect to the primary computing device;
determining, based at least in part upon the fiducial data and a size of the representation in the second image data, a scale of the representation of the fiducial as displayed;
determining an item to be displayed with the second image data;
obtaining an overlay element for the item, the overlay element matching the relative orientation and the scale of the representation of the fiducial; and
displaying the overlay element with the second image data on the primary computing device.

2. The method of claim 1 , further comprising:
capturing additional image data using the camera of the primary computing device;
analyzing the additional image data to determine a current orientation and a current scale of the fiducial with respect to the primary computing device;
obtaining a second overlay element for the item, the second overlay element matching the current orientation and the current scale; and
displaying the second overlay element with the additional image data on the primary computing device.

3. The method of claim 1 , wherein obtaining an overlay element for the item further comprises:
obtaining a set of model data for the item;
determining a point of view based at least in part upon the relative orientation; and
rendering, based at least in part upon the set of model data, the overlay element from the point of view with the scale determined from the representation of the fiducial.

4. The method of claim 1 , wherein obtaining an overlay element for the item further comprises:
determining a set of images for the item, each of the set of images corresponding to a different point of view of the item;
selecting one of the set of images based at least in part upon the relative orientation; and
resizing the selected one image to match the scale determined from the representation of the fiducial.

5. A computer-implemented method, comprising:
capturing a first image data using a camera of a computing device;
analyzing the first image data to locate a representation of a fiducial in the first image data, the fiducial displayed on a display of an auxiliary device when the auxiliary device is determined stationary against a surface for a minimum period of time based on a second image data captured by the auxiliary device, the auxiliary device being determined stationary when the second image data include dark frames for at least the minimum period of time;
extracting fiducial data from the representation of the fiducial, the fiducial data providing identifying information for the auxiliary device;
determining, from fiducial data, a scale of the representation of the fiducial in the first image data; and
displaying a graphical element for an item, along with at least a portion of the first image data, on a display screen of the computing device, the graphical element sized to correspond to the scale of the representation of the fiducial.

6. The computer-implemented method of claim 5 , further comprising:
displaying the graphical element as an overlay over a portion of the first image data corresponding to a location of the representation of the fiducial in the first image data.

7. The computer-implemented method of claim 5 , further comprising:
receiving selection of the item; and
locating at least one of a set of images or model data for the item from an electronic catalog.

8. The computer-implemented method of claim 7 , further comprising:
determining a relative orientation of the fiducial based at least in part upon the representation of the fiducial in the first image data; and
rendering, using the model data, the graphical element from a point of view corresponding to the relative orientation.

9. The computer-implemented method of claim 7 , further comprising:
determining a relative orientation of the fiducial based at least in part upon the representation of the fiducial in the first image data; and
selecting the at least one of the set of images based at least in part upon the relative orientation.

10. The computer-implemented method of claim 5 , wherein the fiducial includes at least one of a specified pattern, a QR code, a bar code, a watermark, a data matrix, a logo, a periodic display, a text block, or a trackable image including a plurality of trackable features.

11. The computer-implemented method of claim 5 , further comprising:
capturing additional image data using the camera of the computing device;
determining, based on analyzing the additional image data, a change in relative orientation of the fiducial with respect to the computing device; and
updating the graphical element for the item to contain a view corresponding to a current relative orientation as a result of the change.

12. The computer-implemented method of claim 5 , wherein the item is at least one of a product, an animated character, or a virtual assistant.

13. The computer-implemented method of claim 5 , further comprising:
enabling multiple graphical elements for multiple items to be displayed on the display screen of the primary device with views corresponding to the relative orientation and the scale of the representation of the fiducial.

14. The computer-implemented method of claim 5 , further comprising:
receiving navigation input corresponding to the graphical element; and
causing a display location for the graphical element to change according to the navigation input.

15. The computer-implemented method of claim 5 , further comprising:
causing the first image data to be analyzed by a remote service in communication with the computing device.

16. A computing device, comprising:
at least one processor;
a display screen; and
memory storing instructions that, when executed by the at least one processor, cause the computing device to:
execute a specified application on the computing device;
determine, by the specified application, that the computing device is stationary against a surface for at least a minimum period of time based on image data captured by the computing device, the computing device being determined stationary when the image data include dark frames for at least the minimum period of time;
generate a fiducial element based at least in part upon identifying information for the computing device; and
display the fiducial element on the display screen, wherein a second device analyzing the fiducial element is enabled to determine a scale of the fiducial element based at least in part upon the identifying information for the computing device.

17. The computing device of claim 16 , wherein the instructions when executed further cause the computing device to:
select the fiducial element fr

Assignees

• A9 Com Inc

Inventors

• Yalniz Ismet Zeki
• Bhotika Rahul
• Cao Song
• Cutter Michael Patrick
• Taylor Colin Jon
• Waldo Mark Scott
• Wang Chun-Kai
• Zamalieva Daniya

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T7/74

  involving reference images or patches · CPC title

• G06T2207/30204

  Marker · CPC title

• G06T7/0044

  Physics · mapped topic

• G06K9/00671

  Physics · mapped topic