Augmenting a live view

Patent metadata

Field	Value
Publication number	US-9911239-B2
Application number	US-201715402016-A
Country	US
Kind code	B2
Filing date	Jan 9, 2017
Priority date	May 27, 2011
Publication date	Mar 6, 2018
Grant date	Mar 6, 2018

Abstract

Techniques for augmenting an image of an object captured and displayed in real time with associated content are disclosed. In one embodiment, the method for augmenting the image includes receiving information defining a sampled frame of a video being captured by an electronic device in substantially real time, determining information representative of an object captured in the sampled frame based on the received information, causing the determined information to match stored information defining a plurality of items to locate an item matched to the captured object, retrieving content associated with the matched item, and providing the retrieved content for display with the captured image on the electronic device. The retrieved content may be rendered in an overlay element that overlays the captured image displayed on the electronic device. The rendered content is configured to enable a user to interact with the content.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
capturing an image of an item using an image capturing element;
matching a feature represented in the image to stored features associated with items in an item repository to identify a matched item;
retrieving selectable content associated with the matched item;
determining a plurality of corners of the item using feature extraction to extract interest points from the image that correspond to the plurality of corners;
determining a bounding box based at least in part on the plurality of corners;
determining, for the bounding box, a perspective distortion that satisfies a predetermined distortion threshold; and
displaying, on a display element, the selectable content as an overlay over the bounding box, wherein the matched item is presented for purchase in response to a selection of the selectable content.

2. The computer-implemented method of claim 1 , wherein the overlay augments the image by overlaying the image, wherein the overlay is selected from one of: a box, a button, a three-dimensional (3D) structure, an animation, audio, video, web page, or interactive user interface.

3. The computer-implemented method of claim 1 , wherein the image capturing element is selected from one of a video camera or a photo camera.

4. The computer-implemented method of claim 1 , wherein the selectable content includes at least one of: a button or a hyperlink.

5. The computer-implemented method of claim 1 , wherein the selectable content includes item information associated with the item and comprising at least one of: price, description, customer rating, customer review, special offers and promotions, or associated information related to associated merchandise.

6. The computer-implemented method of claim 5 , wherein the item information is matched to stored information in a local repository associated with the image capturing element, and wherein the stored information defines the items in an item repository.

7. The computer-implemented method of claim 5 , wherein the item information is matched to stored information remote from an electronic device hosting the image capturing element, and wherein the stored information defines the items in an item repository.

8. The computer-implemented method of claim 1 , further comprising:
determining additional matched items to the item in response to multiple matches being found between the item and the items in an item repository.

9. The computer-implemented method of claim 1 , further comprising: tagging the matched item with at least one of a current location identifier or a timestamp.

10. The computer-implemented method of claim 1 , further comprising snapping a portion of the bounding box to a nearest sharp edge of a representation of the item.

11. A computer system, comprising:
a processor; and
a memory having computer-executable instructions that, when executed on the processor, cause the processor to:
capture an image of an item using an image capturing element;
match a feature represented in the image to stored features associated with items in an item repository to identify a matched item;
retrieve selectable content associated with the matched item;
determine a plurality of corners of the item using feature extraction to extract interest points from the image that correspond to the plurality of corners;
determine a bounding box based at least in part on the plurality of corners;
determine, for the bounding box, a perspective distortion that satisfies a predetermined distortion threshold; and
display, on a display element the selectable content as an overlay over the bounding box, wherein the matched item is presented for purchase in response to a selection of the selectable content.

12. The computer system of claim 11 , the selectable content including at least one of: a button or a hyperlink.

13. The computer system of claim 11 , wherein the selectable content includes item information associated with the item and comprising at least one of: price, description, customer rating, customer review, special offers and promotions, or associated information related to associated merchandise.

14. The computer system of claim 11 , wherein the computer-executable instructions further cause the processor to tag the matched item with at least one of a current location identifier or a timestamp.

15. The computer system of claim 11 , wherein the computer-executable instructions further cause the processor to initiate the purchase of the matched item upon receiving from the user a single interaction with the selectable content, the user having previously submitted account information for the purchase.

16. A non-transitory computer-readable storage medium having computer-executable instructions that, when executed by a computer, cause the computer to:
receive a sampled frame of a video being captured by an electronic device;
determine information representative of an item captured in the sampled frame;
match a feature from the information to stored features associated with items in an item repository to identify a matched item;
retrieve selectable content associated with the matched item;
determine a plurality of corners of the item using feature extraction to extract interest points from the image that correspond to the plurality of corners;
determine a bounding box based at least in part on the plurality of corners;
determine, for the bounding box, a perspective distortion that satisfies a predetermined distortion threshold; and
display, on a display element, the selectable content as overlay over the bounding box, wherein the matched item is presented for purchase in response to a selection of the selectable content.

17. The non-transitory computer-readable storage medium of claim 16 , wherein the selectable content includes at least one of: a button or a hyperlink.

18. The non-transitory computer-readable storage medium of claim 16 , wherein the selectable content includes item information associated with the item and comprising at least one of: price, description, customer rating, customer review, special offers and promotions, and associated information related to associated merchandise.

19. The non-transitory computer-readable storage medium of claim 16 , wherein the overlay is a box.

20. The non-transitory computer-readable storage medium of claim 16 , wherein the computer-executable instructions further cause the computer to initiate purchase of the item upon receiving from the user a single interaction with the selectable content, the user having previously submitted account information.

Assignees

• A9 Com Inc

Inventors

• Ramkumar Gurumurthy D
• Stasior William F
• Feldman Bryan E
• Dhua Arnab S
• Senthamil Nalin Pradeep

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06Q30/0643 Primary

  graphically representing goods, e.g. 3D product representation · CPC title

• G06F16/434

  using image data, e.g. images, photos, pictures taken by a user · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06T19/00

  Manipulating three-dimensional [3D] models or images for computer graphics · CPC title