Augmenting a live view

Patent metadata

Field	Value
Publication number	US-9547938-B2
Application number	US-201113117527-A
Country	US
Kind code	B2
Filing date	May 27, 2011
Priority date	May 27, 2011
Publication date	Jan 17, 2017
Grant date	Jan 17, 2017

Abstract

Techniques for augmenting an image of an object captured and displayed in real time with associated content are disclosed. In one embodiment, the method for augmenting the image includes receiving information defining a sampled frame of a video being captured by an electronic device in substantially real time, determining information representative of an object captured in the sampled frame based on the received information, causing the determined information to match stored information defining a plurality of items to locate an item matched to the captured object, retrieving content associated with the matched item, and providing the retrieved content for display with the captured image on the electronic device. The retrieved content may be rendered in an overlay element that overlays the captured image displayed on the electronic device. The rendered content is configured to enable a user to interact with the content.

First claim

What is claimed is:

1. A computer-implemented method for augmenting an image or a stream of images of a product captured and displayed with associated content, comprising: capturing at least one image in a stream of images of the product using an image capturing element of an electronic device operated by a user to form a captured at least one image; displaying the captured at least one image on a display element of the electronic device; determining one or more features defining a representation of the product in the captured at least one image; determining that the one or more features includes at least one of barcode data or text; matching the one or more features to features defining a plurality of items stored in an item repository; determining a match to identify a matched item; retrieving content associated with the matched item from the item repository; determining a plurality of corners of the product, the determining for the plurality of corners by extracting interest points on the at least one image and performing a feature extraction from the interest points; determining a bounding box that includes the representation of the product based on the plurality of corners; determining a perspective distortion associated with the bounding box; determining that the perspective distortion at least does not exceed a predetermined distortion threshold; and displaying the retrieved content in an overlay element on the display element of the electronic device within the bounding box such that the overlay element augments the image of the product displayed on the display element of the electronic device, the retrieved content representing selectable content enabling the user to provide an indication to purchase the product being augmented by the overlay element.

2. The computer-implemented method of claim 1 , wherein the overlay element augments the image by overlaying the image, wherein the overlay element is selected from one of: a box, a button, a three-dimensional (3D) structure, an animation, audio, video, Web page, or interactive user interface.

3. The computer-implemented method of claim 1 , wherein image capturing element of the electronic device is selected from one of a video camera or a photo camera.

4. The computer-implemented method of claim 1 , further comprising snapping a portion of the bounding box to a nearest sharp edge of the representation of the product.

5. A computer-implemented method for augmenting an image of a product captured and displayed with associated content, comprising: receiving information defining a sampled frame of a video being captured by an electronic device to form a captured at least one image; determining information representative of the product captured in the captured at least one image based at least in part on the received information; determining that the information representative of the product includes at least one of barcode data or text; causing the information to match stored information defining a plurality of items, the stored information stored in an item repository, to locate an item matched to the captured product; determining a match to identify a matched item; determining a plurality of corners of the product, the determining for the plurality of corners by extracting interest points on the at least one image and performing a feature extraction from the interest points; determining a bounding box that includes the information representative of the product based on the plurality of corners; determining a perspective distortion associated with the bounding box; determining that the perspective distortion at least does not exceed a predetermined distortion threshold; retrieving content associated with the matched item from the item repository; and providing the retrieved content for display with the captured at least one image on the electronic device, wherein the retrieved content is rendered in an overlay element that overlays the captured at least one image within the bounding box displayed on the electronic device, the retrieved content representing selectable content enabling a user to provide an indication to purchase the product being augmented by the overlay element.

6. The computer-implemented method of claim 5 , wherein the selectable content includes at least one of: a button or a hyperlink.

7. The computer-implemented method of claim 5 , wherein the selectable content includes information associated with the product comprising at least one of a product price, description of the product, customer rating of the product, customer review of the product, special offers and promotions related to the product, or information related to other merchandise related to the product.

8. The computer-implemented method of claim 5 , wherein the determined information is matched to stored information at the electronic device and wherein information defining a plurality of items is stored in a local repository associated with the electronic device.

9. The computer-implemented method of claim 5 , wherein the determined information is matched to stored information remote from the electronic device and wherein information defining a plurality of items is stored in a remote repository.

10. The computer-implemented method of claim 5 , further comprising:
determining a number of matched items to each product in response to multiple matches being found between one or more products and stored items.

11. The computer-implemented method of claim 5 , further comprising: tagging the matched item with at least one of a current location identifier or a timestamp.

12. A computer-implemented method for augmenting an image of a product captured and displayed in real time with associated content, comprising: capturing at least one image of the product with an electronic device to form a captured at least one image, the captured at least one image being displayed by the electronic device; determining information representative of the product based at least in part on the captured at least one image of the product; determining whether the information representative of the product includes at least one of barcode data or text; matching the information to stored information defining each item of a plurality of items, the stored information stored in an item repository; determining a match to identify a matched item; determining a plurality of corners of the product, the determining for the plurality of corners by extracting interest points on the at least one image and performing a feature extraction from the interest points; determining a bounding box that includes the information representative of the product based on the plurality of corners; determining a perspective distortion associated with the bounding box; determining that the perspective distortion at least does not exceed a predetermined distortion threshold; retrieving content associated with the matched item from the item repository; and overlaying the captured image with the retrieved content rendered in an overlay element within the bounding box, the retrieved content representing selectable content enabling a user to provide an indication to purchase the product being augmented by the overlay element.

13. The computer-implemented method of claim 12 , further comprising: enabling the user to enter a second overlay element and to tag the product with additional content pertaining to the product within the second overlay element.

14. The computer-implemented method of claim 12 , further comprising: in response to the user entering a search query in the overlay element, retrieving search results corresponding to the search query.

15. The computer-implement

Assignees

• Ramkumar Gurumurthy D
• Stasior William F
• Feldman Bryan E
• Dhua Arnab S
• Senthamil Nalin Pradeep
• A9 Com Inc

Inventors

• Ramkumar Gurumurthy D
• Stasior William F
• Feldman Bryan E
• Dhua Arnab S
• Senthamil Nalin Pradeep

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06Q30/0643 Primary

  graphically representing goods, e.g. 3D product representation · CPC title

• G06T19/00

  Manipulating three-dimensional [3D] models or images for computer graphics · CPC title

• G06F16/434

  using image data, e.g. images, photos, pictures taken by a user · CPC title

• G06F16/24575

  using context · CPC title