Augmented reality in-store product detection system

Patent metadata

Field	Value
Publication number	US-10275825-B2
Application number	US-201213693126-A
Country	US
Kind code	B2
Filing date	Dec 4, 2012
Priority date	Dec 4, 2012
Publication date	Apr 30, 2019
Grant date	Apr 30, 2019

Abstract

A method and system create, on a portable electronic device, a real-time graphical view of an in-store product display that pinpoints the location of one or more target products on the product display. The graphical view is generated by capturing merchandise display data, such as a video feed of the in-store product display, and processing the data to detect the one or more target products based on predefined product recognition criteria uniquely associated with the respective target products.

First claim

What is claimed is:

1. A method comprising:
receiving target definition data indicating a shopping list comprising a plurality of target products;
receiving an in-store route that passes at least one of the plurality of target products;
receiving merchandise display data captured by a portable electronic device at a store based on the portable electronic device traversing the in-store route, the merchandise display data indicating physical presence of multiple products on an in-store product display at the store, the merchandise display data including video content of the in-store product display at the store and wireless identifier signals from the multiple products, the wireless identifier signals including location data for at least one of the multiple products;
in an automated operation that is based at least in part on the merchandise display data and that is performed using one or more processor devices configured to perform the automated operation, detecting among the multiple products on the in-store product display a target product forming part of the plurality of target products, the target product detected based on a particular wireless identifier signal of the wireless identifier signals being a unique identifier signal of the target product;
displaying, substantially in real time based on a graphical view of at least a part of the in-store product display being captured by the portable electronic device, an augmented reality video image based on the graphical view, the graphical view including a depiction of a store area where the target product is located;
determining an on-screen position, among the multiple products, of the target product within the graphical view based on particular location data of the particular wireless identifier signal without receiving user input identifying the on-screen position of the target product and while continuing to receive the merchandise display data; and
providing as a part of the graphical view a location indicator that indicates the determined on-screen position, among the multiple products, of the automatically detected target product substantially in real time and in the augmented reality video image.

2. The method of claim 1 , wherein the detecting of the target product comprises processing the merchandise display data to identify one or more product recognition criteria for the target product in the merchandise display data.

3. The method of claim 2 , further comprising capturing image information by use of an onboard camera of the portable electronic device to provide the merchandise display data, the image information representing an image of at least part of the in-store product display and being included in the video content.

4. The method of claim 3 , wherein the detecting of the target product comprises processing the image information to recognize a visual indicator that is unique to the target product.

5. The method of claim 4 , wherein the visual indicator comprises one or more trade dress elements of a packaging of the target product.

6. The method of claim 4 , wherein the visual indicator comprises a visual product code carried on the target product.

7. The method of claim 3 , wherein the location indicator is displayed continuously while the automatically detected target product is visible in the video image.

8. The method of claim 3 , further comprising receiving at the portable electronic device the one or more product recognition criteria that define one or more unique product attributes of the target product.

9. The method of claim 8 , further comprising transmitting to a store computer system a search query for the target product, the product recognition criteria being received by the portable electronic device from the store computer system responsive to the search query.

10. The method of claim 9 , wherein the search query comprises a search image, the method further comprising capturing the search image by use of the onboard camera of the portable electronic device.

11. The method of claim 10 , wherein the search query comprises a visual product code unique to the target product, the method further comprising scanning an instance of the visual product code by use of the onboard camera of the portable electronic device.

12. The method of claim 2 , wherein the processing of the merchandise display data is to identify any of the plurality of target products in the shopping list, and wherein the method further comprises providing as part of the graphical view a respective location indicator for each of the plurality of target products that are identified.

13. The method of claim 12 , further comprising receiving the one or more product recognition criteria for the shopping list from a home pantry system of a user of the portable electronic device.

14. The method of claim 13 , further comprising:
monitoring stock levels of products in the home pantry system;
determining that a stock level for a particular product in the home pantry system is below a threshold level;
responsive to the determination, automatically adding the particular product to the shopping list; and
communicating the shopping list to the portable electronic device from the home pantry system by transmitting the one or more product recognition criteria for the respective target products forming part of the shopping list.

15. A system comprising:
a receiver module to receive:
target definition data indicating a shopping list comprising a plurality of target products;
an in-store route that passes at least one of the target products; and
merchandise display data captured by a portable electronic device at a store along the in-store route, the merchandise display data indicating physical presence of multiple products on an in-store product display at the store, the merchandise display data including video content of the in-store product display at the store and wireless identifier signals from the multiple products, the wireless identifier signals including location data for at least one of the multiple products;

a product locator module comprising one or more computer devices configured to automatically detect, based at least in part on the merchandise display data, among the multiple products on the in-store product display a target product forming part of the plurality of target products, the target product detected based on a particular wireless identifier signal of the wireless identifier signals being a unique identifier signal of the target product;
a display module to display, substantially in real time based on a graphical view of at least a part of the in-store product display being captured by the portable electronic device, an augmented reality video image based on the graphical view, the graphical view including a depiction of a store area where the target product is located; and
an indicator module comprising at least one computer processor device configured to:
determine an on-screen position, among the multiple products, of the target product within the graphical view based on particular location data of the particular wireless identifier signal without the system having received any user input that identifies the on-screen position of the target product and while the receiver module continues to receive the merchandise display data;
provide in the graphical view a location indicator that indicates the determined on-screen position, among the multiple products, of the automatically detected target product substantially in real time and in the augmented reality video image.

16. The system of claim 15 , wherein the product locator m

Assignees

• Paypal Inc

Inventors

• Zises Matthew Scott

Classifications

• G06Q30/0643 Primary

  graphically representing goods, e.g. 3D product representation · CPC title

• G06Q30/0641

  utilising user interfaces specially adapted for shopping · CPC title

• G06Q30/0623

  by investigating goods or services · CPC title