Augmented-reality shopping using a networked mobile device

Patent metadata

Field	Value
Publication number	US-9449343-B2
Application number	US-201213646606-A
Country	US
Kind code	B2
Filing date	Oct 5, 2012
Priority date	Oct 5, 2012
Publication date	Sep 20, 2016
Grant date	Sep 20, 2016

Abstract

A consumer user can perform a number of augmented reality shopping operations using a local client shopping application executable on a mobile device. Features related to implementations of the current subject matter can include, but are not limited to, the ability to view an augmented reality depiction of a shopping display using the mobile device and to execute various “e-commerce” functions accessible via the augmented reality depiction. Related methods, systems, articles of manufacture, and the like are described.

First claim

What is claimed:

1. A computer program product comprising a non-transitory machine-readable medium storing instructions that, when executed by a mobile device comprising at least one programmable processor, cause the mobile device to perform operations comprising:
scanning a quick response code from a shopping display, the quick response code identifying an augmented three dimensional model of the shopping display from a plurality of three dimensional models of shopping displays available to a backend server and indicating that the mobile device is proximate to the shopping display;
providing location information comprising a direction, an angle, and a vertical height of the mobile device when a camera of the mobile device is directed at the shopping display;
capturing a two dimensional image of the shopping display for mapping with the augmented reality three dimensional model of the shopping display at which the mobile device is directed;
determining, based on the provided location information and captured two dimensional image of the shopping display, a two dimensional overlay comprising an augmented reality marker corresponding to a physical item visible in the captured two dimensional image of the shopping display, the determining comprising use of an augmented reality three dimensional model captured using a three dimensional camera and stored as part of an augmented reality shopping program, the determining further comprising creating the two dimensional overlay to represent a view of the augmented reality three dimensional model based on the provided location information;
displaying, via a screen on the mobile device, an augmented reality view of the shopping display, the augmented reality view comprising the two dimensional image and the two dimensional overlay presented together such that the augmented reality marker is associated with the physical item in the augmented reality view;
receiving, via a user input on the screen, a selection of the augmented reality marker; and
presenting, on the screen of the mobile device in response to the received selection, an item page corresponding to the physical item, the item page comprising additional information about at least one of the physical item and a good or service represented by the physical item.

2. A computer program product as in claim 1 , wherein the operations further comprise: downloading, from a server to the mobile device, the augmented reality three dimensional model of the shopping display.

3. A computer program product as in claim 2 , wherein the operations further comprise: transmitting information from the mobile device to the server, the information being usable by the server to identify the augmented reality three dimensional model of the shopping display from the plurality of augmented reality three dimensional models of shopping displays stored at or accessible by the server.

4. A computer program product as in claim 1 , wherein the determining, the displaying, the receiving, and the presenting are performed by a local client shopping application running on the mobile device.

5. A computer program product as in claim 1 , wherein the additional information comprises at least one of identification information relating to the physical item or the good or service represented by the physical item, sizing information, user interface functionality that permits selection by the consumer user of a specific size or color or shape, user interface functionality for adding the physical item or the good or service represented by the physical item to a shopping cart or watch list, user interface functionality for immediately purchasing the physical item or the good or service represented by the physical item, user interface functionality that supports sending information about the physical item or the good or service represented by the physical item to another person, user interface functionality that supports sending a request for additional information about the physical item or the good or service represented by the physical item, user interface functionality that supports providing additional views of the physical item or the good or service represented by the physical item, and user interface functionality that supports viewing of related goods or services.

6. A computer program product as in claim 1 , wherein the two dimensional overlay further comprises a second augmented reality marker comprising an image representation of a good or service, the image representation not being physically present in the shopping display.

7. A computer program product as in claim 1 , wherein the augmented reality three dimensional model comprises a captured image with an item identified by the retail location for selection and viewing with the augmented reality shopping program, the identified item being associated with the augmented reality marker.

8. A system comprising:
a mobile device comprising at least one programmable processor; and
a machine-readable medium storing instructions that, when executed by the at least one processor, cause the at least one programmable processor to perform operations comprising:
scanning a quick response code from a shopping display, the quick response code identifying an augmented three dimensional model of the shopping display from a plurality of three dimensional models of shopping displays available to a backend server and indicating that the mobile device is proximate to the shopping display;
providing location information comprising a direction, an angle, and a vertical height of the mobile device when a camera of the mobile device is directed at the shopping display;
capturing a two dimensional image of the shopping display for mapping with the augmented reality three dimensional model of the shopping display at which the mobile device is directed;
determining, based on the provided location information and captured two dimensional image of the shopping display, a two dimensional overlay comprising an augmented reality marker corresponding to a physical item visible in the captured two dimensional image of the shopping display, the determining comprising use of an augmented reality three dimensional model captured using a three dimensional camera and stored as part of an augmented reality shopping program, the determining further comprising creating the two dimensional overlay to represent a view of the augmented reality three dimensional model based on the provided location information;
displaying, via a screen on the mobile device, an augmented reality view of the shopping display, the augmented reality view comprising the two dimensional image and the two dimensional overlay presented together such that the augmented reality marker is associated with the physical item in the augmented reality view;
receiving, via a user input on the screen, a selection of the augmented reality marker; and
presenting, on the screen of the mobile device in response to the received selection, an item page corresponding to the physical item, the item page comprising additional information about at least one of the physical item and a good or service represented by the physical item.

9. A system as in claim 8 , wherein the operations further comprise: downloading, from a server to the mobile device, the augmented reality three dimensional model of the shopping display.

10. A system as in claim 9 , wherein the operations further comprise: transmitting information from the mobile device to the server, the information being usable by the server to identify the augmented reality three dimensional model of the shopping display from the plurality of augmented reality three dimensional models of shopping displays stored at or accessible by the server

Assignees

• Mayerle Jochen
• Klein Udo
• Jacob Clemens
• Koshy Sanu
• Hoyer Timo
• Tatzel Steffen
• Sap Se

Inventors

• Mayerle Jochen
• Klein Udo
• Jacob Clemens
• Koshy Sanu
• Hoyer Timo
• Tatzel Steffen

Classifications

• G06T11/00

  Two-dimensional [2D] image generation · CPC title

• G06Q30/0276

  Advertisement creation · CPC title

• G06Q30/0643 Primary

  graphically representing goods, e.g. 3D product representation · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06Q30/0268

  at point-of-sale [POS] · CPC title