Determining networked mobile device position and orientation for augmented-reality window shopping

Patent metadata

Field	Value
Publication number	US-9367870-B2
Application number	US-201213646590-A
Country	US
Kind code	B2
Filing date	Oct 5, 2012
Priority date	Oct 5, 2012
Publication date	Jun 14, 2016
Grant date	Jun 14, 2016

Abstract

A consumer user can perform a number of augmented reality shopping operations using a local client shopping application executable on a mobile device. Features related to implementations of the current subject matter can include, but are not limited to, the ability to view an augmented reality depiction of a shopping display using the mobile device and to execute various “e-commerce” functions accessible via the augmented reality depiction. Visual image processing based on one or more inputs from sensors on the mobile device can be used to compute and efficiently provide the two dimensional projection of the three dimensional model to be presented on the screen of the mobile device. Related methods, systems, articles of manufacture, and the like are described.

First claim

What is claimed is:

1. A computer program product comprising a non-transitory machine-readable medium storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations comprising:
receiving, at a processor from a mobile device executing a local client shopping application, sensor data generated by one or more sensors associated with the mobile device;
narrowing, at the processor using the sensor data generated by the one or more sensors, a number of available degrees of freedom relating to a current view of a shopping display being sensed by a camera of the mobile device;
estimating, at the processor based on the narrowed available degrees of freedom and using a computer vision algorithm, a pose and position of the camera relative to a three dimensional model of a shopping display;
computing, at the processor based on the pose and position of the camera, a two dimensional projection of the three dimensional model;
augmenting, at the processor based on the two dimensional projection, a two dimensional picture comprising an augmented reality marker associated with a physical item in the shopping display; and
transmitting the two dimensional picture to a mobile device for presentation on a screen of the mobile device.

2. A computer program product as in claim 1 , wherein the one or more sensors comprise a sensor selected from a group consisting of a radio signal triangulation sensor, a global positioning transceiver, a compass sensor, and an accelerometer sensor.

3. A computer program product as in claim 1 , wherein the operations further comprise:
identifying, based on the sensor data provided by the one or more sensors, a most likely shopping display being viewed through a camera associated with the mobile device; and
determining the three dimensional model of the shopping display.

4. A computer program product as in claim 1 , wherein the operations further comprise: identifying the shopping display based on a visual or textual encoding of a location code entered into the mobile device via the camera or a textual interface on the mobile device.

5. A computer program product as in claim 1 , wherein the operations further comprise: presenting, via the screen, the two dimensional picture.

6. A computer program product as in claim 1 , wherein the operations further comprise:
receiving, via a user input, a selection of the augmented reality marker; and
presenting, on the screen of the mobile device in response to the received selection, an item page corresponding to the physical item, the item page comprising additional information about at least one of the physical item and a good or service represented by the physical item.

7. A system comprising:
computer hardware configured to perform operations comprising:
receiving, at a processor from a mobile device executing a local client shopping application, sensor data generated by one or more sensors associated with the mobile device;
narrowing, at the processor using the sensor data generated by the one or more sensors, a number of available degrees of freedom relating to a current view of a shopping display being sensed by a camera of the mobile device;
estimating, at the processor based on the narrowed available degrees of freedom and using a computer vision algorithm, a pose and position of the camera relative to a three dimensional model of a shopping display;
computing, at the processor based on the pose and position of the camera, a two dimensional projection of the three dimensional model;
augmenting, at the processor based on the two dimensional projection, a two dimensional picture comprising an augmented reality marker associated with a physical item in the shopping display; and
transmitting the two dimensional picture to a mobile device for presentation on a screen of the mobile device.

8. A system as in claim 7 , wherein the one or more sensors comprise a sensor selected from a group consisting of a radio signal triangulation sensor, a global positioning transceiver, a compass sensor, and an accelerometer sensor.

9. A system as in claim 7 , wherein the operations further comprise:
identifying, based on the sensor data provided by the one or more sensors, a most likely shopping display being viewed through a camera associated with the mobile device; and
determining the three dimensional model of the shopping display.

10. A system as in claim 7 , wherein the operations further comprise:
identifying the shopping display based on a visual or textual encoding of a location code entered into the mobile device via the camera or a textual interface on the mobile device.

11. A system as in claim 7 , wherein the operations further comprise: presenting, via the screen, the two dimensional picture.

12. A computer program product as in claim 7 , wherein the operations further comprise:
receiving, via a user input, a selection of the augmented reality marker; and
presenting, on the screen of the mobile device in response to the received selection, an item page corresponding to the physical item, the item page comprising additional information about at least one of the physical item and a good or service represented by the physical item.

13. A computer-implemented method comprising:
receiving, at a processor from a mobile device executing a local client shopping application, sensor data generated by one or more sensors associated with the mobile device;
narrowing, at the processor using the sensor data generated by the one or more sensors, a number of available degrees of freedom relating to a current view of a shopping display being sensed by a camera of the mobile device;
estimating, at the processor based on the narrowed available degrees of freedom and using a computer vision algorithm, a pose and position of the camera relative to a three dimensional model of a shopping display;
computing, at the processor based on the pose and position of the camera, a two dimensional projection of the three dimensional model;
augmenting, at the processor based on the two dimensional projection, a two dimensional picture comprising an augmented reality marker associated with a physical item in the shopping display; and
transmitting the two dimensional picture to a mobile device for presentation on a screen of the mobile device.

14. A computer-implemented method as in claim 13 , wherein the one or more sensors comprise a sensor selected from a group consisting of a radio signal triangulation sensor, a global positioning transceiver, a compass sensor, and an accelerometer sensor.

15. A computer-implemented method as in claim 13 , further comprising:
identifying, based on the sensor data provided by the one or more sensors, a most likely shopping display being viewed through a camera associated with the mobile device; and
determining the three dimensional model of the shopping display.

16. A computer-implemented method as in claim 13 , further comprising:
identifying the shopping display based on a visual or textual encoding of a location code entered into the mobile device via the camera or a textual interface on the mobile device.

17. A computer-implemented method as in claim 13 , wherein the operations further comprise: presenting, via the screen, the two dimensional picture.

18. A computer-implemented method as in claim 13 , further comprising:
receiving, via a user input, a selection of the augmented reality marker; and
presenting, on the screen of the mobile device in response to the received selecti

Assignees

• Klein Udo
• Mayerle Jochen
• Hoyer Timo
• Tatzel Steffen
• Koshy Sanu
• Sap Se

Inventors

• Klein Udo
• Mayerle Jochen
• Hoyer Timo
• Tatzel Steffen
• Koshy Sanu

Classifications

• G06T1/00

  General purpose image data processing · CPC title

• G06Q30/0643 Primary

  graphically representing goods, e.g. 3D product representation · CPC title