Foreground identification for virtual objects in an augmented reality environment

Patent metadata

Field	Value
Publication number	US-10147237-B2
Application number	US-201615271418-A
Country	US
Kind code	B2
Filing date	Sep 21, 2016
Priority date	Sep 21, 2016
Publication date	Dec 4, 2018
Grant date	Dec 4, 2018

Abstract

Systems described herein apply visual computer-generated elements into real-world images with an appearance of depth by using information available via conventional mobile devices. The systems receive a reference image and reference image data collected contemporaneously with the reference image. The reference image data includes a geo-location, a direction heading, and a tilt. The systems identify one or more features within the reference image and receive a user's selection of a foreground feature from the one or more features. The systems receive a virtual object definition that includes an object type, a size, and an overlay position of the virtual object relative to the foreground feature. The virtual object is provided in the virtual layer appearing behind the foreground feature. The systems store, in a memory, the reference image data associated with the virtual object definition.

First claim

What is claimed is:

1. A method performed by a computing device, the method comprising:
receiving a reference image and reference image data collected contemporaneously with the reference image, wherein the reference image data includes a geo-location, a direction heading, and a tilt;
identifying, for a user of the computing device, one or more features that are available for selection as foreground features within the reference image;
receiving a user's selection of a foreground feature within the reference image from the one or more features;
receiving a user's selection of a virtual object from a catalog of virtual objects;
receiving a user's size and placement information for the virtual object within the reference image where the virtual object and the foreground feature overlap to make the foreground feature appear in front of the virtual object;
compiling a virtual object definition that includes the selected virtual object, the size, and the placement information of the virtual object relative to the foreground feature, wherein the virtual object is provided in a virtual layer appearing behind the foreground feature within the reference image; and
storing, in a memory, the reference image data and virtual object definition.

2. The method of claim 1 , further comprising:
providing, to a mobile device, an augmented reality application that includes a user interface to enable a user to select the foreground feature and provide the virtual object definition.

3. The method of claim 1 , further comprising:
receiving, from a mobile device, an indication that the mobile device is in a reference position that corresponds to the reference image data;
receiving, from the mobile device, a video feed while the mobile device is in the reference position;
detecting the foreground feature in an image from the video feed; and
instructing the mobile device to present the virtual object in the virtual layer appearing behind the foreground feature in images from the video feed, when the foreground feature is detected.

4. The method of claim 3 , wherein, when instructing the mobile device to present the virtual object, the virtual object is rendered so as to appear at least partially occluded by the foreground feature.

5. The method of claim 3 , further comprising:
providing, to the mobile device, an augmented reality application that includes a user interface to enable the mobile device to obtain the reference image data and the virtual object definition.

6. The method of claim 1 , wherein the virtual object is a two-dimensional representation of a three-dimensional object.

7. The method of claim 1 , wherein identifying the one or more features within the reference image includes:
receiving, via a user interface on a mobile device, user input to trace an outline of the one or more features.

8. The method of claim 1 , wherein identifying the one or more features within the reference image includes:
automatically detecting the one or more features from a library of known features.

9. The method of claim 1 , wherein receiving the user's selection of the foreground feature within the reference image from the one or more features includes:
indicating, via a user interface on a mobile device, at least two features of the one or more features; and
soliciting, via the user interface, user input to identify the foreground feature from the at least two features.

10. The method of claim 1 , wherein identifying the one or more features within the reference image comprises:
detecting the one or more features from a simplified edge model extracted from the reference image.

11. The method of claim 1 , wherein the mobile device includes a single camera facing an opposite direction to a display screen, and wherein the computing device includes a server device for a service provider network.

12. A device comprising:
one or more memories to store instructions; and
one or more processors to execute the instructions to:
receive a reference image and reference image data collected contemporaneously with the reference image, wherein the reference image data includes a geo-location, a direction heading, and a tilt;
identify, for a user of the device, one or more features within the reference image that are available for selection as foreground features;
receive a user's selection of a foreground feature within the reference image from the one or more features;
provide, via a user interface, a catalog of virtual objects;
receive a user's selection of a virtual object from the catalog;
receive a user's size and placement information for the virtual object within the reference image where the virtual object and the foreground feature overlap to make the foreground feature appear in front of the virtual object;
compile a virtual object definition that includes the selected virtual object, the size, and the placement information of the virtual object relative to the foreground feature, wherein the virtual object is provided in a virtual layer appearing behind the foreground feature within the reference image; and
store, in a memory, the reference image data and virtual object definition.

13. The device of claim 12 , wherein the one or more processors are further to execute the instructions to:
receive, from another device, an indication that the other device is in a reference position that corresponds to the reference image data;
receive, from the other device, a video feed while the other device is in the reference position;
detect the foreground feature in an image from the video feed; and
instruct the other device to present the virtual object in the virtual layer appearing behind the foreground feature in images of the video feed, when the foreground feature is detected.

14. The device of claim 13 , wherein, when instructing the other device to present the virtual object, the one or more processors are further to execute the instructions to:
render the virtual object to appear at least partially occluded by the foreground feature.

15. The device of claim 13 , wherein the one or more processors are further to execute the instructions to:
provide, to the other device, an augmented reality application that includes a user interface to enable the other device to obtain the reference image data and the virtual object definition.

16. The device of claim 12 , wherein, when receiving the user's selection of the foreground feature within the reference image from the one or more features, the one or more processors are further to execute the instructions to:
indicate, via a user interface on the device, at least two features of the one or more features; and
solicit, via the user interface, user input to identify the foreground feature from the at least two features.

17. The device of claim 12 , wherein, when identifying the one or more features within the reference image, the one or more processors are further to execute the instructions to:
receive user input, via a user interface on the device, to trace an outline of the one or more features.

18. A non-transitory computer-readable medium containing instructions executable by at least one processor, the computer-readable medium comprising one or more instructions to:
receive a reference image and reference image data collected contemporaneously with the reference image, wherein the reference image data includes a geo-location, a direction heading, and a tilt;
identify, for a user of the computing device, one or more features within the reference image that are available for selection as fore

Assignees

• Verizon Patent & Licensing Inc

Inventors

• Sharma Manish
• Prakash Anand
• Blong Devin
• Zhang Qing
• Nagaraj Chetan Mugur
• Rangarajan Srivatsan

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F11/3058

  Monitoring arrangements for monitoring environmental properties or parameters of the computing system or of the computing system component, e.g. monitoring of power, currents, temperature, humidity, position, vibrations (thermal management in cooling arrangements of a computing system G06F1/206) · CPC title

• G06T2200/04

  involving 3D image data · CPC title

• G06F3/04842

  Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) · CPC title

• G06T2219/024

  Multi-user, collaborative environment · CPC title