Data manipulation based on real world object manipulation

Patent metadata

Field	Value
Publication number	US-9639984-B2
Application number	US-201313909046-A
Country	US
Kind code	B2
Filing date	Jun 3, 2013
Priority date	Jun 3, 2013
Publication date	May 2, 2017
Grant date	May 2, 2017

Abstract

A system and method for data manipulation based on real world object manipulation is described. A device captures an image of a physical object. The image is communicated via a network to a remote server. The remote server includes virtual object data associated with the image and a communication notification for a user of the computing device. The device receives the virtual object data and displays the virtual image in a virtual landscape using the virtual object data. In response to relative movement between the computing device and the physical object caused by the user, the virtual image is modified.

First claim

What is claimed is:

1. A method comprising:
receiving, by a computing device, a primary content dataset from a remote server, the primary content dataset including a subset of known visual references stored by the remote server that were determined to be most popular by the remote server, wherein each known visual reference corresponds to a virtual object model;
storing, by the computing device, the primary content dataset in a local memory of the computing device;
after receiving the primary content dataset, capturing, by an optical sensor of the computing device, an image of a physical object, the image including a captured visual reference associated with the physical object;
searching, by the computing device, the primary content dataset stored in the local memory of the computing device based on the captured visual reference to identify a virtual object model corresponding to the captured visual reference associated with the physical object;
in response to determining that the captured visual reference is not included in the primary content dataset in the local memory of the computing device, transmitting, by the computing device, the image to the remote server, causing the remote server to search the known visual references stored by the remote server based on the captured visual reference to identify the visual object model corresponding to the captured visual reference;
receiving, from the remote server, the virtual object model corresponding to the captured visual reference associated with the physical object;
displaying, based on the virtual object model, a virtual image over a live image of the physical object;
in response to detecting a movement of the computing device relative to the physical object caused by the user, modifying the virtual image;
detecting a physical manipulation of the physical object;
identifying a virtual function assigned to the physical manipulation of the physical object; and
performing the identified virtual function, the virtual function controlling a physical device electrically unconnected to the physical object.

2. The method of claim 1 , further comprising:
receiving a virtual functionality dataset comprising virtual functionalities associated with physical manipulations of the physical object.

3. The method of claim 1 , further comprising:
receiving first notification data from a first account associated with an identification of a first physical object; and
receiving second notification data from a second account associated with an identification of a second physical object, the first notification data and the second notification data comprising an email notification, a calendar reminder notification, or a social network notification.

4. The method of claim 3 , further comprising:
accessing reference identifiers associated with the first notification data;
accessing an augmented reality visualization associated with the reference identifiers associated with the first notification data; and
generating a virtual object engaged with a reference identifier in the image of the physical object using the augmented reality visualization associated with the reference identifier in the image of the physical object.

5. The method of claim 4 , further comprising:
changing a feature of the virtual object, or replacing the virtual object with another virtual object based on the augmented reality visualization associated with the reference identifier in the image of the physical object.

6. The method of claim 2 , further comprising:
generating the virtual function corresponding to the identified physical manipulation of the physical object.

7. The method of claim 6 , further comprising:
generating a virtual user interface corresponding to the identified physical manipulation of the physical object.

8. The method of claim 6 , further comprising:
activating a function on the computing device in response to the identified physical manipulation of the physical object.

9. The method of claim 6 , further comprising:
generating a command for the virtual function to control another device in response to the identified physical manipulation of the physical object, the other device having no electronic communication with the physical object.

10. A non-transitory computer-readable medium storing instructions that, when executed by one or more computer processors of a computing device, cause the computing device to:
receive a primary content dataset from a remote server, the primary content dataset including a subset of known visual references stored by the remote server that were determined to be most popular by the remote server, wherein each known visual reference corresponds to a virtual object model;
store the primary content dataset in a local memory of the computing device;
after receiving the primary content dataset, capture an image of a physical object, the image including a captured visual reference associated with the physical object;
search the primary content dataset stored in the local memory of the computing device based on the captured visual reference to identify a virtual object model corresponding to the captured visual reference associated with the physical object;
in response to determining that the captured visual reference is not included in the primary content dataset in the local memory of the computing device, transmit the image to the remote server, causing the remote server to search the known visual references stored by the remote server based on the captured visual reference to identify the visual object model corresponding to the captured visual reference;
receive, from the remote server, the virtual object model corresponding to the captured visual reference associated with the physical object;
display, based on the virtual object model, a virtual image over a live image of the physical object;
in response to detecting a movement of the computing device relative to the physical object caused by the user, modify the virtual image;
detect a physical manipulation of the physical object;
identify a virtual function assigned to the physical manipulation of the physical object; and
perform the identified virtual function, the virtual function controlling a physical device electrically unconnected to the physical object.

11. A computing device comprising:
one or more computer processors; and
one or more computer-readable mediums storing instructions that, when executed by the one or more computer processors, cause the computing device to:
receive a primary content dataset from a remote server, the primary content dataset including a subset of known visual references stored by the remote server that were determined to be most popular by the remote server, wherein each known visual reference corresponds to a virtual object model;
store the primary content dataset in a local memory of the computing device;
after receiving the primary content dataset, capture an image of a physical object, the image including a captured visual reference associated with the physical object;
search the primary content dataset stored in the local memory of the computing device based on the captured visual reference to identify a virtual object model corresponding to the captured visual reference associated with the physical object;
in response to determining that the captured visual reference is not included in the primary content dataset in the local memory of the computing device, transmit the image to the remote server, causing the remote server to search the known visual references stored by the remote server based on the captured visual reference to identify the visual object model corresponding to the captured visual r

Assignees

• Daqri Llc

Inventors

• Mullins Brian

Classifications

• G06T19/20

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• G06T2219/2016

  Rotation, translation, scaling · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T2215/16

  Using real world measurements to influence rendering · CPC title