Augmented reality bindings of physical objects and virtual objects

Patent metadata

Field	Value
Publication number	US-10937240-B2
Application number	US-201815862280-A
Country	US
Kind code	B2
Filing date	Jan 4, 2018
Priority date	Jan 4, 2018
Publication date	Mar 2, 2021
Grant date	Mar 2, 2021

Abstract

Systems, apparatus, and computer-readable media for managing data storage for generating virtual bindings are provided. In embodiments, a user may perform one or more gestures and/or voice commands to create virtual bindings with physical objects, where the created virtual bindings may take on attributes and create/perform actions based on attributes of the physical objects. A projection device may recognize the physical objects and cause the bindings and/or projected virtual objects to perform various actions in response to different user gestures and/or voice commands. Additionally, the system may instruct some physical objects (e.g., robots, electromechanical devices, etc.) in response to user gestures/voice commands to cause those physical devices to perform various actions. Other embodiments are described and/or claimed.

First claim

The invention claimed is:

1. A computer device employed in an augmented reality (AR) platform, the computer device comprising:
modeling circuitry arranged to obtain first sensor data from a sensor array, wherein the first sensor data is representative of a physical environment and a physical object (PO) in the physical environment, and to track the PO in the physical environment based on the first sensor data;
user input circuitry arranged to obtain, from the sensor array, second sensor data that is representative of a plurality of user inputs;
object recognition circuitry arranged to identify a tracked PO;
AR circuitry arranged to:
generate a first instance of a virtual binding (VB) in response to a first user input of the plurality of user inputs, generation of the first instance of the VB is based on the identified PO and the first user input, wherein the first instance of the VB has one or more VB attributes that influence one or more actions to be performed by the VB with respect to the PO based on a relationship between the identified PO and the VB,
generate a second instance of the VB in response to a second user input of the plurality of user inputs, wherein the second instance is representative of a manipulation of the VB, the second user input simulates a direct, physical manipulation of the VB and the manipulation is to cause the VB to act on the PO; and

interface circuitry communicatively coupled with the AR circuitry, the interface circuitry is arranged to:
provide the first instance of the VB to be displayed such that, upon display of the first instance, the VB is to appear to be attached to the PO, and
provide the second instance of the VB to be displayed such that, upon display of the second instance, the VB is to appear to interact with the PO according to the manipulation and the one or more VB attributes, and
cause a message to be transmitted to the PO over a network connection to instruct the PO to perform one or more actions in response to the manipulation of the VB.

2. The computer device of claim 1 , wherein the AR circuitry is further arranged to generate the first or second instance of the VB when the first or second user input is a gesture to indicate a selection of a particular VB.

3. The computer device of claim 2 , wherein the gesture is a drawing gesture performed by the user to indicate a size and shape of the VB, and the relationship between the identified PO and the VB is based on the size and shape of the VB and a position of the VB with respect to the PO.

4. The computer device of claim 2 , wherein the first instance is a selection instance, the second instance is a VB to PO (VB-PO) association instance of the VB, and wherein:
the AR circuitry is further arranged to:
generate the VB-PO association instance of the VB when the second user input is a gesture to indicate to associate the VB with the PO, and
identify one or more VB-PO attributes of the association of the VB with the PO; and

the interface circuitry is further arranged to provide the VB-PO association instance of the VB to be displayed, wherein the one or more VB-PO attributes indicate whether, upon display of the VB-PO association instance, the VB is to appear to be attached to the PO or appear to interact with the PO.

5. The computer device of claim 4 , wherein the second instance is a PO-manipulate instance, and wherein:
the AR circuitry is further arranged to generate a PO-manipulate instance of the VB when the second user input is a gesture of interacting with the VB to manipulate the VB; and
the interface circuitry is further arranged to provide the PO-manipulate instance of the VB to be displayed such that the VB is to appear to be manipulated in response to the second user input.

6. The computer device of claim 5 , further comprising:
context circuitry arranged to determine one or more contextual attributes, wherein the one or more contextual attributes comprise user activity, semantic location, social circumstances, ambient conditions, presence of one or more electronic devices, schedules, and user communications,
wherein the AR circuitry is arranged to generate the PO-manipulate instance based on the one or more contextual attributes.

7. The computer device of claim 5 , wherein the interface circuitry is further arranged to:
cause the message to be transmitted to the PO over the network connection in response to the second user input.

8. The computer device of claim 1 , wherein the first instance is a selection instance, the second instance is a VO-manipulate instance of the VB, and wherein:
the AR circuitry is further arranged to:
generate a virtual object (VO) when the first user input is a gesture indicating selection of the VO,
generate a VB to VO (VB-VO) association instance of the VB when a third user input of the plurality of user inputs is a gesture indicating to associate the VB with the VO,
identify one or more VB-VO attributes of the association of the VB with the VO, and
generate the VO-manipulate instance of the VB when the second gesture is a gesture to manipulate the VB; and

the interface circuitry is further arranged to:
provide the VO and the VB-VO association instance of the VB to be displayed, wherein the one or more VB-VO attributes indicate whether, upon display of the VB-VO association instance and the VO, the VB is to appear to be attached to the VO or appear to interact with the VO, and
provide the VO-manipulate instance of the VB to be displayed such that the VB is to appear to be manipulated in response to performance of the gesture to manipulate the VB.

9. The computer device of claim 1 , wherein the modeling circuitry is further arranged to generate a three-dimensional (3D) model of the physical environment, and wherein the object recognition circuitry is arranged to identify the PO within the 3D model.

10. The computer device of claim 4 , wherein the interface circuitry is further arranged to:
cause a network connection to be established with the PO; and
provide a VB-PO instance of the VB to be displayed, wherein the more VB-PO attributes indicate whether, upon display of the VB-PO instance, the VB is to appear to be attached to the PO or appear to interact with the PO.

11. One or more non-transitory computer-readable media (NTCRM) comprising instructions, wherein execution of the instructions by one or more processors of a computer device is to cause the computer device to:
obtain first sensor data from a sensor array, wherein the first sensor data is representative of a physical environment and an instrumented object (IO) in the physical environment;
detect the IO in the physical environment based on the first sensor data;
identify the IO and IO attributes associated with the identified IO;
obtain, from the sensor array, second sensor data that is representative of gestures performed by a user;
generate, in response to identification of a first gesture of the gestures represented by the second sensor data, a first instance of a virtual binding (VB) based on the PO attributes, wherein the instance of the VB has one or more VB attributes that influence one or more actions to be performed by the VB, the VB attributes being based on a relationship between the identified IO and the VB;
generate a second instance of the VB, in response to identification of a second gesture of the gestures represented by the second sensor data, the second instance is representative of manipulation of the VB, the second gesture simulates a direct, physical manipulation of the VB, and the manipulation is to cause the VB to act on the IO;
provide the first instance of the VB to be displayed such that, upon display of the first instance, the

Assignees

• Intel Corp

Inventors

• Anderson Glen J

Classifications

• G06V40/20

  Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) · CPC title

• G06V20/20

  in augmented reality scenes · CPC title

• G06T19/20

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title