Augmentation modification based on user interaction with augmented reality scene

Patent metadata

Field	Value
Publication number	US-9791917-B2
Application number	US-201514667302-A
Country	US
Kind code	B2
Filing date	Mar 24, 2015
Priority date	Mar 24, 2015
Publication date	Oct 17, 2017
Grant date	Oct 17, 2017

Abstract

Apparatuses, methods, and storage media for modifying augmented reality in response to user interaction are described. In one instance, the apparatus for modifying augmented reality may include a processor, a scene capture camera coupled with the processor to capture a physical scene, and an augmentation management module to be operated by the processor. The augmentation management module may obtain and analyze the physical scene, generate one or more virtual articles to augment a rendering of the physical scene based on a result of the analysis, track user interaction with the rendered augmented scene, and modify or complement the virtual articles in response to the tracked user interaction. Other embodiments may be described and claimed.

First claim

What is claimed is:

1. An apparatus for providing augmented reality computing, comprising:
a processor;
a scene capture camera coupled with the processor to capture a physical scene;
an interaction capture camera coupled with the processor to capture information about a user; and
an augmentation management module to be operated by the processor to:
obtain and analyze the physical scene,
obtain and analyze the information about the user to determine at least one user modality;
determine one or more virtual articles to augment a rendering of the physical scene, based on a result of the analysis,
render the physical scene to be viewed by the user, wherein the rendered physical scene comprises a rendered augmented scene that includes the physical scene and the determined one or more virtual articles,
track user interaction with the rendered augmented scene, and
modify or complement the one or more virtual articles, in response to the tracked user interaction,
wherein to obtain and analyze the information about the user to determine the at least one user modality is to occur prior to determination of the one or more virtual articles and prior to render of the physical scene and wherein the apparatus is to remain substantially stationary on a substantially horizontal surface to enable the capture of the physical scene and the user interaction.

2. The apparatus of claim 1 , wherein the interaction capture camera is to capture the user interaction with the rendered augmented scene and is to provide information about the captured interaction to the augmentation management module for tracking.

3. The apparatus of claim 2 , wherein the apparatus is selected from one of: a laptop computing device, a tablet computing device, a mobile computing device, or an all-in-one (AIO) computing device.

4. The apparatus of claim 2 , wherein each of the cameras comprises a two-dimensional (2D) or three-dimensional (3D) camera to capture a real-time depth data and color data associated with the physical scene or the user interaction respectively, wherein the color data includes red, green, and blue (RGB) data.

5. The apparatus of claim 1 , wherein the augmentation management module to track user interaction with the rendered augmented scene includes to obtain an indication of user interaction with at least one of the one or more virtual articles in the rendered augmented scene.

6. The apparatus of claim 5 , wherein the augmentation management module to modify or complement the rendered augmented scene includes to:
align the at least one of the one or more virtual articles in the rendered augmented scene with the indicated user interaction with the at least one of the one or more virtual articles in the rendered augmented scene;
alter a position of the at least one of the one or more virtual articles in the rendered augmented scene in response to the indication of the user interaction with the at least one of the one or more virtual articles in the rendered augmented scene; or
alter the at least one of the one or more virtual articles in the rendered augmented scene in response to the indication of the user interaction with the at least one of the one or more virtual articles in the rendered augmented scene.

7. The apparatus of claim 5 , wherein the indication of user interaction includes at least a selected one of: a gesture, a change of facial expression, a verbal command, a change in eye gaze, a change of posture, or a change of a head pose.

8. The apparatus of claim 1 , further comprising: a display device coupled with the processor to display the rendered augmented scene to the user.

9. The apparatus of claim 1 , wherein the augmentation management module to augment a rendering of the physical scene with one or more virtual articles includes to augment the rendering based at least in part on one or more markers associated with the physical scene.

10. The apparatus of claim 1 , wherein the augmentation management module is to modify or complement the rendered augmented scene substantially simultaneously with the tracking of the user interaction.

11. A computer-implemented method for providing augmented reality computing, comprising:
capturing and analyzing visual information about a user to determine at least one user modality;
augmenting, by a computing device, a rendering of a physical scene with one or more virtual articles;
rendering, on the computing device, the physical scene by providing a rendered augmented scene that includes the physical scene and the one or more virtual articles;
tracking, by the computing device, user interaction with the rendered augmented scene, and
modifying or complementing, by the computing device, the one or more virtual articles in the rendered augmented scene in response to the tracking of the user interaction,
wherein capturing and analyzing the information about the user are performed prior to augmenting the rendering of the physical scene and rendering of the physical scene and wherein the computing device is to remain substantially stationary on a substantially horizontal surface to enable the capture of the physical scene and the user interaction.

12. The computer-implemented method of claim 11 , further comprising:
obtaining and analyzing, by the computing device, the physical scene; and
determining, by the computing device, the one or more virtual articles to augment the rendering of the physical scene, based on the analyzing the physical scene.

13. The computer-implemented method of claim 11 , further comprising:
rendering, by the computing device, the rendered augmented scene for display.

14. The computer-implemented method of claim 13 , wherein tracking user interaction with the rendered augmented scene includes:
obtaining, by the computing device, an indication of user interaction with at least one of the one or more virtual articles in the rendered augmented scene, and
wherein modifying or complementing the one or more articles further includes:
aligning, by the computing device, the at least one of the one or more virtual articles in the rendered augmented scene with the indicated user interaction substantially simultaneously with and in response to the obtaining of the indication of the user interaction with the at least one of the one or more virtual articles in the rendered augmented scene.

15. The computer-implemented method of claim 14 , wherein obtaining an indication of user interaction includes detecting, by the computing device, at least a selected one of: a gesture, a change of facial expression, a verbal command, a change in eye gaze, a change of posture, or a change of a head pose.

16. One or more non-transitory computer-readable media having instructions for providing augmented reality computing stored thereon that, in response to execution by a computing device, provide the computing device with an augmentation management environment to:
obtain and analyze an image including a user to determine at least one user modality associated with the user;
augment a rendering of a physical scene with one or more virtual articles;
render the physical scene by providing a rendered augmented scene that includes the physical scene and the one or more virtual articles;
track user interaction with the rendered augmented scene, and
modify or complement the one or more articles in the rendered augmented scene in response to the tracking of the user interaction,
wherein to obtain and analyze the image including the user to determine the at least one user modality associated with the user is to occur prior to augment the rendering of the p

Assignees

• Intel Corp

Inventors

• Kamhi Gila
• Moran Amit

Classifications

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06F3/005 Primary

  Input arrangements through a video camera · CPC title

• G06F3/167

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title

• G06F3/013 Primary

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title