Systems and methods for immersive interaction with virtual objects

Patent metadata

Field	Value
Publication number	US-10486065-B2
Application number	US-201113192305-A
Country	US
Kind code	B2
Filing date	Jul 27, 2011
Priority date	May 29, 2009
Publication date	Nov 26, 2019
Grant date	Nov 26, 2019

Abstract

A system to present the user a 3-D virtual environment as well as non-visual sensory feedback for interactions that user makes with virtual objects in that environment is disclosed. In an exemplary embodiment, a system comprising a depth camera that captures user position and movement, a three-dimensional (3-D) display device that presents the user a virtual environment in 3-D and a haptic feedback device provides haptic feedback to the user as he interacts with a virtual object in the virtual environment. As the user moves through his physical space, he is captured by the depth camera. Data from that depth camera is parsed to correlate a user position with a position in the virtual environment. Where the user position or movement causes the user to touch the virtual object, that is determined, and corresponding haptic feedback is provided to the user.

First claim

What is claimed:

1. A system for providing non-visual sensory feedback to a user interacting with a virtual object, comprising:
a processor; and
a memory communicatively coupled to the processor when the system is operational, the memory bearing processor-executable instructions that, when executed on the processor, cause the system to at least:
receive data captured by a capture device, the data comprising at least one of a velocity, acceleration or rotation of a body part associated with a user;
predict, based on the data, a future interaction by the user with the virtual object at a future time;
compute, prior to the future time, and based on the predicted future interaction by the user with the virtual object, a non-visual sensory feedback that corresponds to the future interaction by the user with the virtual object and a duration for providing the non-visual sensory feedback, computing the non-visual sensory feedback comprising computing, based on the predicted future interaction, an amount of non-visual sensory feedback prior to the future time and before the predicted future interaction actually occurs; and
provide the computed non-visual sensory feedback to the user at the future time.

2. The system of claim 1 , wherein the memory further bears processor-executable instructions, that when executed on the processor, cause the system to at least:
indicate to a three-dimensional (3-D) display to display the virtual object to the user.

3. The system of claim 1 , wherein the non-visual sensory feedback comprises:
haptic feedback or olfactory feedback.

4. The system of claim 3 , wherein the virtual object corresponds to a second user interacting with the user across a communications network, and the instructions that cause the system to at least compute, prior to the future time, the non-visual sensory feedback that corresponds to the future interaction further cause the system to at least:
compute, prior to the future time, a non-visual sensory feedback that corresponds to the future interaction based on a haptic feedback received by the second user.

5. The system of claim 3 , wherein the haptic feedback comprises:
haptic feedback produced by an ultrasonic wave, haptic feedback produced by a stream of air, or haptic feedback produced by a device coupled to the location of the user.

6. The system of claim 1 , wherein the instructions that cause the system to at least provide the computed non-visual sensory feedback to the user at the at the future time further cause the system to at least:
provide the computed non-visual sensory feedback to the user at the at a future location of the user at the future time, the future location of the user corresponding to a location of the user that is likely to interact with the virtual object at the future time.

7. The system of claim 1 , wherein the memory further bears processor-executable instructions that, when executed on the processor, cause the system to at least:
determine a characteristic of feedback to provide to the future user location based on at least one quality of the user's interaction with the virtual object.

8. The system of claim 7 , wherein the quality of user's interaction comprises:
a strength of the user, a task being completed, a velocity of the virtual object, a mass of the virtual object or the user, or a material of the virtual object.

9. The system of claim 1 , wherein instructions that cause the system to at least provide the computed non-visual sensory feedback to the user at the future time further cause the system to at least:
direct audio from an audio speaker substantially centered on the user.

10. The system of claim 9 , wherein a second user is present in a scene of the user, and the instructions that cause the system to at least direct audio from an audio speaker substantially centered on the user further cause the system to at least:
direct audio to the user such that it is understandable exclusively to the user.

11. A method for providing non-visual sensory feedback to a user interacting with a virtual object, comprising:
receiving data captured by a capture device, the data comprising at least one of a velocity, acceleration or rotation of a body part associated with a user;
predicting, based on the data, a future interaction by the user is with the virtual object at a future time;
computing, prior to the future time, and based on the predicted future interaction by the user with the virtual object, a non-visual sensory feedback that corresponds to the future interaction by the user with the virtual object and a duration for providing the non-visual sensory feedback, computing the non-visual sensory feedback comprising computing, based on the predicted future interaction, an amount of non-visual sensory feedback prior to the future time and before the predicted future interaction actually occurs; and
providing the computed non-visual sensory feedback to the user at the future time.

12. The method of claim 11 , further comprising:
displaying the virtual object to the user via a three-dimensional (3-D) display, wherein the 3-D display is one from a set, the set comprising:
an active 3-D display and a passive 3-D display.

13. The method of claim 11 , wherein the non-visual sensory feedback is one from a set, the set comprising:
haptic feedback, and olfactory feedback.

14. The method of claim 13 , wherein an amount of the haptic feedback corresponds to a strength of the user.

15. The method of claim 13 , wherein a type of the haptic feedback comprises:
a velocity of the virtual object, a mass of the virtual object or the user, or a material of the virtual object.

16. The method of claim 13 , wherein the haptic feedback corresponds to Braille.

17. The method of claim 11 , wherein the feedback corresponds to a task being completed.

18. The method of claim 11 , wherein the location of the user corresponds to a representation on a display device and the virtual object is displayed on the display device.

19. A computer-readable storage device bearing computer-readable instructions that, when executed on a computer, cause the computer to perform operations comprising:
displaying a virtual object to a user via a three-dimensional (3-D) display;
receiving data captured by a capture device, the data comprising at least one of a velocity, acceleration or rotation of a body part associated with the user;
predicting, based on the data, a future interaction by a user with the virtual object at a future time;
computing, prior to the future time, and based on the predicted future interaction by the user with the virtual object, a non-visual sensory feedback that corresponds to the future interaction by the user with the virtual object and a duration for providing the non-visual sensory feedback, computing the non-visual sensory feedback comprising computing, based on the predicted future interaction, an amount of non-visual sensory feedback prior to the future time and before the predicted future interaction actually occurs; and
providing the computed non-visual sensory feedback to the user at the future time.

20. The computer-readable storage device of claim 19 , wherein the virtual object corresponds to a second user interacting with the user across a communications network, and wherein computing, prior to the future time, the non-visual sensory feedback that corresponds to the future interaction comprises:
computing, prior to the future time, a non-visual sensory feedback that corresponds to the future interac

Assignees

• Kipman Alex
• Tsunoda Kudo
• Holmdahl Todd Eric
• Clavin John
• Perez Kathryn Stone
• Microsoft Technology Licensing Llc

Inventors

• Kipman Alex
• Tsunoda Kudo
• Holmdahl Todd Eric
• Clavin John
• Perez Kathryn Stone

Classifications

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• A63F13/42

  by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• A63F13/285 Primary

  Generating tactile feedback signals via the game input device, e.g. force feedback · CPC title

• A63F13/54

  involving acoustic signals, e.g. for simulating revolutions per minute [RPM] dependent engine sounds in a driving game or reverberation against a virtual wall · CPC title