Autonomous drones for tactile feedback in immersive virtual reality

Patent metadata

Field	Value
Publication number	US-9836117-B2
Application number	US-201514724360-A
Country	US
Kind code	B2
Filing date	May 28, 2015
Priority date	May 28, 2015
Publication date	Dec 5, 2017
Grant date	Dec 5, 2017

Abstract

A “Tactile Autonomous Drone” (TAD) (e.g., flying drones, mobile robots, etc.) supplies real-time tactile feedback to users immersed in virtual reality (VR) environments. TADs are not rendered into the VR environment, and are therefore not visible to users immersed in the VR environment. In various implementations, one or more TADs track users as they move through a real-world space while immersed in the VR environment. One or more TADs apply tracking information to autonomously position themselves, or one or more physical surfaces or objects carried by the TADs, in a way that enables physical contact between those surfaces or objects and one or more portions of the user's body. Further, this positioning of surfaces or objects corresponds to some real-time virtual event, virtual object, virtual character, virtual avatar of another user, etc., in the VR environment to provide real-time tactile feedback to users immersed in the VR environment.

First claim

What is claimed is:

1. A system, comprising:
a processor; and
a memory comprising instructions that, when executed by the processor, cause the processor to perform a method comprising:
generating a real-time rendering of a virtual environment;
receiving real-time tracking information from one or more sensors, the real-time tracking information relating to real motions and positions of a user immersed in the real-time rendering of the virtual environment, the real-time tracking information further relating to motions and positions of an autonomous mobile drone in a real-world environment around the user;
responsive to receipt of the real-time tracking information, directing the autonomous mobile drone to move relative to the user;
further responsive to receipt of the real-time tracking information, directing the autonomous mobile drone to automatically deposit a real-world object in a position within the real-world environment that enables one or more user body parts to contact that object based on real user motions while the user interacts with the virtual environment;
responsive to depositing the real-world object by the autonomous mobile drone, rendering a virtual representation of the real-world object into a corresponding position in the virtual environment; and
wherein contact between the real-world object and the one or more user body parts generates a physically tactile sensation for the virtual representation of the real-world object.

2. The system of claim 1 further comprising directing one or more additional autonomous mobile drones to automatically move within the real-world environment around the user to maintain optimal positions to apply one or more integral sensors to track and model one or more specific user body parts.

3. The system of claim 1 wherein the movement of the autonomous mobile drone relative to the user is further synchronized to the real-time rendering of the virtual environment.

4. The system of claim 1 further comprising evaluating real-time positions and motions of the user and the autonomous mobile drone and automatically directing movement of the autonomous mobile drone to ensure that no unintended contact is made between the user and any portion of the autonomous mobile drone.

5. The system of claim 1 directing the autonomous mobile drone to automatically position itself relative to the user in a manner that causes one or more tactile virtual extensions of the autonomous mobile drone to contact one or more user body parts based on real user motions while the user interacts with the virtual environment.

6. The system of claim 5 further comprising causing the tactile virtual extension of the autonomous mobile drone to initiate a force-feedback upon contact between the tactile virtual extension and the one or more user body parts.

7. The system of claim 1 wherein a plurality of tactile virtual extensions are coupled to movable members of the autonomous mobile drone, each movable member capable of moving in multiple dimensions to accurately position one or more of the tactile virtual extensions relative to the user.

8. The system of claim 1 further comprising presenting the real-time rendering of the virtual environment to the user in an otherwise empty real-world environment around the user that allows the user full freedom of motion.

9. The system of claim 1 further comprising rendering one or more safety markers into the immersive virtual environment to alert the user as to real elements existing within the real-world environment around the user.

10. The system of claim 1 further comprising replacing the real-time rendering of a virtual environment with a live video of the real-world environment around the user based on a measured distance between the user and the autonomous mobile drone to ensure that no unintended contact is made between the user and the autonomous mobile drone.

11. The system of claim 1 wherein the autonomous mobile drone is capable of autonomous flight in the real-world environment around the user.

12. The system of claim 1 further comprising an audio output device coupled to the autonomous mobile drone, and further comprising causing the audio output device of the autonomous mobile drone to deliver sound from one or more points in the real-world space around the user, said sound being coordinated with and mapped to positions of corresponding virtual elements in the virtual environment.

13. A method, comprising:
generating a real-time rendering of a virtual environment;
displaying the real-time rendering of the virtual environment via a head-worn virtual reality display device;
receiving real-time tracking information from one or more sensors, the real-time tracking information relating to real motions and positions of a user immersed in the real-time rendering of the virtual environment, the real-time tracking information further relating to motions and positions of an autonomous mobile drone in a real-world environment around the user;
responsive to receipt of the real-time tracking information, directing the autonomous mobile drone to move relative to the user;
further responsive to receipt of the real-time tracking information, directing the autonomous mobile drone to automatically deposit a real-world object in a position within the real-world environment that enables one or more user body parts to contact that object based on real user motions while the user interacts with the virtual environment;
responsive to depositing the real-world object by the autonomous mobile drone, rendering a virtual representation of the real-world object into a corresponding position in the virtual environment; and
wherein contact between the real-world object and the one or more user body parts generates a physically tactile sensation for the virtual representation of the real-world object.

14. The method of claim 13 further comprising directing one or more additional autonomous mobile drones to automatically move within the real-world environment around the user to maintain optimal positions to apply one or more integral sensors to track and model one or more specific user body parts.

15. The method of claim 13 further comprising synchronizing the movement of the autonomous mobile drone relative to the user to the real-time rendering of the virtual environment.

16. The method of claim 13 further comprising evaluating real-time positions and motions of the user and the autonomous mobile drone and automatically directing movement of the autonomous mobile drone to ensure that no unintended contact is made between the user and any portion of the autonomous mobile drone.

17. The method of claim 13 further comprising directing the autonomous mobile drone to automatically position itself relative to the user in a manner that causes one or more tactile virtual extensions of the autonomous mobile drone to contact one or more user body parts based on real user motions while the user interacts with the virtual environment.

18. The method of claim 13 further comprising directing a plurality of tactile virtual extensions that are coupled to movable members of the autonomous mobile drone to move in one or more dimensions to accurately position one or more of the tactile virtual extensions relative to one or more body parts of the user.

19. The method of claim 13 further comprising rendering one or more safety markers into the immersive virtual environment to alert the user as to real elements existing within the real-world environment around the user.

20. A computer-r

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Shapira Lior

Classifications

• B64U2201/10

  autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS] · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• B25J9/1605

  Simulation of manipulator lay-out, design, modelling of manipulator · CPC title

• G02B27/017

  Head mounted · CPC title

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title