Autonomous drones for tactile feedback in immersive virtual reality

Patent metadata

Field	Value
Publication number	US-2016349835-A1
Application number	US-201514724360-A
Country	US
Kind code	A1
Filing date	May 28, 2015
Priority date	May 28, 2015
Publication date	Dec 1, 2016
Grant date	—

Abstract

A “Tactile Autonomous Drone” (TAD) (e.g., flying drones, mobile robots, etc.) supplies real-time tactile feedback to users immersed in virtual reality (VR) environments. TADs are not rendered into the VR environment, and are therefore not visible to users immersed in the VR environment. In various implementations, one or more TADs track users as they move through a real-world space while immersed in the VR environment. One or more TADs apply tracking information to autonomously position themselves, or one or more physical surfaces or objects carried by the TADs, in a way that enables physical contact between those surfaces or objects and one or more portions of the user's body. Further, this positioning of surfaces or objects corresponds to some real-time virtual event, virtual object, virtual character, virtual avatar of another user, etc., in the VR environment to provide real-time tactile feedback to users immersed in the VR environment.

First claim

What is claimed is:

1 . A system, comprising:
a real autonomous mobile drone having one or more tactile virtual extensions; one or more sensors that capture real-time tracking information relating to real motions and positions of a user immersed in a real-time rendering of a virtual environment, and further relating to motions and positions of the autonomous mobile drone in a real-world environment around the user; applying the real-time tracking information to cause the autonomous mobile drone to move relative to the user based on the tracking information; the autonomous mobile drone automatically positioning itself relative to the user in a manner that causes one of tactile virtual extensions of the autonomous mobile drone to move relative to the user and to contact one or more user body parts based on real user motions while the user interacts with the virtual environment; and wherein the contact between the tactile virtual extension and the one or more user body parts generates a physically tactile sensation for one or more virtual elements of the virtual environment.

2 . The system of claim 1 wherein one or more autonomous mobile drones automatically move within the real-world environment around the user to maintain optimal positions to apply one or more integral sensors to track and model one or more specific user body parts.

3 . The system of claim 1 wherein the movement of the autonomous mobile drone relative to the user is further synchronized to the real-time rendering of the virtual environment.

4 . The system of claim 1 further comprising evaluating real-time positions and motions of the user and the autonomous mobile drone and automatically moving the autonomous mobile drone to ensure that no unintended contact is made between the user and any portion of the autonomous mobile drone.

5 . The system of claim 1 further comprising applying one or more autonomous mobile drones to place one or more real objects in the real-world environment around the user in positions mapped to virtual objects in the virtual environment.

6 . The system of claim 1 wherein a plurality of tactile virtual extensions are coupled to movable members of the autonomous mobile drone, each movable member capable of moving in multiple dimensions to accurately position one or more of the tactile virtual extensions relative to the user.

7 . The system of claim 1 further comprising presenting the real-time rendering of the virtual environment to the user in an otherwise empty real-world environment around the user that allows the user full freedom of motion.

8 . The system of claim 1 further comprising rendering one or more safety markers into the immersive virtual environment to alert the user as to real elements existing within the real-world environment around the user.

9 . The system of claim 1 further comprising replacing the real-time rendering of a virtual environment with a live video of the real-world environment around the user whenever it is likely that the user may make unintended contact with the autonomous mobile drone.

10 . The system of claim 1 wherein the autonomous mobile drone is capable of autonomous flight in the real-world environment around the user.

11 . The system of claim 1 wherein the autonomous mobile drone causes the tactile virtual extension to initiate a force-feedback upon contact between the tactile virtual extension and the one or more user body parts.

12 . The system of claim 1 further comprising an audio output device coupled to the autonomous mobile drone such that the autonomous mobile drone delivers sound from one or more points in the real-world space around the user, said sound being coordinated with and mapped to positions of corresponding virtual elements in the virtual environment.

13 . A method, comprising:
applying one or more sensors to capture real-time tracking information; the real-time tracking information including real motions and positions of a user immersed in a real-time virtual reality (VR), and motions and positions of a plurality of real autonomous drones; applying the real-time tracking information to cause one or more of the autonomous drones to automatically position one or more tactile virtual extensions coupled to the autonomous drone to contact one or more user body parts based on real user motions while the user interacts with the real-time VR; wherein the contact between the tactile virtual extension and the one or more user body parts is synchronized and mapped to one or more virtual elements of the real-time VR such that the contact generates a physically tactile sensation for one or more virtual elements of the real-time VR; and applying a safety mechanism that evaluates real-time positions and motions of the user and each of the autonomous drones, and that automatically causes the autonomous drones to move to positions that ensure that no unintended contact is made between the user and any portion of any autonomous drone.

14 . The method of claim 13 further comprising applying one or more sensors integral to one or more of the autonomous drones to maintain optimal positions to track and model one or more specific user body parts.

15 . The method of claim 13 further comprising applying one or more autonomous drones to place one or more real objects in a real-world environment around the user in positions mapped to virtual elements of the real-time VR.

16 . The method of claim 13 wherein one or more autonomous drones causes one or more of the tactile virtual extensions to initiate a force-feedback via the tactile virtual extension upon contact between the tactile virtual extension and the one or more user body parts.

17 . The method of claim 13 wherein a plurality of tactile virtual extensions are coupled to movable members of one or more of the autonomous drones, each movable member capable of moving in multiple dimensions to accurately position one or more of the tactile virtual extensions relative to the user.

18 . A computer-implemented process comprising using a computer to perform process actions for:
directing a plurality of autonomous drones to maintain optimum positions for obtaining tracking information for one or more specific body parts of a user; applying the tracking information to model the specific body parts and render those body parts into an immersive virtual environment being presented to the user; applying the tracking information to cause one or more of the autonomous drones to automatically position one or more tactile virtual extensions coupled to the autonomous drones to contact one or more user body parts based on real user motions while the user interacts with the immersive virtual environment; and wherein the contact between the tactile virtual extension and the one or more user body parts is synchronized and mapped to one or more virtual elements of the immersive virtual environment such that the contact generates a physically tactile sensation for one or more virtual elements of the immersive virtual environment.

19 . The computer-implemented process of claim 18 further comprising process actions for:
applying a safety mechanism that evaluates real-time positions and motions of the user and each of the autonomous drones; and
automatically directing the autonomous drones to move to positions that ensure that no unintended contact is made between the user and any portion of any autonomous drone.

20 . The computer-implemented process of claim 18 further comprising process actions for

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Shapira Lior

Classifications

• B64U2201/10

  autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS] · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G05D1/0038

  by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation (images analyzed by a computer and used for automatic navigation G05D1/0246) · CPC title

• G06F3/016

  Input arrangements with force or tactile feedback as computer generated output to the user · CPC title