Safety for wearable virtual reality devices via object detection and tracking

Patent metadata

Field	Value
Publication number	US-9754167-B1
Application number	US-201514690296-A
Country	US
Kind code	B1
Filing date	Apr 17, 2015
Priority date	Apr 17, 2014
Publication date	Sep 5, 2017
Grant date	Sep 5, 2017

Abstract

The technology disclosed can provide improved safety by detecting potential unsafe conditions (e.g., collisions, loss of situational awareness, etc.) confronting the user of a wearable (or portable) sensor configured to capture motion and/or determining the path of an object based on imaging, acoustic or vibrational waves. Implementations can enable improved safety to users of virtual reality for machine control and/or machine communications applications using wearable (or portable) devices, e.g., head mounted displays (HMDs), wearable goggles, watch computers, smartphones, and so forth, or mobile devices, e.g., autonomous and semi-autonomous robots, factory floor material handling systems, autonomous mass-transit vehicles, automobiles (human or machine driven), and so forth, equipped with suitable sensors and processors employing optical, audio or vibrational detection.

First claim

What is claimed is:

1. A method of detecting an imminent collision with a physical object during a virtual reality experience, the method including:
using at least one forward facing camera mounted to a head mounted virtual reality device (HMVRD) to capture a sequence of images of a real world environment ahead of a user wearing the HMVRD;
using features in the images to track proximity of at least one physical object ahead of the user;
correlating multiple images to detect relative motion between at least one approaching physical object and the user; and
responsive to detecting the approaching physical object within a proximity threshold, automatically generating an alert to be injected into a virtual reality experience of the user;
at a first time t0, using a sensor attached to the HMVRD, sensing a first position of at least one physical object in a first reference frame of the real world environment, including tracking portions of the physical object;
causing display of a first virtual representation of the physical object at the first position, wherein the first virtual representation is rendered in a virtual environment of the HMVRD;
at a second time t1, sensing, in the real world environment, a second position of the physical object and at least some of the portions different from the first position responsive to repositioning of the real world environment and the attached sensor due to body movement of the user, wherein the physical object has not moved in the real world environment between t0 and t1; and
causing display of a second virtual representation of the physical object at an actual second position, including:
sensing motion of the attached sensor and calculating a second reference frame that accounts for repositioning of the attached sensor;
calculating a transformation that renders the first position in the first reference frame and the second position in the second reference frame into a common reference frame; and
transforming the first and second positions of the physical object into the common reference frame, wherein the common reference frame has a fixed point of reference and an initial orientation of axes, whereby the sensed second position is transformed to the actual second position.

2. The method of claim 1 , wherein automatically generating the alert further includes generating data that represents the approaching physical object in a virtual environment of the HMVRD.

3. The method of claim 1 , further including entering a pass through mode that interrupts the virtual reality experience and substitutes a live feed of the real world environment in a virtual environment of the HMVRD.

4. The method of claim 1 , wherein automatically generating the alert further includes generating a physical warning feedback.

5. The method of claim 4 , wherein the physical warning feedback is a haptic feedback.

6. The method of claim 4 , wherein the physical warning feedback is an audio message generated by the HMVRD.

7. The method of claim 4 , wherein the physical warning feedback is a visual feedback generated in a virtual environment of the HMVRD.

8. The method of claim 1 , wherein the common reference frame is a world reference frame that does not change as the attached sensor is repositioned.

9. The method of claim 1 , wherein the common reference frame is the second reference frame.

10. The method of claim 1 , wherein the transforming the first and second positions of the physical object into the common reference frame further includes applying an affine transformation.

11. The method of claim 1 , further including determining the orientation of the physical object at the first position with respect to the first reference frame and causing the display of the physical object accordingly.

12. The method of claim 1 , further including determining the orientation of the physical object at the second position with respect to the second reference frame and causing the display of the physical object accordingly.

13. The method of claim 1 , further including using features in the images to distinguish among different approaching physical objects in the real world environment.

14. The method of claim 1 , further including using features in the images to distinguish among different perspectives of a particular approaching physical object from varying vantage points of the forward facing camera.

15. The method of claim 1 , wherein using features in the images to track proximity of at least one physical object ahead of the user further includes detecting at least one difference between the features between times t0 and t1 using at least one of a:
vector field flow; and
probabilistic flow.

16. The method of claim 1 , the method further including:
automatically signaling the virtual reality device to generate an alert to a user of the virtual reality device.

17. The method of claim 1 , the method further including:
creating a virtual three-dimensional (3D) map of a real world environment surrounding a user of a virtual reality device using at least one camera coupled to the virtual reality device that captures a sequence of images of the real world environment;
using features in the images to detect positions and shapes of one or more physical objects in the real world environment and incorporating the detected physical objects as 3D constraints in the 3D map, wherein the 3D constraints represent actual positions and shapes of the physical objects in the real world environment;
localizing a position of the virtual reality device in the real world environment; and
responsive to detecting an approaching 3D constraint within a proximity threshold of the user using the virtual reality device, automatically signaling the virtual reality device to generate an alert to the user.

18. The method of claim 17 , wherein automatically generating the alert further includes at least one of:
generating data that represents the approaching physical object in a virtual environment of the virtual reality device;
entering a pass through mode that interrupts the virtual reality experience and substitutes a live feed of the real world environment in a virtual environment of the virtual reality device; and
generating a physical warning feedback, including:
a haptic feedback;
an audio message generated by the virtual reality device; and

a visual feedback generated in a virtual environment of the virtual reality device.

19. A non-transitory computer readable medium impressed with instructions for detecting an imminent collision with a physical object during a virtual reality experience, which instructions when executed by one or more processors cause the processors to perform the following:
using at least one forward facing camera mounted to a head mounted virtual reality device (HMVRD) to capture a sequence of images of a real world environment ahead of a user wearing the HMVRD;
using features in the images to track proximity of at least one physical object ahead of the user;
correlating multiple images to detect relative motion between at least one approaching physical object and the user; and
responsive to detecting the approaching physical object within a proximity threshold, automatically generating an alert to be injected into a virtual reality experience of the user;
at a first time t0, using a sensor attached to the HMVRD, sensing a first position of at least one physical object in a first reference frame of the real world environment, including tracking portions of the physical object;
causing display of a first vir

Assignees

• Leap Motion Inc

Inventors

• Holz David S
• Gordon Robert Samuel
• Perez Matias

Classifications

• G06F3/00

  Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T7/248

  involving reference images or patches · CPC title

• G06T2207/10004

  Still image; Photographic image · CPC title