Determining coordinate frames in a dynamic environment

Patent metadata

Field	Value
Publication number	US-10223799-B2
Application number	US-201715474687-A
Country	US
Kind code	B2
Filing date	Mar 30, 2017
Priority date	May 1, 2014
Publication date	Mar 5, 2019
Grant date	Mar 5, 2019

Abstract

Embodiments are disclosed for methods and systems of distinguishing movements of features in a physical environment. For example, on a head-mounted display device, one embodiment of a method includes obtaining a representation of real-world features in two or more coordinate frames and obtaining motion data from one or more sensors external to the head-mounted display device. The method further includes distinguishing features in one coordinate frame from features in another coordinate frame based upon the motion data.

First claim

The invention claimed is:

1. On a head-mounted display device, a method comprising:
obtaining via image data from a camera of the head-mounted display device a representation of real-world features in two or more coordinate frames;
displaying, on a see-through display of the head-mounted display device, a first virtual object at a location relative to a first real-world feature in a first coordinate frame in a field of view of the head-mounted display device and a second virtual object at a location relative to a second real-world feature in a second coordinate frame in the field of view of the head-mounted display device;
determining relative motion characteristics of the first coordinate frame and the second coordinate frame based upon the image data;
distinguishing features in the first coordinate frame from features in the second coordinate frame based upon the relative motion characteristics; and
updating the location of the first virtual object based upon a relative motion of the first coordinate frame and updating the location of the second virtual object based upon a relative motion of the second coordinate frame.

2. The method of claim 1 , further comprising tracking locations of the real-world features relative to the camera over time via the image data.

3. The method of claim 2 , further comprising determining a motion characteristic for each of the real-world features based on tracking changes in the locations of the real-world features.

4. The method of claim 3 , further comprising obtaining motion data from one or more sensors of the head-mounted display device, wherein distinguishing features in the first coordinate frame from features in the second coordinate frame comprises comparing the motion characteristic for each real-world feature to the motion data from the one or more sensors.

5. The method of claim 4 , wherein the one or more sensors includes an inertial measurement unit.

6. The method of claim 5 , further comprising:
obtaining motion data from the inertial measurement unit; and
displaying, on the see-through display of the head-mounted display device, a virtual object at a location relative to a selected real-world object in the field of view, and updating the location of the virtual object based upon the motion data.

7. The method of claim 4 , wherein a first sensor is associated with the first coordinate frame, the method further comprising associating a first real-world feature to the first coordinate frame by comparing the motion characteristic determined for the first real-world feature with motion data from the first sensor.

8. The method of claim 7 , wherein the one or more sensors includes a second sensor associated with the second coordinate frame.

9. The method of claim 1 , wherein obtaining the representation of the real-world features comprises detecting one or more objects, corners of objects, and/or edges of objects in a physical environment.

10. The method of claim 1 , wherein the head-mounted display device further comprises a global positioning sensor unit.

11. The method of claim 10 , further comprising isolating movement of the head-mounted display device relative to a moving element in a second coordinate frame by subtracting the relative motion of the first coordinate frame from the relative motion of the second coordinate frame.

12. A head-mounted display device, comprising:
a plurality of sensors including a camera;
a see-through display;
a logic machine; and
a memory holding instructions executable by the logic machine to
obtain via the camera a representation of real-world features in two or more coordinate frames;
determine relative motion characteristics of a first coordinate frame and a second coordinate frame based upon the representation;
distinguish a first real-world feature in the first coordinate frame from a second real-world feature in the second coordinate frame based upon the relative motion characteristics; and
display a first virtual object at a location relative to the first real-world feature in the first coordinate frame based upon the relative motion characteristics and display a second virtual object at a location relative to the second real-world feature in the second coordinate frame based upon the relative motion characteristics.

13. The head-mounted display device of claim 12 , wherein obtaining the representation of the real-world features further comprises tracking a location of each of the first and second real-world features relative to the camera over time based on image data from the camera.

14. The head-mounted display device of claim 12 , wherein a first sensor of the plurality of sensors is associated with the first coordinate frame, and wherein a second sensor of the plurality of sensors is associated with the second coordinate frame.

15. The head-mounted display device of claim 12 , wherein the plurality of sensors includes a depth camera.

16. The head-mounted display device of claim 12 , wherein the plurality of sensors includes a global positioning sensor unit.

17. The head-mounted display device of claim 12 , wherein the instructions are further executable to update the location at which the first virtual object is displayed based upon a detected change in a location of the first real-world feature relative to the first coordinate frame.

18. On a head-mounted display device, a method comprising:
obtaining via image data from a camera of the head-mounted display device a representation of real-world features in two or more coordinate frames;
displaying, on a see-through display of the head-mounted display device, a first virtual object at a location relative to a first real-world feature in a first coordinate frame in a field of view of the head-mounted display device and a second virtual object at a location relative to a second real-world feature in a second coordinate frame in the field of view of the head-mounted display device;
determining relative motion characteristics of the first coordinate frame and the second coordinate frame based upon the image data and data from one or more sensors;
distinguishing features in the first coordinate frame from features in the second coordinate frame based upon the relative motion characteristics; and
updating the location of the first virtual object based upon a relative motion of the first coordinate frame and updating the location of the second virtual object based upon a relative motion of the second coordinate frame.

19. The method of claim 18 , further comprising merging the first and second coordinate frames into a single coordinate frame if the first and second coordinate frames have no relative motion based on the relative motion characteristics.

20. The method of claim 19 , wherein the data from the one or more sensors includes acceleration data from an inertial measurement unit of the head-mounted display device, and wherein determining the relative motion characteristics comprises comparing the acceleration data to the image data.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Poulos Adam G
• Tomlin Arthur
• Ambrus Tony
• Cole Jeffrey
• Mcintyre Ian Douglas
• Steedly Drew
• Schaffalitzky Frederik
• Klein Georg
• Mulcahy Kathleen P

Classifications

• G06F3/04815

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F3/012 Primary

  Head tracking input arrangements · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G02B2027/0141

  characterised by the informative content of the display · CPC title