Determining coordinate frames in a dynamic environment

Patent metadata

Field	Value
Publication number	US-9626802-B2
Application number	US-201414267755-A
Country	US
Kind code	B2
Filing date	May 1, 2014
Priority date	May 1, 2014
Publication date	Apr 18, 2017
Grant date	Apr 18, 2017

Abstract

Embodiments are disclosed for methods and systems of distinguishing movements of features in a physical environment. For example, on a head-mounted display device, one embodiment of a method includes obtaining a representation of real-world features in two or more coordinate frames and obtaining motion data from one or more sensors external to the head-mounted display device. The method further includes distinguishing features in one coordinate frame from features in another coordinate frame based upon the motion data.

First claim

The invention claimed is:

1. On a head-mounted display device, a method comprising:
obtaining via image data from a camera a representation of real-world features in two or more coordinate frames;
obtaining motion data from one or more sensors external to the head-mounted display device, each sensor being associated with a corresponding coordinate frame;
displaying, on a see-through display of the head-mounted display device, a first virtual object at location relative to a first real-world feature in a first coordinate frame in a field of view of the display device and a second virtual object at location relative to a second real-world feature in a second coordinate frame in the field of view of the display device;
determining relative motion characteristics of the sensors based upon the motion data;
distinguishing features in the first coordinate frame from features in the second coordinate frame based upon the relative motion characteristics; and
updating the location of the first virtual object based upon a relative motion of the first coordinate frame and updating the location of the second virtual object based upon a relative motion of the second coordinate frame.

2. The method of claim 1 , further comprising tracking locations of the first real-world feature and the second real-world feature relative to the camera over time via the image data.

3. The method of claim 2 , further comprising determining a motion characteristic for the first real-world feature and a motion characteristic for the second real-world feature based on tracking changes in the locations of the first real-world feature and the second real-world feature.

4. The method of claim 3 , wherein distinguishing the features in the first coordinate frame from the features in the second coordinate frame comprises comparing the motion characteristics for each of the first real-world feature and the second real-world feature to the motion data from the one or more sensors.

5. The method of claim 4 , wherein a first sensor is associated with the first coordinate frame, and the method further comprising associating the first real-world feature to the first coordinate frame by comparing the motion characteristic determined for the first real-world feature with motion data from the first sensor.

6. The method of claim 5 , wherein the one or more sensors includes a second sensor associated with the second coordinate frame.

7. The method of claim 1 , wherein obtaining the representation of the real-world features comprises detecting one or more objects, corners of objects, and/or edges of objects in a physical environment.

8. The method of claim 1 , wherein displaying the first virtual object and displaying the second virtual object comprise displaying the first virtual object and the second virtual object simultaneously.

9. The method of claim 1 , wherein the one or more sensors includes an inertial measurement unit external to the head-mounted display device.

10. The method of claim 9 , further comprising isolating movement of the head-mounted display device in the first coordinate frame relative to another moving element in the second coordinate frame by subtracting a motion signal from the inertial measurement unit external to the head-mounted display device from a motion signal from an inertial measurement unit on the head-mounted display device.

11. A head-mounted display device, comprising:
a plurality of sensors;
a see-through display;
a logic machine; and
a memory holding instructions executable by the logic machine to
obtain a representation of real-world features in two or more coordinate frames, the representation obtained from image data from a camera;
receive sensor data from a first sensor corresponding to a first coordinate frame and a second sensor corresponding to a second coordinate frame, and one or more of the first sensor and the second sensor being located external to the head-mounted display device;
determine relative motion characteristics of the first sensor and the second sensor based upon motion data;
distinguish a first real-world feature in the first coordinate frame from a second real-world feature in the second coordinate frame based upon the relative motion characteristics; and
display a first virtual object at location relative to the first real-world feature in the first coordinate frame based upon the relative motion characteristics and display a second virtual object at location relative to the second real-world feature in the second coordinate frame based upon the relative motion characteristics.

12. The head-mounted display device of claim 11 , wherein obtaining the representation of the real-world features further comprises tracking locations of the first real-world feature and the second real-world feature relative to the camera over time based on the image data from the camera.

13. The head-mounted display device of claim 11 , wherein the first sensor and the second sensor are external to the head-mounted display device.

14. The head-mounted display device of claim 11 , wherein the first sensor and the second sensor each includes one or more of an external camera and an external inertial measurement unit.

15. The head-mounted display device of claim 11 , wherein the plurality of sensors includes a global positioning sensor unit.

16. The head-mounted display device of claim 11 , wherein the instructions are further executable to update a location at which the first virtual object is displayed based upon a detected change in the location of the first real-world feature relative to a coordinate frame of the head-mounted display device.

17. On a head-mounted display device, a method comprising:
obtaining a representation of real-world features in a coordinate frame using image data from a camera;
obtaining motion data from two or more inertial measurement units, one or more of the inertial measurement units being external to the head-mounted display device;
detecting movement of a first real-world feature relative to a second real-world feature based on information from the image data and the motion data;
separating the coordinate frame into different coordinate frames including a first coordinate frame and a second coordinate frame, the first real-world feature being associated with the first coordinate frame based upon data from a first inertial measurement unit and the second real-world feature being associated with the second coordinate frame based upon data from a second inertial measurement unit; and
displaying a first virtual object associated with the first real-world feature in the first coordinate frame and a second virtual object associated with the second real-world feature in the second coordinate frame.

18. The method of claim 17 , further comprising merging the first and second coordinate frames into a single coordinate frame based at least on the first and second coordinate frames not having relative motion.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Poulos Adam G
• Tomlin Arthur
• Ambrus Tony
• Cole Jeffrey
• Mcintyre Ian Douglas
• Steedly Drew
• Schaffalitzky Frederik
• Klein Georg
• Mulcahy Kathleen P

Classifications

• G02B2027/014

  comprising information/image processing systems · CPC title

• G06F3/012 Primary

  Head tracking input arrangements · CPC title

• G02B2027/0141

  characterised by the informative content of the display · CPC title

• G02B2027/0178

  Eyeglass type (eyeglass details G02C) · CPC title

• G06T7/60

  Analysis of geometric attributes · CPC title