Recalibration of a flexible mixed reality device

Patent metadata

Field	Value
Publication number	US-9311883-B2
Application number	US-201113294825-A
Country	US
Kind code	B2
Filing date	Nov 11, 2011
Priority date	Nov 11, 2011
Publication date	Apr 12, 2016
Grant date	Apr 12, 2016

Abstract

The technology provides embodiments for recalibration of outward facing cameras supported by a see-through, head mounted, mixed reality display system having a flexible portion between see-through displays for the eyes. Each outward facing camera has a fixed spatial relationship with a respective or corresponding see-through display positioned to be seen through by a respective eye. For front facing cameras, the fixed spatial relationship allows a predetermined mapping between positions on an image sensor of each camera and positions on the respective display. The mapping may be used to register a position of a virtual object to a position of a real object. A change in a first flexible spatial relationship between the outward facing cameras can be automatically detected. A second spatial relationship between the cameras is determined. A registration of a virtual object to a real object may be updated based on the second spatial relationship.

First claim

What is claimed is:

1. A method of recalibrating a head mounted, mixed reality display system having a frame including a flexible frame portion disposed between displays such that a spatial relationship between the displays changes when the flexible frame portion flexes, the displays respectively including respective see-through displays, the method comprising:
detecting a change in a first spatial relationship between outward facing and digital image data capturing cameras supported by the frame of the head mounted, mixed reality display system, the frame positioning the see-through displays for respective positioning before respective ones of user eyes, each outward facing camera having a fixed spatial relationship with respect to its respective see-through display, the flexible frame portion being positioned in-between the outward facing cameras and in-between the see-through displays so that flexing of the flexible frame portion changes at least one spatial relationship between the cameras as well as changing a corresponding at least one spatial relationship between the see-through displays due to the corresponding fixed spatial relationship between each camera and its respective see-through display;
determining a second spatial relationship between the outward facing cameras based on the detected change in the first spatial relationship between the outward facing cameras; and
automatically updating processing of image data based on the second spatial relationship and the fixed spatial relationship of each outward facing camera with respect to its respective see-through display.

2. The method of claim 1 wherein detecting a change in the first spatial relationship between the outward facing cameras of the head mounted mixed reality display system is based on performing photogrammetry on overlapping image data from the outward facing cameras.

3. The method of claim 1 wherein detecting a change in the first spatial relationship between the outward facing cameras of the head mounted, mixed reality display system is based on motion sensor data indicating different changes in position of one camera from that of another beyond a threshold.

4. The method of claim 1 wherein determining the second spatial relationship between the outward facing cameras based on the change in the first spatial relationship further comprises determining a scale for image data based on a predetermined geometrical measurement of an object in a field of view of the head mounted, mixed reality display system.

5. The method of claim 1 wherein determining the second spatial relationship between the outward facing cameras based on the change in the first spatial relationship further comprises determining a geometrical measurement of a real object in a field of view of the head mounted, mixed reality display system based on depth data.

6. The method of claim 1 wherein the change in the first spatial relationship between the outward facing cameras includes a rotational change.

7. The method of claim 1 wherein the change in the first spatial relationship between the outward facing cameras includes a translational change.

8. A head mounted, mixed reality display system comprising:
a frame including a flexible frame portion positioned in between see-through displays positioned by the frame and positioned in between outward facing, digital image data capturing cameras supported by the frame such that a first spatial relationship between the displays changes when the flexible frame portion flexes and such that a second spatial relationship between the cameras correspondingly changes when the flexible frame portion flexes;
each display being positioned by the frame to be seen by a respective eye;
each outward facing camera being configured for capturing image data of a real object in a field of view of a respective one of the see-through displays and having a fixed spatial relationship with respect to its respective see-through display thus providing for a fixed mapping from position coordinates on an image sensor of the outward facing camera to positions coordinates on the respective see-through display;
a memory operatively coupled to the cameras for storing corresponding image data respectively captured by the cameras of a common real object in the respective fields of views of the cameras, the memory being further operatively coupled to store data representing the second spatial relationship and to store data representing the respective fixed spatial relationship for each outward facing camera and its respective display; and
one or more processors having access to the memory and being communicatively coupled to the cameras and displays, the one or more processors being configured to automatically detect a change in the second spatial relationship of the outward facing cameras relative to each other based on respective image data provided by the outward facing cameras and to automatically update a registration between a position of the common real object and a position of a virtual object responsive to the determined change in the second spatial relationship between the outward facing cameras.

9. The head mounted, mixed reality display system of claim 8 wherein the flexible frame portion is a flexible nose bridge section capable of rotating and translating.

10. The head mounted, mixed reality display system of claim 8 further comprising:
one or more motion sensing units communicatively coupled to the one or more processors for sending data to the one or more processors as bases for automatically detecting a change in the second spatial relationship between the outward facing cameras of the head mounted, mixed reality display.

11. The head mounted mixed reality display system of claim 8 further comprising:
the one or more processors automatically detecting a change in the second spatial relationship between the outward facing cameras based on detecting an area size change in overlapping image data from each of the outward facing cameras.

12. The head mounted, mixed reality display of claim 11 further comprising:
a depth camera for providing depth data of a real object wherein the one or more processors determine a fixed geometrical measurement with respect to the real object for defining a scale for image data; and
the one or more processors perform photogrammetry based on the scale determined using the fixed geometrical measurement.

13. The head mounted, mixed reality display system of claim 8 further comprising:
a network interface communicatively coupled to the one or more processors for accessing a computer system with access to one or more databases of images of objects of standard size over a network;
the one or more processors sending image data from the outward facing cameras to the computer system and requesting an identification of any real object in the image data which matches an object of standard size; and
responsive to receiving an identification of a standard size real object in the image data, the one or more processors perform photogrammetry using a predetermined fixed geometrical measurement of the standard size real object.

14. One or more processor readable storage devices having instructions encoded thereon which when executed cause one or more processors to perform a method of recalibrating a head mounted, mixed reality display system having a flexible frame portion disposed between displays such that a first spatial relationship between the displays changes when the flexible frame portion flexes, the displays respectively including respective see-through displays, the method comprising:
automatically detecting a change in a second spat

Assignees

• Cajigas Elmer S
• O'Brien Paul M
• Tantos Andras
• Microsoft Technology Licensing Llc

Inventors

• Cajigas Elmer S
• O'Brien Paul M
• Tantos Andras

Classifications

• G06F3/012 Primary

  Head tracking input arrangements · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title

• G02B27/017

  Head mounted · CPC title

• G02B27/0172

  characterised by optical features · CPC title

• G02B2027/014

  comprising information/image processing systems · CPC title