Enhanced detection of sensor misalignments by coordinating sensors utilizing multiple sensing modalities

Patent metadata

Field	Value
Publication number	US-11512952-B1
Application number	US-202117364783-A
Country	US
Kind code	B1
Filing date	Jun 30, 2021
Priority date	Jun 30, 2021
Publication date	Nov 29, 2022
Grant date	Nov 29, 2022

Abstract

The techniques disclosed herein detect sensor misalignments in a display device by the use of sensors operating under different modalities. In some configurations, a near-to-eye display device can include a number of sensors that can be used to track movement of the device relative to a surrounding environment. The device can utilize multiple sensors operating under multiple modalities. For each sensor, there is a set of intrinsic and extrinsic properties that are calibrated. The device is also configured to determine refined estimations of the intrinsic and extrinsic properties at runtime. The refined estimations of the intrinsic and extrinsic properties can then be used to derive knowledge on how the device has deformed over time. The device can then use the refined estimations of the intrinsic and extrinsic properties and/or any other resulting data that quantifies any deformations of the device to make adjustments to rendered images at runtime.

First claim

The invention claimed is:

1. A display device comprising:
at least three sensors affixed to a frame of the display device, wherein individual sensors of the at least three sensors having at least two different sensing modalities;
one or more processors; and
a non-transitory computer-readable storage medium having encoded thereon computer-executable instructions to cause the one or more processors to:
receive signals from the at least three sensors to generate a first set of properties for each of the at least three sensors having at least two different sensing modalities;
receive additional signals during operation of the display device from the at least three sensors to generate a second set of properties for each of the at least three sensors having at least two different sensing modalities;
determine that the first set of properties and the second set of properties meet one or more criteria; and
control a display of a rendered object on a display screen in response to determining that the first set of properties and the second set of properties meet one or more criteria, wherein the control of the display of the rendered object maintains an alignment between the rendered object and at least one real-world object while the frame of the display device subjected to a deformation.

2. The display device of claim 1 , wherein at least two sensors of a first sensing modality are configured to use a triangulation method for determining a first distance measurement of the real-world object, wherein at least one sensor of a second sensing modality is configured to use a time-of-flight method for determining a second distance measurement of a real-world object, wherein the first distance measurement and the second distance measurement are determined at a first time to generate the first set of properties, wherein the first distance measurement and the second distance measurement are determined at a second time to generate the second set of properties.

3. The display device of claim 2 , wherein the first set of properties and the second set of properties meet one or more criteria when the first distance measurement and the second distance measurement determined at the first time and the first distance measurement, and the second distance measurement determined at the second time indicate a deformation of the display device.

4. The display device of claim 1 , wherein the first set of properties and the second set of properties meet one or more criteria when the first distance measurement and the second distance measurement determined at the first time and the first distance measurement and the second distance measurement determined at the second time indicate a deformation of the display device.

5. The display device of claim 1 , wherein control a display of the rendered object maintains the alignment between the rendered object and the at least one real-world object by maintaining a position of the rendered object at a position in a display screen.

6. The display device of claim 1 , wherein control a display of the rendered object maintains the alignment between the rendered object and the at least one real-world object by allowing movement or rotation of the rendered object when the first set of properties and the second set of properties indicate less than a threshold level of deformation.

7. The display device of claim 1 , wherein the first set of properties includes a first error level for a first rotation measurement of the display device, and the second set of properties includes a second error level for second first rotation measurement of the display device, wherein the first set of properties and the second set of properties meet one or more criteria when the first error level is outside of a threshold of the second error level.

8. A method for execution on a display device, comprising:
receiving input signals from at least three sensors to generate a first set of properties for each of the at least three sensors having at least two different sensing modalities;
receiving additional signals during operation of the display device from the at least three sensors to generate a second set of properties for each of the at least three sensors having at least two different sensing modalities;
determining that the first set of properties and the second set of properties meet one or more criteria; and
controlling a display of a rendered object on a display screen in response to determining that the first set of properties and the second set of properties meet one or more criteria, wherein the control of the display of the rendered object compensates for visual anomalies caused by a deformation of the frame of the display device.

9. The method of claim 8 , wherein at least two sensors of a first sensor modality are configured to use a triangulation method for determining a first distance measurement of the real-world object, wherein at least one sensor of a second sensor modality is configured to use a time-of-flight method for determining a second distance measurement of a real-world object, wherein the first distance measurement and the second distance measurement are determined at a first time to generate the first set of properties, wherein the first distance measurement and the second distance measurement are determined at a second time to generate the second set of properties.

10. The method of claim 9 , wherein the first set of properties and the second set of properties meet one or more criteria when the first distance measurement and the second distance measurement determined at the first time and the first distance measurement, and the second distance measurement determined at the second time indicate a deformation of the display device.

11. The method of claim 8 , wherein the at least three sensors include imaging sensors, wherein a first imaging sensor is positioned between a second imaging sensor and a third imaging sensor, wherein the second imaging sensor and the third imaging sensor utilize triangulation to determine a first distance measurement to the real-world object, wherein the first set of properties and the second set of properties meet one or more criteria when images from the at least three sensors indicate a rectification error, wherein controlling the display of the rendered object includes displaying the rendered object at a position that is based on a depth error caused by a deformation of the display device.

12. The method of claim 11 , wherein a direction of a translation of the rendered object within a display screen is based on a direction indicated by a detected pixel with respect to an expected pixel.

13. The method of claim 8 , wherein control a display of the rendered object maintains the alignment between the rendered object and the at least one real-world object by allowing movement or rotation of the rendered object when the first set of properties and the second set of properties indicate less than a threshold level of deformation.

14. The method of claim 8 , wherein the first set of properties includes a first error level for a first rotation measurement of the display device, and the second set of properties includes a second error level for second first rotation measurement of the display device, wherein the first set of properties and the second set of properties meet one or more criteria when the first error level is outside of a threshold of the second error level.

15. A method for correcting a misalignment of a rendered object caused by a deformation of a device, the method configured for execution on the device, comprising:
receive signals from a first inertial measurement unit and a se

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Poulad Navid
• Demaster-Smith Rayna
• Klein Georg
• Margolis Jeffrey Neil

Classifications

• G02B27/0172

  characterised by optical features · CPC title

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G02B2027/0178

  Eyeglass type (eyeglass details G02C) · CPC title

• G02B2027/014

  comprising information/image processing systems · CPC title

• G01D21/02

  Measuring two or more variables by means not covered by a single other subclass · CPC title