Compensation for deformation in head mounted display systems

Patent metadata

Field	Value
Publication number	US-11200646-B2
Application number	US-202017129645-A
Country	US
Kind code	B2
Filing date	Dec 21, 2020
Priority date	Jan 15, 2020
Publication date	Dec 14, 2021
Grant date	Dec 14, 2021

Abstract

The systems and methods described can include approaches to calibrate head-mounted displays for improved viewing experiences. Some methods include receiving data of a first target image associated with an undeformed state of a first eyepiece of a head-mounted display device; receiving data of a first captured image associated with deformed state of the first eyepiece of the head-mounted display device; determining a first transformation that maps the first captured image to the image; and applying the first transformation to a subsequent image for viewing on the first eyepiece of the head-mounted display device.

First claim

What is claimed is:

1. A computer-implemented method comprising:
transmitting data of a first target image that is associated with an undeformed state of a first eyepiece of a head-mounted display device to a first projector that is optically coupled to the first eyepiece;
capturing the data of the first target image that is associated with the undeformed state of the first eyepiece of the head-mounted display device by a first imaging sensor that is optically coupled to the first projector and that is part of the head-mounted display device;
receiving data of a first captured image that is associated with deformed state of the first eyepiece of the head-mounted display device, the first captured image representing a transformed state of the first target image and being received by the first imaging sensor that is optically coupled to the first projector of the first eyepiece;
determining a first transformation that maps the first captured image to the first target image; and
applying the first transformation to a subsequent image for viewing on the first eyepiece of the head-mounted display device.

2. The computer-implemented method of claim 1 , further comprising:
transmitting the transformed subsequent image to the first projector optically coupled to the first eyepiece.

3. The computer-implemented method of claim 1 , further comprising:
receiving a temperature measurement of the first imaging sensor; and
accounting for the received temperature measurement in the first transformation.

4. The computer-implemented method of claim 1 , further comprising:
receiving data from one or more sensors representing a state of a real-world environment external to the head-mounted display device,
wherein the first transformation aligns features of the real-world environment with corresponding features from the first target image.

5. The computer-implemented method of claim 1 , wherein the first transformation aligns each individual color channel of the first captured image with each individual color channel of the first target image.

6. The computer-implemented method of claim 1 , further comprising:
receiving a trigger signal,
wherein determining the first transformation is responsive to receiving the trigger signal.

7. The computer-implemented method of claim 1 , wherein the first eyepiece of the head-mounted display device is optically transparent and wherein the first transformation accounts for a relative position of the first imaging sensor to the first eyepiece.

8. The computer-implemented method of claim 1 , further comprising:
receiving data of a second target image associated with an undeformed state of a second eyepiece of the head-mounted display device, the second eyepiece comprising a second projector that is optically coupled to the second eyepiece;
receiving data of a second captured image associated with deformed state of the second eyepiece, the second captured image representing a transformed state of the second target image and is received by a second imaging sensor optically coupled to a second projector of the second eyepiece, the second imaging sensor being part of the head-mounted display device;
determining a second transformation that maps the second captured image to the second target image; and
applying the second transformation to a subsequent image for viewing on the second eyepiece of the head-mounted display device.

9. The computer-implemented method of claim 8 , wherein the first transformation is dependent on the second transformation and wherein the second transformation aligns subsequent images for viewing on the second eyepiece with subsequent images for viewing on the first eyepiece.

10. The computer-implemented method of claim 8 , wherein the first imaging sensor is rigidly connected to the second imaging sensor and wherein the first eyepiece and the second eyepiece are deformable relative to the first imaging sensor and the second imaging sensor.

11. The computer-implemented method of claim 8 , wherein the second transformation accounts for a relative position of the second imaging sensor relative to the second eyepiece.

12. The computer-implemented method of claim 8 , further comprising:
receiving a temperature measurement of the second imaging sensor; and
accounting for the received temperature measurement in the second transformation.

13. A head-mounted display device comprising:
a wearable frame;
a first eyepiece elastically mounted to the wearable frame, the first eyepiece comprising a first projector rigidly and optically coupled to the first eyepiece, the first projector being configured to emit light to the first eyepiece;
a first imaging sensor elastically mounted to the wearable frame and configured to capture emitted light from the first eyepiece;
a processor communicatively coupled to the first projector and the first imaging sensor, the processor configured to:
transmit data of a first target image that is associated with an undeformed state of the first eyepiece of the head-mounted display device to the first projector that is optically coupled to the first eyepiece;
capture the data of the first target image that is associated with the undeformed state of the first eyepiece of the head-mounted display device by the first imaging sensor that is optically coupled to the first projector and that is part of the head-mounted display device;
receive data from the first imaging sensor of a first captured image representing a transformed state of the first target image;
determine a first transformation that maps the first captured image to the captured data of the first target image; and
apply the first transformation to a subsequent image for viewing on the first eyepiece of the head-mounted display device.

14. The head-mounted display device of claim 13 , further comprising:
one or more sensors associated mounted to the wearable frame and configured to receive data regarding a state of a real-world environment external to the head-mounted display device, wherein the first transformation aligns features of the real-world environment with corresponding features from the first target image.

15. The head-mounted display device of claim 13 , further comprising:
a temperature sensor configured to measure a temperature of the first imaging sensor, and wherein the first transformation is based, at least in part, on the measured temperature of the first imaging sensor, and
a trigger sensor configured to transmit a signal to the processor for initiating the process performed by the processor,
wherein the first eyepiece is optically transparent.

16. The head-mounted display device of claim 13 , further comprising:
a second eyepiece elastically mounted to the wearable frame, the second eyepiece comprising a second projector rigidly and optically coupled to the second eyepiece, the second projector being configured to emit light to the second eyepiece;
a second imaging sensor elastically mounted to the wearable frame and configured to capture emitted light from the second eyepiece, and
wherein the processor is further configured to:
transmit data of a second target image to the second projector;
receive data from the second imaging sensor of a second captured image representing a transformed state of the second target image;
determine a second transformation that maps the second captured image to the second target image; and
apply the second transformation to a subsequent image for viewing on the second eyepiece of the head-mounted display device.

17. The head-mounted display device of claim 16

Assignees

• Magic Leap Inc

Inventors

• Edwin Lionel Ernest
• Miller Samuel A
• Grossmann Etienne Gregoire
• Clark Brian Christopher
• Johnson Michael Robert
• Zhao Wenyi
• Shah Nukul Sanjay
• Huang Po-Kang

Classifications

• H04N23/54

  Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils · CPC title

• H04N13/344 Primary

  with head-mounted left-right displays · CPC title

• G02B2027/0178

  Eyeglass type (eyeglass details G02C) · CPC title

• G02B2027/014

  comprising information/image processing systems · CPC title

• G02B2027/011

  comprising device for correcting geometrical aberrations, distortion · CPC title