Realtime lens aberration correction from eye tracking

Patent metadata

Field	Value
Publication number	US-10502959-B2
Application number	US-201916351466-A
Country	US
Kind code	B2
Filing date	Mar 12, 2019
Priority date	Sep 30, 2014
Publication date	Dec 10, 2019
Grant date	Dec 10, 2019

Abstract

Systems and methods include receiving an image for presenting on a display screen of a head mounted display (HMD). The image is provided by an application. The received image is pre-distorted to enable optics provided in a HMD to render the image. An alignment offset is identified for an eye of a user wearing the HMD by determining a position of the eye relative to an optical axis of at least one lens of the optics of the HMD. The pre-distorted image provided by the application is adjusted to define a corrected pre-distorted image that accounts for the alignment offset. The corrected pre-distorted image is forwarded to the display screen of the HMD for rendering, such that the image presented through the optics of the HMD removes aberrations caused by the alignment offset.

First claim

The invention claimed is:

1. A method, comprising:
a) identifying a position of an eye of a user wearing a head mounted display (HMD), the position of the eye used in determining an alignment offset of the eye relative to an optical axis of at least one lens of optics provided in the HMD;
b) predicting a shift in the position of the eye of the user based on media content that is being rendered on a display screen of the HMD, wherein image of the media content is pre-distorted to enable the optics of the HMD to render the image;
c) selecting pre-defined distortion maps to determine a specific distortion correction for the alignment offset, the pre-defined distortion maps are selected to correspond with a location where the position of the eye is predicted to shift;
d) adjusting the image of the media content that is pre-distorted by applying the specific distortion correction to generate a corrected image that accounts for the alignment offset, wherein an amount of adjusting for the alignment offset is determined from the one or more pre-defined distortion maps selected for a shape and a type of lens used in the optics of the HMD; and
e) sending the corrected image for rendering on the display screen of the HMD when the shift in the position of the eye of the user is detected, such that the image when viewed through the optics is devoid of aberration caused by the alignment offset.

2. The method of claim 1 , wherein selecting the pre-defined distortion maps further includes performing a linear interpolation of data from the pre-defined distortion maps selected for the location where the position of the eye is predicted to shift, to generate a distortion adjustment map in substantial real-time, the distortion adjustment map used in determining the specific distortion correction.

3. The method of claim 1 , wherein the adjusting of the image is done in substantial real-time, based on the predicted shift.

4. The method of claim 1 , wherein the pre-defined distortion maps are two dimensional maps.

5. The method of claim 1 , wherein the pre-defined distortion maps are generated by performing optical simulation for each type and shape of optics used in the HMD.

6. The method of claim 5 , wherein the pre-defined distortion maps are defined for each color in a primary color spectrum used in rendering the image.

7. The method of claim 1 , wherein the pre-defined distortion maps provide mapping of different locations within a visible field of view of the display screen of the HMD to corresponding alignment offsets.

8. The method of claim 1 , wherein operations b) through e) are performed during rendering of frames of the media content included in the image.

9. The method of claim 1 , wherein operations b) through e) are performed at a frame rate of frames of the media content rendered at the HMD.

10. The method of claim 1 , wherein identifying the alignment offset includes identifying one of an x-shift, or a y-shift, or a z-shift, or an x-and-y shift, or an x-and-z shift, or a y-and-z shift, or an x-y-z shift.

11. The method of claim 1 , wherein the position of the eye is captured using one or more internal sensors disposed on the HMD.

12. The method of claim 1 , wherein the shift in the position of the eye of the user is predicted based on an activity occurring in an application providing the media content and the image of the media content rendered at the HMD captures the activity.

13. The method of claim 1 , wherein identifying the alignment offset includes tracking the position of the eye in substantial real-time as the image is being presented on the display screen of the HMD.

14. A head mounted display device, comprising:
a lens of optics for viewing a pre-distorted image of multimedia content rendered on a display screen of the head mounted display (HMD);
an eye position sensor configured to,
(a) determine position of an eye of a user wearing the HMD in relation to an optical axis of the lens of optics provided in the HMD;
(b) determine alignment offset of the eye of the user; and

an optics distortion processor configured to,
(c) predict a shift in the position of the eye of the user based on the multimedia content being rendered on the display screen;
(d) select pre-defined distortion maps for a type and shape of lens used in the optics, the pre-defined distortion maps selected correspond with a location where the position of the eye is predicted to shift and are used to determine a specific distortion correction for the alignment offset;
(e) adjust the pre-distorted image of multimedia content provided for rendering on the display screen of the HMD by applying the specific distortion correction to generate a corrected image that accounts for the alignment offset, wherein an amount of adjusting for the alignment offset is determined from the pre-defined distortion maps, wherein the corrected image of the multimedia content is transmitted to the display screen of the HMD for rendering when the shift in the position of the eye of the user is detected.

15. The head mounted display device of claim 14 , wherein the eye position sensor includes a position tracker to track the position of the eye of the user so as to perform adjustment to the pre-distorted image in substantial real-time based on tracking information provided by the position tracker.

16. The head mounted display device of claim 14 , wherein the pre-defined distortion maps are generated based on optical simulation for each type and shape of lens of the optics used in the HMD.

17. The head mounted display device of claim 14 , wherein the shift in the position of the eye is predicted based on an activity occurring in an application providing the multimedia content and the image of the multimedia content rendered at the HMD captures the activity.

Assignees

• Sony Interactive Entertainment Inc

Inventors

• Stafford Jeffrey Roger
• Larsen Eric

Classifications

• G02B27/0093

  with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking · CPC title

• G02B2027/011

  comprising device for correcting geometrical aberrations, distortion · CPC title

• G02B2027/014

  comprising information/image processing systems · CPC title

• G02B2027/0187

  slaved to motion of at least a part of the body of the user, e.g. head, eye · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title