Systems and methods for retinal imaging and tracking

Patent metadata

Field	Value
Publication number	US-11828946-B2
Application number	US-202217901326-A
Country	US
Kind code	B2
Filing date	Sep 1, 2022
Priority date	Mar 20, 2020
Publication date	Nov 28, 2023
Grant date	Nov 28, 2023

Abstract

A head mounted display system configured to project light to an eye of a user to display augmented reality images can include a frame configured to be supported on a head of the user, a camera disposed temporally on said frame, an eyepiece configured to direct light into said user's eye to display augmented reality image content to the user's vision field, a reflective element disposed on the frame, and at least one VCSEL disposed to illuminate said eye, wherein the camera is disposed with respect to the reflective element such that light from the VCSEL is reflected from the user's eye to the reflective element and is reflected from the reflective element to the camera to form images of the eye that are captured by the camera.

First claim

What is claimed:

1. A display system comprising:
a frame configured to be wearable on a head of a user;
at least one image projector supported by the frame, the at least one image projector configured to project augmented reality image content;
an eyepiece supported by the frame, the eyepiece configured to receive the augmented reality image content from the at least one image projector and direct the augmented reality image content to an eye of the user;
at least one light source configured to direct light to the eye;
at least one reflective element disposed on at least one of the frame or the eyepiece, wherein the at least one reflective element comprises a segmented optical element;
at least one camera supported by the frame on at least one temple side of the head of the user, the at least one camera arranged to receive a portion of the light that is reflected from the eye to the at least one reflective element and that is reflected from the at least one reflective element to the at least one camera to form at least one image of the eye that is captured by the at least one camera;
processing electronics in communication with the at least one image projector; and
memory storing instructions that, when executed by the processing electronics, cause the processing electronics to performing operations comprising
receiving the at least one image of the eye that is captured by the at least one camera,
estimating a location of at least one feature of the eye based on the at least one image, wherein the at least one feature includes a retina of the eye, and
determining a gaze direction of the eye based on assessing where at least part of the retina captured by the at least one camera fits within a retinal map, wherein determining the gaze direction comprises
defining a bounding region within a first image of the at least one image, wherein the first image contains the at least one feature,
identifying a center of mass of the bounding region,
determining a 3D position of a pupil center and an eye ball center based on the center of mass of the bounding region, and
determining the gaze direction based on a coordinate system having an origin defined as a location of the eye ball center, wherein the gaze direction passes through the pupil center.

2. The display system of claim 1 , wherein the at least one light source includes at least one vertical-cavity surface-emitting laser (VCSEL).

3. The display system of claim 1 , wherein at least a portion of the eyepiece is transparent to visible light, to provide the user a view of the environment in front of the user while the frame is being worn by the user.

4. The display system of claim 1 , wherein the at least one reflective element is transparent to visible light.

5. The display system of claim 1 , wherein the at least one light source comprises an infrared light source that outputs more of the light as infrared light than visible light.

6. The display system of claim 1 , wherein the at least one light source is supported by the frame on at least one temple side of the head of the user.

7. The display system of claim 1 , wherein the at least one light source is mounted on the eyepiece.

8. The display system of claim 1 , wherein the at least one light source is mounted on a side of the eyepiece that is proximal to the eye.

9. The display system of claim 1 , wherein the at least one light source is mounted on a side of the eyepiece that is distal to the eye.

10. The display system of claim 1 , wherein the at least one light source is integrated with the at least one camera.

11. The display system of claim 1 , wherein the at least one light source is configured to output polarized light.

12. The display system of claim 1 , further comprising electronics configured to modulate the at least one light source to reduce speckle.

13. The display system of claim 1 , further comprising at least one diffractive optical element having a chromatic dispersion that counters at least a portion of the chromatic aberration introduced by the at least one reflective element.

14. The display system of claim 1 , wherein the eyepiece is configured to direct the augmented reality image content to the eye of the user such that the augmented reality image content appears to originate from different depths.

15. The display system of claim 1 , wherein the eyepiece comprises one or more waveguides.

16. The display system of claim 1 , wherein the location of the at least one feature is estimated in three-dimensional space.

17. The display system of claim 1 , wherein the at least one feature includes one or more of a pupil of the eye.

18. The display system of claim 1 , wherein the location is a center of a pupil of the eye.

19. The display system of claim 1 , wherein the operations further comprise producing the retinal map based on a plurality of retinal images obtained for different eye gazes.

20. The display system of claim 1 , wherein the retinal map is produced in part by displaying a plurality of targets to the user's eye, wherein the targets are located at different positions to cause the user's eye to alter gaze direction, wherein different gaze directions of the user's eye are correlated to images of the retina obtained by the at least one camera at the different gaze directions, respectively.

21. The display system of claim 1 , wherein the operations further comprises producing a gaze matrix based on correlating images of the user's retina with gaze directions of the user's eye.

22. The display system of claim 1 , wherein the location of the at least one feature is estimated based on at least a first image of the retina and a second image of the retina captured by the at least one camera, wherein the first image of the retina and the second image of the retina are obtained at different perspectives.

23. The display system of claim 1 , wherein the at least one camera comprises a first camera and a second camera, wherein the location of the at least one feature is estimated based on at least a first image and a second image of the retina captured by the first camera and the second camera, respectively.

24. The display system of claim 1 , wherein the at least one camera comprises a first camera and a second camera, wherein the location of the at least one feature is estimated based on a plurality of first images of the retina captured by the first camera and a plurality of second images of the retina captured by the second camera.

25. The display system of claim 1 , wherein the location of the at least one feature is estimated based on a determination of a location of blood vessels of the retina.

26. The display system of claim 25 , wherein the location of the at least one feature is estimated based on a determination of the location of the blood vessels of the retina relative to a reference location.

27. The display system of claim 1 , wherein the location of the at least one feature is based on a determination of a location of a spot in the image of the eye captured by the at least one camera.

28. The display system of claim 1 , wherein the operations further comprise determining a center of mass or centroid of the spot in the image of the eye captured by the at least one camera.

29. The display system of claim 1 , wherein the segmented optical element comprises a segmented diffractive or holographic optical

Assignees

• Magic Leap Inc

Inventors

• Meitav Nizan
• Freedman Barak
• Weinstein Ori
• Pellman Asaf
• Oh Chulwoo
• Montfort Frédéric

Classifications

• G02B27/0172 Primary

  characterised by optical features · CPC title

• G02B27/0093 Primary

  with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking · CPC title

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06T7/73

  using feature-based methods · CPC title

• H04N23/56

  provided with illuminating means · CPC title