In-cell gaze tracking for near-eye display

Patent metadata

Field	Value
Publication number	US-9703374-B1
Application number	US-201514971477-A
Country	US
Kind code	B1
Filing date	Dec 16, 2015
Priority date	Dec 16, 2015
Publication date	Jul 11, 2017
Grant date	Jul 11, 2017

Abstract

A near-eye display device includes a display panel with an array of photon-emitting cells interspersed with photon-detecting cells and a display controller coupled to the display panel, the display controller to control the display panel to display imagery using the array of photon-emitting cells. The device further includes a camera controller coupled to the display panel, the camera controller to control the display panel to capture imagery of an eye of a user using the photon-detecting cells. The device also includes an eye-tracking module coupled to the camera controller, the eye-tracking module to construct a three-dimensional representation of the eye based on the captured imagery.

First claim

What is claimed is:

1. A near-eye display device comprising:
a display panel comprising an array of photon-emitting cells interspersed with photon-detecting cells;
a display controller coupled to the display panel, the display controller to control the display panel to display imagery using the array of photon-emitting cells; and
a camera controller coupled to the display panel, the camera controller to control the display panel to capture imagery of an eye of a user using the photon-detecting cells; and
an eye-tracking module coupled to the camera controller, the eye-tracking module to construct a three-dimensional representation of the eye based on the captured imagery.

2. The near-eye display device of claim 1 , further comprising:
a set of one or more infrared (IR) light sources disposed at corresponding positions in association with the display panel, the set of one or more IR light sources to illuminate the eye during the capture of the imagery of the eye.

3. The near-eye display device of claim 2 , wherein:
the camera controller is to activate the set of one or more IR light sources to illuminate the eye during a vertical synchronization period of the display panel for the array of photon-emitting cells.

4. The near-eye display device of claim 2 , wherein:
the eye-tracking module is to construct the three-dimensional representation of the eye based on a time-of-flight analysis.

5. The near-eye display device of claim 2 , wherein:
the set of one or more IR light sources comprises a set of one or more vertical-cavity surface-emitting lasers.

6. The near-eye display device of claim 1 , wherein:
the photon-detecting cells are arranged into a first subset and a second subset, wherein each photon-detecting cell of the first subset is paired with a corresponding photon-detecting cell of the second subset;
the camera controller is to concurrently capture a first image of the eye using the photon-detecting cells of the first subset and to capture a second image of the eye using the photon-detecting cells of the second subset; and
the eye-tracking module is to determine the three-dimensional representation of the eye using parallax-based analysis of the first and second images.

7. The near-eye display device of claim 6 , wherein:
the display panel further comprises a microlens disposed over each photon-detecting cell of at least some of the photon-detecting cells of the display panel.

8. The near-eye display device of claim 1 , wherein:
the eye-tracking module further is to determine, based on the three-dimensional representation of the eye, at least one of: a presence of the eye; a position of an eyelid of the eye; a position of the eye; an orientation of the eye; and a gaze direction of the eye.

9. The near-eye display device of claim 8 , wherein:
at least one component of the near-eye display device is controlled based on the determined presence of the eye, position of the eyelid of the eye, position of the eye, orientation of the eye, or gaze direction of the eye.

10. The near-eye display device of claim 1 , wherein the near-eye display device is a head-mounted display (HMD) device.

11. In a near-eye display device having a display panel comprising an array of photon-emitting cells interspersed with photon-detecting cells, a method comprising:
controlling, using a display controller of the device, the display panel to display imagery using the array of photon-emitting cells; and
controlling, using a camera controller of the device, the display panel to capture imagery of an eye of a user using the photon-detecting cells; and
constructing, using an eye-tracking module of the device, a three-dimensional representation of the eye based on the captured imagery.

12. The method of claim 11 , further comprising:
activating a set of one or more infrared (IR) light sources disposed at corresponding positions in association with the display panel so as to illuminate the eye during the capture of the imagery of the eye.

13. The method of claim 12 , wherein:
activating the set of one or more IR light sources comprises activating the set of one or more IR light sources during a vertical synchronization period of the display panel for the array of photon-emitting cells.

14. The method of claim 12 , wherein:
constructing the three-dimensional representation of the eye comprises constructing the three-dimensional representation of the eye based on a time-of-flight analysis using the photon-detecting cells.

15. The method of claim 11 , wherein:
the photon-detecting cells are arranged into a first subset and a second subset, wherein each photon-detecting cell of the first subset is paired with a corresponding photon-detecting cell of the second subset;
controlling the display panel to capture imagery of the eye comprises controlling the display panel to capture a first image of the eye using the photon-detecting cells of the first subset and to capture a second image of the eye using the photon-detecting cells of the second subset; and
constructing the three-dimensional representation of the eye comprises constructing the three-dimensional representation of the eye using parallax-based processing of the first and second images.

16. The method of claim 11 , further comprising:
determining, using the eye-tracking module, at least one of: a presence of the eye; a position of an eyelid of the eye; a position of the eye; an orientation of the eye; and a gaze direction of the eye based on the three-dimensional representation of the eye.

17. The method of claim 16 , further comprising:
controlling at least one component of the near-eye display device based on the determined presence of the eye, position of the eyelid of the eye, position of the eye, orientation of the eye, or gaze direction of the eye.

18. In a near-eye display device, a method comprising:
displaying imagery to an eye of a user via a set of photon-emitting cells of a display panel of the device;
capturing imagery of the eye via a set of photon-detecting cells interspersed among the photon-emitting cells of the display panel;
determining a gaze characteristic of the eye based on the captured imagery; and
controlling an operation of the device based on the gaze characteristic.

19. The method of claim 18 , wherein:
determining the gaze characteristic comprises determining at least one of: a presence of the eye and a position of an eyelid of the eye; and
controlling an operation of the device comprises at least one of: activating or deactivating the display panel based on at least one of the presence of the eye and the position of the eyelid.

20. The method of claim 18 , wherein:
determining the gaze characteristic comprises determining a gaze direction of the eye; and
controlling an operation of the device comprises modifying the imagery displayed to the eye of the user based on the gaze direction.

21. The method of claim 18 , wherein:
determining the gaze characteristic comprises determining a three-dimensional representation of the eye based on the captured imagery.

22. The method of claim 21 , wherein:
determining the three-dimensional representation of the eye comprises determining the three-dimensional representation of the eye based on time-of-flight analysis of the captured imagery.

23. The method of claim 21 , wherein:
capturing imagery of the eye comprises concurrently capturing a first image of the eye using a first subset of the photon-detect

Assignees

• Google Inc
• Google Inc

Inventors

• Zhang Zhibin

Classifications

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title

• G06T7/20

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title

• G02B27/017

  Head mounted · CPC title

• G02B2027/0187

  slaved to motion of at least a part of the body of the user, e.g. head, eye · CPC title

• G02B27/0172

  characterised by optical features · CPC title