Methods, systems, and computer readable media for leveraging user gaze in user monitoring subregion selection systems

Patent metadata

Field	Value
Publication number	US-10165176-B2
Application number	US-201415031760-A
Country	US
Kind code	B2
Filing date	Oct 31, 2014
Priority date	Oct 31, 2013
Publication date	Dec 25, 2018
Grant date	Dec 25, 2018

Abstract

A system for leveraging user gaze in a user monitoring subregion selection system includes a first camera configured to capture an image of a scene. A pattern generator is configured to generate and project a pattern onto a surface of at least one of a user's eyes. A sensor is configured to obtain an image of the pattern reflected from at least one of the user's eyes. In an alternate implementation, the pattern generator may be omitted, and the sensor may be a stereo user facing camera. A gaze estimation and scene mapping module is configured to estimate a gaze direction of the user using the image captured by the user facing sensor and to map the estimated gaze direction to the image of the scene based on a location of an object of interest within the scene. A subregion selection module is configured to select a subregion of the image of the scene based on the mapped user gaze direction.

First claim

What is claimed is:

1. A system for leveraging user gaze in a user monitoring subregion selection system, the system comprising:
a first camera configured to capture an image of a scene;
a sensor configured to obtain an image reflected from one of the user's eyes;
a gaze estimation and scene mapping module configured to estimate a gaze direction of the user using the image captured by the sensor and to map the estimated gaze direction to the image of the scene based on a location of an object of interest within the scene, wherein the gaze estimation and scene mapping module is configured to estimate the gaze direction by identifying, as a visual axis, a ray that originates at a corneal center of the eye and passes through an adjusted pupil center of the eye, and to map the estimated gaze direction to the image of the scene by projecting the ray into an image plane of the first camera to determine a three dimensional point of regard (PoR) of the user; and
a scene subregion selection module configured to select a subregion of the image of the scene based on the mapped user gaze direction.

2. The system of claim 1 comprising an image capture activation module configured to initiate image capture by the first camera in response to a command or gesture from the user.

3. The method of claim 2 wherein the image capture activation module invokes framing of the image of the scene, obtaining of the image, estimation of the gaze direction, selection of the subregion, and the image capture by the first camera in response to a single command or gesture of the user.

4. The system of claim 1 comprising a pattern generator for generating and projecting a pattern onto the user's eye and wherein the image detected by the sensor comprises an image of the pattern reflected from the user's eye.

5. The system of claim 4 wherein the pattern generator comprises a plurality of light emitting diodes (LEDs) spaced from each other to generate and project the pattern, wherein the sensor comprises a second camera for capturing the image of the pattern reflected from the eye of the user, and wherein the gaze estimation and scene mapping module is configured to estimate the gaze direction of the user based on the reflected image.

6. The system of claim 1 wherein the sensor comprises a stereo user facing camera, and wherein the gaze estimation and scene mapping module is configured to estimate the gaze direction without using a pattern projected by a pattern generator.

7. The system of claim 4 wherein the pattern generator comprises a display for displaying and projecting the pattern, wherein the sensor comprises a second camera for detecting the pattern reflected from the eye of the user, and wherein the gaze estimation and scene mapping module is configured to estimate the gaze direction of the user based on the reflected image.

8. The system of claim 1 comprising a scene saliency identification module configured to, based on the estimated gaze direction, identify a portion of the scene at which the user is gazing and predict a current activity of the user.

9. The system of claim 1 wherein the gaze estimation and scene mapping module is configured to estimate the gaze direction of one of the user's eyes.

10. The system of claim 1 wherein the gaze estimation and scene mapping module is configured to estimate the gaze direction of both of the user's eyes.

11. The system of claim 1 wherein the sensor comprises a single camera configured to obtain the image reflected from the eye.

12. The system of claim 1 comprising a housing, wherein the first camera, the sensor, the gaze estimation and scene mapping module, and the scene subregion selection module are located within the housing.

13. The system of claim 12 wherein the housing is configured to be held in at least one of a user's hands.

14. The system of claim 12 comprising a mechanism for mounting the housing to the user's head.

15. The system of claim 1 wherein the first camera comprises a user facing camera of a mobile phone and the sensor comprises a scene facing camera of the mobile phone.

16. The system of claim 1 wherein the first camera comprises a scene facing camera of a mobile phone and the sensor comprises a user facing camera of the mobile phone.

17. The system of claim 1 wherein the gaze estimation and scene mapping module is configured to continually calibrate the mapping between the user gaze direction and the image of the scene.

18. The system of claim 17 wherein the gaze estimation and scene mapping module is configured to perform the continual calibration based on locations of the object of interest in a moving window of images of the scene.

19. A method for leveraging user gaze in a user monitoring subregion selection system, the method comprising:
capturing an image of a scene using a first camera;
detecting, using a sensor, an image reflected from one of the user's eyes;
estimating a gaze direction of the user using the image captured by the sensor, wherein estimating the gaze direction includes identifying, as a visual axis, a ray that originates at a corneal center of the eye and passes through an adjusted pupil center of the eye;
mapping the estimated gaze direction to the image of the scene based on a location of an object of interest within the scene, wherein mapping the gaze direction to the image of the scene includes projecting the ray into an image plane of the first camera to determine a three dimensional point of regard (PoR) of the user; and
selecting a subregion of the image of the scene based on the mapped user gaze direction.

20. The method of claim 19 comprising initiating image capture by the first camera in response to a command or gesture from the user.

21. The method of claim 20 comprising initiating the framing of the image, the obtaining of the image, the estimating of the gaze direction, the selection of the subregion, and the image capture in response to a single command or gesture of the user.

22. The method of claim 19 comprising generating and projecting a pattern onto the user's eye and wherein the image detected by the sensor comprises an image of the pattern reflected from the user's eye.

23. The method of claim 22 wherein generating and projecting the pattern includes utilizing a plurality of light emitting diodes (LEDs) spaced from each other, wherein obtaining the image of the pattern includes utilizing a second camera, and wherein estimating the gaze direction includes utilizing the image detected by the second camera.

24. The method of claim 22 wherein generating and projecting the pattern includes utilizing a display, wherein obtaining the image of the pattern includes utilizing a second camera, and wherein estimating the gaze direction includes utilizing the image detected by the second camera.

25. The method of claim 19 comprising identifying, based on the estimated gaze direction, a portion of the scene at which the user is gazing and predict a current activity of the user.

26. The method of claim 19 wherein estimating the gaze direction includes estimating the gaze direction of one of the user's eyes.

27. The method of claim 19 wherein estimating the gaze direction includes estimating the gaze direction of both of the user's eyes.

28. The method of claim 22 wherein obtaining the image of the pattern reflected from the user's eye comprises utilizing a single

Assignees

• Univ North Carolina Chapel Hill

Inventors

• Frahm Jan-Michael
• Perra David Paul

Classifications

• G06F3/013 Primary

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• H04N23/611

  where the recognised objects include parts of the human body · CPC title

• H04N23/62

  Control of parameters via user interfaces · CPC title

• G06V10/145

  Illumination specially adapted for pattern recognition, e.g. using gratings · CPC title

• G06K9/2036

  Physics · mapped topic