Automated E-tran application

Patent metadata

Field	Value
Publication number	US-10353475-B2
Application number	US-201615284124-A
Country	US
Kind code	B2
Filing date	Oct 3, 2016
Priority date	Oct 3, 2016
Publication date	Jul 16, 2019
Grant date	Jul 16, 2019

Abstract

Techniques for text entry using gestures are disclosed. As disclosed, a camera may capture a frame and the face of the user can be detected therein. Landmarks can be aligned to the face in the captured frame. A left eye image and a right eye image may be extracted from the captured frame. The left eye image and the right image each may be resized and compared to a calibration template. A direction of eye gaze may be determined based upon the comparison. A character or word may be predicted based upon the determination of the direction of eye gaze and a known configuration of an eye gaze board (e.g., an E-tran board). The predicted character or word can be included as a part of a text-based message.

First claim

What is claimed is:

1. A system, comprising:
A rear camera of a mobile device configured to capture at least one frame that includes a face of a user;
a database configured to store the at least one frame;
a processor communicatively coupled to the rear camera and the database, the processor configured to:
detect the face in the captured at least one frame;
extract a left eye image and a right eye image of the face from the at least one frame;
resize the left eye image to a size matching one of a calibrated set of left eye images for the user and the right eye image to a size matching one of a calibrated set of right eye images for the user;
compare the resized left eye image to the calibrated set of left eye images for the user, wherein each image in the calibrated set of left eye images corresponds to a known eye gaze direction for a left eye of the user;
compare the resized right eye image to the calibrated set of right eye images for the user, wherein each image in the calibrated set of right eye images corresponds to a known eye gaze direction for a right eye of the user;
determine a direction of eye gaze based upon the comparison of the resized left eye image to the calibrated set of images for the left eye and the comparison of the resized right eye image to the calibrated set of images for the right eye;
identify, based upon the determination of the direction of eye gaze and a known configuration of an eye gaze board having a plurality of quadrants, a quadrant of the eye gaze board corresponding to the direction of eye gaze, wherein each of the plurality of quadrants of the eye gaze board comprise at least one character; and

display the at least one character from the quadrant as a part of a text-based message.

2. The system of claim 1 , the processor is further configured to generate the calibration template comprising the calibrated set of left eye images and the calibrated set of right eye images, comprising:
receive an indication of eye gaze of the user in a direction;
receiving a second at least one frame of the user that is captured from the rear camera;
detect the face of the user in the second at least one frame;
extract a calibrated left eye image and a calibrated right eye image of the face from the second at least one frame; and
store the calibrated left eye image and the calibrated right eye image to the database as part of the calibration template for the direction of eye gaze.

3. The system of claim 2 , the processor further configured to:
determine lighting condition based on an amount of light detected by the rear camera; and
associate the lighting condition with the calibration template.

4. The system of claim 1 , the processor further configured to:
based upon the at least one character in the quadrant, determine at least one word associated with the at least one character using a predictive text algorithm that determines words based upon frequency of usage.

5. The system of claim 4 , the processor further configured to:
align landmarks to the face in the captured at least one frame.

6. The system of claim 1 , the processor further configured to:
cause an audible or visible indication of the at least one character.

7. The system of claim 1 , the processor further configured to:
receive a gesture from the user; and
perform an action based upon the gesture.

8. The system of claim 7 , wherein the action is to delete the at least one character from the text-based message.

9. The system of claim 1 , the processor further configured to:
convert the left eye image to hue, saturation, and value (HSV) color space and retain only the value portion; and
convert the right eye image to HSV color space and retain only the value portion.

10. The system of claim 1 , the processor further configured to:
receiving a plurality of eye gazes from the user to form a sequence of eye gazes; and
predicting a word based upon the sequence of eye gazes using a predictive text algorithm.

11. A computer-implemented method, comprising:
capturing, by a rear camera of a mobile device, at least one frame that includes a face of a user;
detecting the face in the captured at least one frame;
extracting a left eye image and a right eye image of the face from the at least one frame;
resizing the left eye image to a size matching one of a calibrated set of left eye images for the user and the right eye image to a size matching one of a calibrated set of right eye images for the user;
comparing the resized left eye image to the calibrated set of left eye images for the user, wherein each image in the calibrated set of left eye images corresponds to a known eye gaze direction for a left eye of the user;
comparing the resized right eye image to the calibrated set of right eye images for the user, wherein each image in the calibrated set of right eye images corresponds to a known eye gaze direction for a right eye of the user;
determining a direction of eye gaze based upon the comparison of the resized left eye image to the calibrated set of images for the left eye and the comparison of the resized right eye image to the calibrated set of images for the right eye;
identifying, based upon the determination of the direction of eye gaze and a known configuration of an eye gaze board having a plurality of quadrants, a quadrant of eye gaze board corresponding to the direction of eye gaze, wherein each of the plurality of quadrants of the eye gaze board comprise at least one character; and
displaying the at least one character from the quadrant as a part of a text-based message.

12. The method of claim 11 , further comprising generating the calibration template comprising the calibrated set of left eye images and the calibrated set of right eye images, comprising:
receiving an indication of eye gaze of the user in a direction;
capturing a second at least one frame of the user,
detecting the face of the user in the second at least one frame;
extracting a calibrated left eye image and a calibrated right eye image of the face from the second at least one frame; and
storing the calibrated left eye image and the calibration right eye image as part of the calibration template for the direction of eye gaze.

13. The method of claim 12 , further comprising:
determining lighting condition based on an amount of light detected by the rear camera; and
associating the lighting condition with the calibration template.

14. The method of claim 13 , further comprising:
based upon the at least one character in the quadrant, determining at least one word associated with the at least one character using predictive text algorithm that determines words based upon frequency of usage.

15. The method of claim 14 , further comprising:
receiving a selection of at least one word associated with the at least one character.

16. The method of claim 11 , further comprising:
generating an audible or visible indication of the at least one character.

17. The method of claim 11 , further comprising:
receiving a gesture from the user; and
performing an action based upon the gesture.

18. The method of claim 11 , further comprising:
converting the left eye image to hue, saturation, and value (HSV) color space and retain only the value portion; and
converting the right eye image to HSV color space and retain only the value portion.

19. The method of claim 11 , further comprising:
receiving a plurality of eye gazes from the user to form a sequence of eye gazes; and
predicting a word based upon the sequence of

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Zhang Xiaoyi
• Kulkarni Harish Sripad
• Morris Meredith

Classifications

• G06V10/32

  Normalisation of the pattern dimensions · CPC title

• G06F3/0237

  using prediction or retrieval techniques · CPC title

• G06K9/00604

  Physics · mapped topic

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06K9/42

  Physics · mapped topic