Method and system for correcting gaze offset

Patent metadata

Field	Value
Publication number	US-9335820-B2
Application number	US-201514859306-A
Country	US
Kind code	B2
Filing date	Sep 20, 2015
Priority date	May 26, 2010
Publication date	May 10, 2016
Grant date	May 10, 2016

Abstract

A method of correcting gaze offset in an image of at least one individual having eyes is disclosed. The method comprises: processing the image to extract location of at least one eye over the image, processing the image to replace imagery data associated with each location of each eye with replacement data thereby providing a corrected image, and transmitting the corrected image to a display device. The replacement data are preferably previously-recorded imagery data which respectively correspond to the same eye but a different gaze.

First claim

What is claimed is:

1. A method of correcting gaze offset in an image of at least one individual having eyes, comprising:
processing the image so as to extract location of at least one eye over the image;
determining whether an eyelid of said at least one eye is close or open; and
conditionally to a determination that said eyelid is open, processing the image to replace imagery data associated with each location of each eye with replacement data, said replacement data being previously-recorded imagery data which respectively correspond to the same eye but a different gaze, thereby providing a corrected image, and transmitting said corrected image to a display device.

2. The method of claim 1 , further comprising, prior to said extraction of said locations, capturing at least one image of each of the at least one individual at one or more predetermined gazes, thereby providing said previously-recorded data.

3. The method according to claim 1 , further comprising processing said previously-recorded data so as to compute eye parameters wherein said replacing is based on said computed parameters.

4. The method according to claim 3 , further comprising applying a morphological transformation to said previously-recorded data according to said parameters.

5. The method according to claim 3 , further comprising varying said parameters so as to optimize a predetermined objective function describing matching between said replacement data and said imagery data associated with said location of said eye.

6. The method according to claim 1 , further comprising, for each eye, determining a characteristic gaze direction of said eye, wherein said imagery data is replaced with said replacement data only if said gaze direction is other than a predetermined gaze direction.

7. The method according to claim 1 , further comprising, processing the image to replace imagery data associated with each location of each eye with additional replacement data to provide at least one additional corrected image such that each corrected image is characterized by a different gaze, and transmitting each corrected image to a different display device.

8. The method according to claim 7 , further comprising for each eye, determining a characteristic gaze direction of said eye, wherein said at least one additional corrected image is generated responsively to said determined characteristic gaze direction.

9. The method according to claim 1 , wherein said extraction of said locations comprises, for each eye: detecting eye corners, defining a region-of-interest based on said corners and scaling said region-of-interest to predetermined dimension.

10. The method according to claim 1 , wherein said previously-recorded imagery data is stored in a computer-accessible specific database including imagery data corresponding to the eyes of the individual, and wherein said replacement of said imagery data comprises searching over said specific database for said previously-recorded imagery data.

11. The method of claim 10 , wherein said searching comprises comparing imagery data of the image with imagery data entries in said specific database, determining a similarity level responsively to said comparison, and selecting said previously-recorded imagery data based on said similarity level.

12. The method according to claim 1 , wherein said image is a frame of a video stream.

13. The method of claim 12 , wherein said replacement data are selected from a specific database including imagery data corresponding to the eyes of the individual according to an optimization procedure which comprises a first objective function for reducing differences between said replacement data and replacement data of a previous frame, and a second objective function for reducing differences between said replacement data and replacement data of a frame immediately following an eyelid opening of the respective eye.

14. A computer software product, comprising a non-transitory computer-readable medium in which program instructions are stored, which instructions, when read by a processer, cause the processer to process the image so as to extract locations of at least one eye over the image, to determine whether an eyelid of said at least one eye is close or open, to provide a corrected image conditionally to a determination that said eyelid is open by replacing imagery data associated with each location of each eye with replacement data being previously-recorded imagery data respectively corresponding to the same eye but a different gaze, and to transmit said corrected image to a display device.

15. An imaging system, comprising:
(a) a camera for capturing an image of at least one individual having eyes; and
(b) a processer configured for:
processing the image so as to extract locations of at least one eye over the image,
determining whether an eyelid of said at least one eye is close or open; and
conditionally to a determination that said eyelid is open, processing the image to replace imagery data associated with each location of each eye with replacement data, said replacement data being previously-recorded imagery data which respectively correspond to the same eye but a different gaze.

16. The system of claim 15 , wherein said previously-recorded imagery data are processed by the processor so as to compute eye parameters wherein said imagery data are replaced based on said computed parameters.

17. The system according to claim 16 , wherein said previously-recorded data are transformed by said processor using a morphological transformation according to said parameters.

18. The system according to claim 16 , wherein said parameters are varied by said processor so as to optimize a predetermined objective function describing matching between said replaced data and said replacement data.

19. The system according to claim 15 , wherein the image is processed by said processor to determine whether an eyelid of said eye is close or open, and wherein said imagery data is replaced with said replacement data only if said eyelid is open.

20. The system according to claim 15 , wherein the image is processed by said processor to determine a characteristic gaze direction of said eye wherein said imagery data is replaced with said replacement data only if said characteristic gaze direction is other than a predetermined gaze direction.

21. The system according to claim 15 , wherein the image is processed by said processor to replace imagery data associated with each location of each eye with additional replacement data to provide at least one additional corrected image, such that each corrected image is characterized by a different gaze, and transmitting each corrected image to a different display device.

22. The system according to claim 21 , wherein the image is processed by said processor to determine a characteristic gaze direction of said eye, wherein said at least one additional corrected image is generated responsively to said determined characteristic gaze direction.

Assignees

• Univ Ramot

Inventors

• Wolf Lior
• Freund Ziv

Classifications

• H04N7/144 Primary

  camera and display on the same optical axis, e.g. optically multiplexing the camera and display for eye to eye contact · CPC title

• G06F3/013 Primary

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• H04N7/15

  Conference systems · CPC title

• G06K9/00617

  Physics · mapped topic

• G06K9/0061

  Physics · mapped topic