Building saccade models for predicting a landing point of a saccade experienced by a player

Patent metadata

Field	Value
Publication number	US-11947719-B2
Application number	US-202217968706-A
Country	US
Kind code	B2
Filing date	Oct 18, 2022
Priority date	May 17, 2018
Publication date	Apr 2, 2024
Grant date	Apr 2, 2024

Abstract

A method for predicting eye movement in a head mounted display (HMD). The method including tracking movement of an eye of a user with a gaze tracking system disposed in the HMD at a plurality of sample points. The method including determining velocity of the movement based on the movement of the eye. The method including determining that the eye of the user is in a saccade upon the velocity reaching a threshold velocity. The method including predicting a landing point on the display of the HMD corresponding to a direction of the eye for the saccade.

First claim

What is claimed is:

1. A method, comprising:
receiving eye orientation data of a plurality of test subjects playing a plurality of video games;
training a machine learning engine using the eye orientation data to build a plurality of saccade models of a plurality of saccades experienced by the plurality of test subjects while playing the plurality of video games; and
generating a plurality of saccade recognition algorithms for the plurality of saccade models, wherein the plurality of saccade recognition algorithms is used to classify input data of a saccade experienced by a player as one of the plurality of saccade models,
wherein each of the plurality of saccade recognition algorithms is configured to predict a landing point of the saccade based on the input data of the saccade.

2. The method of claim 1 , further comprising:
providing the input data of the saccade to a classifier configured to execute the plurality of saccade recognition algorithms implemented within a recurrent neural network.

3. The method of claim 2 ,
wherein the recurrent neural network implementing the classifier is configured as at least one of:
a long short term memory (LSTM) network; and
a fully connected multiplayer perceptron network.

4. The method of claim 1 ,
wherein the input data includes first eye orientation data of an eye of the player occurring over a plurality of sample points.

5. The method of claim 1 ,
wherein the eye orientation data corresponds to a plurality of eyes of the plurality of test subjects.

6. The method of claim 1 , wherein the training of the machine learning engine includes:
building a plurality of modeled velocity graphs for the plurality of saccade models.

7. The method of claim 1 ,
wherein the eye orientation data includes velocity, gaze direction, and time for a corresponding sample point of a corresponding saccade in the plurality of saccades.

8. A non-transitory computer-readable medium storing a computer program for a method, the computer-readable medium comprising:
program instructions for receiving eye orientation data of a plurality of test subjects playing a plurality of video games;
program instructions for training a machine learning engine using the eye orientation data to build a plurality of saccade models of a plurality of saccades experienced by the plurality of test subjects while playing the plurality of video games; and
program instructions for generating a plurality of saccade recognition algorithms for the plurality of saccade models, wherein the plurality of saccade recognition algorithms is used to classify input data of a saccade experienced by a player as one of the plurality of saccade models,
wherein each of the plurality of saccade recognition algorithms is configured to predict a landing point of the saccade based on the input data of the saccade.

9. The non-transitory computer-readable medium of claim 8 , further comprising:
program instructions for providing the input data of the saccade to a classifier configured to execute the plurality of saccade recognition algorithms implemented within a recurrent neural network.

10. The non-transitory computer-readable medium of claim 9 ,
wherein in the method, the recurrent neural network implementing the classifier is configured as at least one of:
a long short term memory (LSTM) network; and
a fully connected multiplayer perceptron network.

11. The non-transitory computer-readable medium of claim 8 ,
wherein in the method, the input data includes first eye orientation data of an eye of the player occurring over a plurality of sample points.

12. The non-transitory computer-readable medium of claim 8 ,
wherein in the method, the eye orientation data corresponds to a plurality of eyes of the plurality of test subjects.

13. The non-transitory computer-readable medium of claim 8 , wherein the program instructions for training the machine learning engine includes:
program instructions for building a plurality of modeled velocity graphs for the plurality of saccade models.

14. The non-transitory computer-readable medium of claim 8 ,
wherein in the method, the eye orientation data includes velocity, gaze direction, and time for a corresponding sample point of a corresponding saccade in the plurality of saccades.

15. A computer system, comprising:
a processor; and
memory coupled to the processor and having stored therein instructions that, if executed by the computer system, cause the computer system to execute a method, comprising:
receiving eye orientation data of a plurality of test subjects playing a plurality of video games;
training a machine learning engine using the eye orientation data to build a plurality of saccade models of a plurality of saccades experienced by the plurality of test subjects while playing the plurality of video games; and
generating a plurality of saccade recognition algorithms for the plurality of saccade models, wherein the plurality of saccade recognition algorithms is used to classify input data of a saccade experienced by a player as one of the plurality of saccade models,
wherein each of the plurality of saccade recognition algorithms is configured to predict a landing point of the saccade based on the input data of the saccade.

16. The computer system of claim 15 , the method further comprising:
providing the input data of the saccade to a classifier configured to execute the plurality of saccade recognition algorithms implemented within a recurrent neural network.

17. The computer system of claim 16 ,
wherein in the method, the recurrent neural network implementing the classifier is configured as at least one of:
a long short term memory (LSTM) network; and
a fully connected multiplayer perceptron network.

18. The computer system of claim 15 ,
wherein in the method, the input data includes first eye orientation data of an eye of the player occurring over a plurality of sample points,
wherein in the method, the eye orientation data corresponds to a plurality of eyes of the plurality of test subjects.

19. The computer system of claim 15 , wherein in the method, the training of the machine learning engine includes:
building a plurality of modeled velocity graphs for the plurality of saccade models.

20. The computer system of claim 15 ,
wherein in the method, the eye orientation data includes velocity, gaze direction, and time for a corresponding sample point of a corresponding saccade in the plurality of saccades.

Assignees

• Sony Interactive Entertainment Inc

Inventors

• Young Andrew
• Rico Javier Fernandez

Classifications

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0499

  Feedforward networks · CPC title

• G06F3/013 Primary

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title