Gaze tracking for a mobile device

Patent metadata

Field	Value
Publication number	US-2016282937-A1
Application number	US-201414361127-A
Country	US
Kind code	A1
Filing date	Jan 24, 2014
Priority date	Jan 24, 2014
Publication date	Sep 29, 2016
Grant date	—

Abstract

A method includes detecting, by a camera of a mobile device, a face of a user of the mobile device. The method includes determining a sequence of video images of the face and applying three dimensional (3D) modeling to the sequence of video images of the face to create a 3D model of the face. The method includes identifying rotational movement of the mobile device. The method includes tracking the face to identify rotational movement of the face based on the 3D model of the face, and adjusting the tracking of the face based on the 3D model of the face and rotational movement of the mobile device.

First claim

1 . A computer-implemented method, the method comprising:
detecting, by a camera of a mobile device, a face of a user of the mobile device; determining a sequence of video images of the face; generating a three dimensional (3D) model of a head of the user based on the sequence of video images of the face; identifying rotational movement of the mobile device; tracking the face to identify rotational movement of the face based on the 3D model of the head; and adjusting the tracking of the face based on the 3D model of the head and rotational movement of the mobile device.

2 . The computer-implemented method of claim 1 , further comprising:
detecting particular rotational movement of the face; and generating commands by the mobile device based on the detected particular rotational movement.

3 . The computer-implemented method of claim 2 , wherein the detected particular rotational movement includes rotational movement around a yaw axis of the head.

4 . The computer-implemented method of claim 2 , wherein implementing the command further comprises:
activating an audio output of the mobile device based on the detected particular rotational movement.

5 . The computer-implemented method of claim 4 , wherein generating the command further comprises:
terminating the audio output based on an additional detected particular rotational movement.

6 . The computer-implemented method of claim 1 , further comprising:
detecting at least one of a particular rotational speed and a particular rotational direction with respect to movement of the face; and generating commands by the mobile device based on the at least one of the detected particular rotational speed and the detected particular rotational direction.

7 . The computer-implemented method of claim 1 , wherein applying 3D modeling to the sequence of video images of the face to create a 3D model of the head further comprises:
generating a point cloud representation of the head based on the sequence of video images of the face.

8 . The computer-implemented method of claim 1 , wherein tracking the face further comprises:
tracking the face to a position at which only one eye is visible.

9 . The computer-implemented method of claim 1 , further comprising:
tracking the face based on selected facial feature points including one or more of the eyes, nose, eyebrows and mouth.

10 . The computer-implemented method of claim 1 , wherein adjusting the tracking of the face based on the 3D model of the head further comprises:
filtering out accidental or incidental movement to stabilize tracking per axis based on tracking of the mobile device and tracking of the face.

11 . The computer-implemented method of claim 1 , further comprising:
combining the sequence of video images of the face with a predetermined model of a head to determine the 3D model of the head.

12 . A mobile device, comprising:
a video camera to receive video images of a user; a memory to store a plurality of instructions; and a processor configured to execute instructions in the memory to:
detect, by the video camera, a face of the user;
determine a sequence of video images of the face;
apply three dimensional (3D) modeling to the sequence of video images of the face to create a 3D model of the face;
identify rotational movement of the mobile device;
track the face to identify rotational movement of the face based on the 3D model of the face; and
adjust the tracking of the face based on the 3D model of the face and rotational movement of the mobile device.

13 . The mobile device of claim 12 , wherein the processor is further configured to:
detect particular rotational movement of the face; and generate commands by the mobile device based on the detected particular rotational movement.

14 . The mobile device of claim 13 , wherein the detected particular rotational movement includes rotational movement around a yaw axis of the face.

15 . The mobile device of claim 13 , further comprising an infrared (IR) camera to receive IR images of the user.

16 . The mobile device of claim 12 , wherein the processor is further configured to:
detect at least one of a particular rotational speed and a particular rotational direction with respect to movement of the face; and generate commands by the mobile device based on the at least one of the detected particular rotational speed and the detected particular rotational direction.

17 . The mobile device of claim 12 , wherein, when applying 3D modeling to the sequence of video images of the face to create a 3D model of the face, the processor is further configured to:
generate a point cloud representation of the face based on the sequence of video images.

18 . The mobile device of claim 12 , wherein, when tracking the face, the processor is further configured to:
track the face to a position at which no eyes are visible.

19 . A non-transitory computer-readable medium including instructions to be executed by a processor, the instructions including one or more instructions, when executed by the processor, for causing the processor to
detect, by a video camera, a face of the user of a mobile device; determine a sequence of video images of the face; apply three dimensional (3D) modeling to the sequence of video images of the face to create a 3D model of the face; identify rotational movement of the mobile device; track the face to identify rotational movement of the face based on the 3D model of the face; and adjust the tracking of the face based on the 3D model of the face and rotational movement of the mobile device.

20 . The non-transitory computer-readable medium of claim 19 , wherein, adjusting the tracking of the face based on the 3D model of the face, the instructions further include instructions for causing the processor to:
filter out accidental or incidental movement to stabilize tracking per axis based on tracking of the mobile device and tracking of the face.

Assignees

• Sony Mobile Communications Inc

Inventors

• Thorn Ola

Classifications

• G06F3/0312

  for tracking the rotation of a spherical or circular member, e.g. optical rotary encoders used in mice or trackballs using a tracking ball or in mouse scroll wheels (tracking relative movement in co-operation with a regularly or irregularly patterned surface, e.g. as in optical mice G06F3/0317; constructional details of scroll or thumb-wheels G06F3/0362; optical rotary encoders G01D5/3473) · CPC title

• G06F3/012

  Head tracking input arrangements · CPC title

• G06T2200/04

  involving 3D image data · CPC title

• G06T2207/30201

  Face · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title