Display apparatus and method for controlling the same according to an eye gaze and a gesture of a user

Patent metadata

Field	Value
Publication number	US-10114463-B2
Application number	US-201514872865-A
Country	US
Kind code	B2
Filing date	Oct 1, 2015
Priority date	Oct 1, 2014
Publication date	Oct 30, 2018
Grant date	Oct 30, 2018

Abstract

A display apparatus is provided. The display apparatus includes a communication interface unit configured to receive eye gaze information of a user and user gesture information from an external apparatus; a display unit configured to display a preset screen; and a controlling unit configured to control the display unit so that a preset object is displayed on the preset screen if it is determined that an eye gaze of the user is directed toward the preset screen using the received eye gaze information and a user gesture corresponding to the received gesture information is sensed.

First claim

What is claimed is:

1. A display apparatus, comprising:
a communication circuit;
a display; and
a processor configured to:
control the communication circuit to receive eye gaze information of a user and user gesture information from an external apparatus,
control the display to display a user interface (UI) element for receiving a user input based on an eye gaze and a gesture in response to determining that an eye gaze of the user toward the display and a user gesture are simultaneously detected based on the received eye gaze information and the received gesture information, and
if any one of the eye gaze of the user directed toward the display and the user gesture is not recognized for at least a preset time after the UI element is displayed, control the display to terminate the display of the UI element.

2. The display apparatus as claimed in claim 1 , wherein the processor is further configured to detect the eye gaze of the user directed toward the display.

3. The display apparatus as claimed in claim 2 , wherein if it is determined that the detected eye gaze does not correspond to a position on which the UI element is displayed and a preset user gesture is recognized, the processor is further configured to correct the detected eye gaze to a position on which a second UI element closest to the detected eye gaze is displayed.

4. The display apparatus as claimed in claim 2 , wherein the processor is further configured to perform recognition of the user based on user input information and perform eye gaze detection according to the recognition of the user.

5. The display apparatus as claimed in claim 2 , wherein the processor is further configured to determine a viewing state of the user from information received from the external apparatus and perform eye gaze detection according to the determined viewing state.

6. The display apparatus as claimed in claim 1 , wherein if it is determined that the eye gaze of the user is directed toward the displayed UI element and a preset user gesture is recognized, the processor is further configured to perform an operation corresponding to the UI element toward which the eye gaze of the user is directed and the preset user gesture.

7. The display apparatus as claimed in claim 1 , wherein the processor is further configured to:
control the display to display a plurality of UI elements including the UI element, and
in response to determining that the eye gaze of the user is directed toward the UI element, control the display to display the UI element differently from remaining UI elements of the plurality of UI elements.

8. The display apparatus as claimed in claim 1 , wherein if it is determined that the eye gaze of the user is directed toward the UI element, the processor is further configured to vary the UI element and display the UI element in the varied state, and
if it is determined that the eye gaze of the user is directed toward the UI element and then is directed toward an area of the display where the UI element is not displayed while the UI element is displayed in the varied state and a preset user gesture is recognized, the processor is further configured to perform an operation corresponding to the UI element and the preset user gesture.

9. The display apparatus as claimed in claim 1 , wherein the UI element is at least one of a sound adjustment object, a channel adjustment object, a search box activation object, a keyboard object, and a guide object.

10. The display apparatus as claimed in claim 1 , wherein the communication circuit is further configured to receive the user gesture information from a wearable device worn by the user.

11. A method of controlling a display apparatus, the method comprising:
receiving eye gaze information of a user and user gesture information from an external apparatus;
displaying a user interface (UI) element for receiving a user input based on an eye gaze and a gesture on a screen of the display apparatus in response to determining that an eye gaze of the user toward the screen and a user gesture are simultaneously detected based on the received eye gaze information and the received user gesture information; and
if any one of the eye gaze of the user directed toward the screen and the user gesture is not recognized for at least a preset time after the first object is displayed, terminating the display of the UI element.

12. The method as claimed in claim 11 , further comprising detecting the eye gaze of the user on the screen.

13. The method as claimed in claim 12 , further comprising if it is determined that the detected eye gaze does not correspond to a position on which the UI element is displayed and a preset user gesture is recognized, correcting the detected eye gaze to a position on which a second UI element closest to the detected eye gaze is displayed.

14. The method as claimed in claim 12 , wherein detecting the eye gaze of the user on the screen comprises performing a user recognition based on user input information and performing an eye gaze detection according to the user recognition.

15. The method as claimed in claim 12 , wherein detecting the eye gaze of the user on the screen comprises determining a viewing state of the user from information received from the external apparatus and performing an eye gaze detection according to the determined viewing state.

16. The method as claimed in claim 11 , further comprising if it is determined that the eye gaze of the user is directed toward the displayed UI element and the preset user gesture is recognized, performing an operation corresponding to the displayed UI element toward which the eye gaze of the user is directed and the preset user gesture.

17. The method as claimed in claim 11 , wherein displaying the UI element comprises displaying a plurality of UI elements including the UI element, and
herein the method further comprises in response to determining that the eye gaze of the user is directed toward the UI element, displaying the UI element differently from remaining UI elements of the plurality of UI elements.

18. The method as claimed in claim 11 , further comprising:
if it is determined that the eye gaze of the user is directed toward the UI element, varying the UI element and displaying the UI element in the varied state, and
if it is determined that the eye gaze of the user is directed toward the UI element and then is directed toward an area of the display apparatus where the UI element is not displayed while the UI element is displayed in the varied state and a preset user gesture is recognized, performing an operation corresponding to the UI element and the preset user gesture.

19. The display apparatus as claimed in claim 1 , wherein the processor is configured to display the UI element in response to sensing both the eye gaze of the user directed toward the display and the user gesture, regardless of whether the eye gaze or the user gesture is sensed first.

20. The method as claimed in claim 11 , wherein displaying a UI element comprises displaying the UI element in response to sensing both the eye gaze of the user directed toward the display and the user gesture, regardless of whether the eye gaze or the user gesture is sensed first.

Assignees

• Samsung Electronics Co Ltd
• Samsung Electronics Co Ltd

Inventors

• Cheon Ka-Won
• Kim Hyun-Jin
• Moon Choon-Kyoung
• Rho Jae-Yeon
• Kang Seong-Hoon

Classifications

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F2203/0381

  Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer · CPC title

• G06F3/013 Primary

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• H04N21/42204

  User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor (constructive details of casings for the remote control device H01H9/0235; user interfaces for controlling a tuning device of a television receiver through a remote control H03J9/00; remote control of peripheral devices connected to a television receiver through the remote control device of the television receiver H04B1/205) · CPC title

• G06F3/04845

  for image manipulation, e.g. dragging, rotation, expansion or change of colour · CPC title