System, components and methodologies for gaze dependent gesture input control

Patent metadata

Field	Value
Publication number	US-9244527-B2
Application number	US-201313850668-A
Country	US
Kind code	B2
Filing date	Mar 26, 2013
Priority date	Mar 26, 2013
Publication date	Jan 26, 2016
Grant date	Jan 26, 2016

Abstract

A control system, components and methodologies enable control of vehicle and in-vehicle system functionality via gesture recognition. In illustrative embodiments, the control system, constituent components and methodologies determine what functionality a user wishes to control by tracking the user's gaze to identify a selected function or function set. In combination with identifying the selected function and function set, one or more sensors installed in the vehicle detect one or more user gestures initiated by a user to interact with and/or control the selected function/function set.

First claim

The invention claimed is:

1. A control system comprising:
at least one user interface that outputs data to a user;
at least one gaze tracking sensor that tracks a gaze of the user and outputs data indicating a gaze direction of the user;
a processor that receives the data indicating the gaze direction of the user; and
a plurality of gesture recognition sensors that configured to detect at least one gesture made by the user, on a plurality of disjunct surfaces included in an automotive vehicle in which the user is sitting and outputs data indicating the at least one gesture,
wherein the processor includes a means for determining the functionality a user wishes to control by tracking the user's gaze using the data indicating the gaze direction of the user and identifying a selected area of interest displayed on the at least one user interface based on the data indicating the gaze direction of the user,
wherein the processor also includes a means for recognizing at least one gesture based on the data indicating the at least one gesture made by the user and for controlling at least one operation associated with the selected area of interest based on the data indicating the at least one gesture made by the user, and
wherein the control system controls operation of the vehicle or in-vehicle control system for the vehicle in which the user is sitting.

2. The control system of claim 1 , wherein the at least one user interface includes a touch screen display that both outputs data to the user and receives data from the user.

3. The control system of claim 1 , wherein the system further comprises a plurality of user interfaces including the at least one user interface that outputs data to the user, wherein at least one other user interface included in that plurality includes a touch surface that receives data from the user in the form of a gesture data and provides the at least one gesture recognition sensor.

4. The control system of claim 3 , wherein data input to the touch surface is received from a user by received data indicating one or more positions on the touch surface where the user has drawn.

5. The control system of claim 1 , wherein the system controls in-vehicle infotainment features and/or driver assistance systems within the automotive vehicle.

6. The control system of claim 1 , wherein the means for determining and identifying the selected area of interest alters display of the selected area of interest on the at least one user interface once the selected area of interest is identified.

7. The control system of claim 1 , wherein the means for recognizing at least one gesture and the at least one gesture recognition sensor cooperate to recognize the user's at least one gesture made in free space.

8. The control system of claim 1 , wherein the means for recognizing at least one gesture and the at least one gesture recognition sensor cooperate to recognize the user's at least one gesture made on a surface within arm's reach of the user.

9. The control system of claim 1 , wherein the at least one gesture recognition sensor is a camera-based hand and finger gesture tracking sensor and/or tactile control surface sensor.

10. The control system of claim 1 , wherein the at least one gaze tracking sensor is a camera-based eye tracking sensor.

11. A method for receiving at least one control command from a user using a control system, the method comprising:
tracking a gaze of the user using at least one gaze tracking sensor and outputting data indicating a gaze direction of the user;
receiving, at a processor, the data indicating the gaze direction of the user; detecting via a plurality of gesture recognition sensors at least one gesture performed by the user on a plurality of disjunct surfaces included in an automotive vehicle in which the user is sitting using at least one gesture recognition sensor and outputting data indicating the at least one gesture to the processor;
determining functionality a user wishes to control by tracking the user's gaze using the data indicating the gaze direction of the user and identifying a selected area of interest displayed on at least one user interface based on the data indicating the gaze direction of the user, and
recognizing at least one gesture based on the data indicating the at least one gesture made by the user; and
controlling at least one operation associated with the selected area of interest based on the data indicating the at least one gesture made by the user
wherein the at least one operation controlled by the method is an operation of the vehicle or in-vehicle control system for the vehicle in which the user is sitting.

12. The method of claim 11 , further comprising outputting data to the user via the at least one user interface prior to tracking the gaze direction of the user.

13. The method of claim 12 , wherein a plurality of user interfaces are utilized that include the at least one user interface that outputs data to the user, wherein at least one other user interface included in that plurality includes a touch surface that receives data from the user in the form of a gesture data and provides the at least one gesture recognition sensor.

14. The method of claim 13 , wherein data input to the touch surface is received from a user by received data indicating one or more positions on the touch surface where the user has drawn.

15. The method of claim 11 , wherein the at least one user interface includes a touch screen display that both outputs data to the user and receives data from the user.

16. The method of claim 11 , wherein the at least one operation controls in-vehicle infotainment features and/or driver assistance systems within the automotive vehicle.

17. The method of claim 11 , further comprising altering display of the selected area of interest on the at least one user interface once the selected area of interest is identified.

18. The method of claim 11 , wherein the recognizing of the at least one gesture is performed using at least one gesture recognition sensor that recognizes the user's at least one gesture made in free space.

19. The method of claim 11 , wherein the recognizing of the at least one gesture is performed using at least one gesture recognition sensor that recognizes the user's at least one gesture made on a surface within arm's reach of the user.

20. The method of claim 11 , wherein the at least one gesture recognition sensor is a camera-based hand and finger gesture tracking sensor and/or tactile control surface sensor.

21. The method of claim 11 , wherein the at least one gaze tracking sensor is a camera-based eye tracking sensor.

22. A transportation system comprising:
at least one user interface that outputs data to a user;
at least one gaze tracking sensor that tracks a gaze of the user and outputs data indicating a gaze direction of the user;
a passenger vehicle including the at least one user interface,
a processor that receives the data indicating the gaze direction of the user; and
a plurality of camera-based gesture recognition sensors configured to detect at least one gesture made by the user on a plurality of disjunct surfaces included in an automotive vehicle in which the user is sitting and outputs data indicating the at least one gesture,
wherein the processor includes a means for determining the functionality a user wishes to control by tracking the user's gaze using the data indicating the gaze direction of the user and identifying a selected area of interest displayed

Assignees

• Volkswagen Ag
• Audi Ag

Inventors

• Lathrop William Brian
• Mejia Gonzalez Maria Esther
• Grant Bryan
• Maiwand Heiko

Classifications

• G06F2203/0381

  Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer · CPC title

• G06F3/013 Primary

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06F3/0416

  Control or interface arrangements specially adapted for digitisers · CPC title