Gesture-based user interface

Patent metadata

Field	Value
Publication number	US-11275446-B2
Application number	US-201715644008-A
Country	US
Kind code	B2
Filing date	Jul 7, 2017
Priority date	Jul 7, 2016
Publication date	Mar 15, 2022
Grant date	Mar 15, 2022

Abstract

A computer-implemented method for enabling gesture-based interactions between a computer program and a user is disclosed. According to certain embodiments, the method may include initiating the computer program. The method may also include detecting that a condition has occurred. The method may also include activating a gesture-based operation mode of the computer program. The method may also include receiving gesture data generated by a sensor, the gesture data representing a gesture performed by the user. The method may further include performing a task based on the gesture data.

First claim

What is claimed is:

1. A computer-implemented method used in a computer program to enable gesture-based interactions between the computer program and a user, the method comprising:
initiating the computer program;
detecting that a condition has occurred by:
determining an identity of a user based on facial changes associated with speech of the user; and
determining that the user is a disabled user;

automatically activating, based on the detection that the condition occurred, a gesture-based operation mode of the computer program;
receiving first gesture data generated by a sensor, the first gesture data representing a first one of a plurality of gestures;
determining that the first gesture does not correspond to a command; and
performing a task based on the first gesture data, wherein performing the task comprises:
determining a plurality of suggested commands using a machine-learning algorithm to analyze historical patterns of user behavior, the commands being determined based on first behavior data regarding a behavior related to the user;
presenting, through a user interface, information regarding the commands;
receiving an input from the user for selecting one of the commands; and
executing the selected command.

2. The method of claim 1 , wherein detecting that the condition has occurred further comprises:
receiving a signal indicating that an emergency has occurred.

3. The method of claim 1 , wherein detecting that the condition has occurred further comprises:
receiving a second input from the user for activating the gesture-based operation mode.

4. The method of claim 1 , further comprising:
receiving, from a server, second behavior data regarding behaviors of multiple users that use the computer program; and
determining the commands based on the second behavior data.

5. The method of claim 1 , further comprising:
presenting a prompt for the user to perform a second gesture, the prompt comprising the selected command;
receiving second gesture data generated by the sensor, the second gesture data representing a second one of the gestures;
determining the second gesture based on the second gesture data; and setting a corresponding relationship between the second gesture and the selected command.

6. The method of claim 1 , wherein performing the task further comprises:
determining a plurality of gesture candidates based on a collective usage trend for a plurality of second gestures of multiple users, the collective usage trend being determined by evaluating how often the second gestures are used by the multiple users;
presenting, through a user interface, information regarding the gesture candidates; and
receiving a second input from the user for selecting a gesture candidate.

7. The method of claim 1 , wherein:
receiving the gesture data comprises:
presenting, through a user interface, a prompt for the user to perform the first gesture, and
within a predetermined amount of time after the prompt is presented, receiving the first gesture data; and

performing the task comprises:
determining whether the first gesture data represents the first gesture; and
when it is determined that the gesture data represents the first gesture, setting a corresponding relationship between the first gesture and a predetermined command.

8. The method of claim 1 , wherein performing the task comprises:
determining a quality score based on the first gesture data; and
generating, based on the quality score, suggestions for the user to improve performance of the first gesture.

9. The method of claim 1 , wherein determining an identity of the user is based on at least one of unique hand or finger positioning patterns related to a gesture of the user.

10. A non-transitory computer readable medium having stored instructions, which when executed, cause at least one processor to perform a method for enabling gesture-based interactions between a computer program and a user, the method comprising:
initiating the computer program;
detecting that a condition has occurred by:
determining an identity of a user based on facial changes associated with speech of the user; and
indicating that the user is a disabled user;
automatically activating, based on the detection that the condition occurred, a gesture-based operation mode of the computer program;
receiving first gesture data generated by a sensor, the first gesture data representing a first one of a plurality of gestures;
determining that the first gesture does not correspond to a command; and
performing a task based on the gesture data, wherein performing the task comprises:
determining a plurality of suggested commands using a machine-learning algorithm to analyze historical patterns of user behavior, the commands being determined based on first behavior data regarding a behavior related to the user;
presenting, through a user interface, information regarding the commands;
receiving an input from the user for selecting one of the commands; and
executing the selected command.

11. The medium of claim 10 , wherein detecting that the condition has occurred further comprises:
receiving a signal indicating that an emergency has occurred.

12. The medium of claim 10 , wherein performing the task further comprises:
receiving, from a server, second behavior data regarding behaviors of multiple users that use the computer program; and
determining the commands based on the second behavior data.

13. The medium of claim 10 , wherein performing the task further comprises:
presenting a prompt for the user to perform a second gesture, the prompt comprising the selected command;
receiving new gesture data generated by the sensor, the new gesture data representing the second gesture performed by the user;
determining the second gesture based on the new gesture data; and
setting a corresponding relationship between the second gesture and the suggested command that is selected.

14. The medium of claim 10 , further comprising determining an identity of the user based on at least one of unique hand or finger positioning patterns related to a gesture of the user.

15. The medium of claim 10 , wherein performing the task further comprises:
determining a plurality of gesture candidates based on a collective usage trend for a plurality of second gestures of multiple users, the collective usage trend being determined by evaluating how often the second gestures are used by the multiple users;
presenting, through a user interface, information regarding the gesture candidates; and
receiving a second input from the user for selecting a gesture candidate.

16. The medium of claim 10 , wherein presenting information regarding the suggested commands comprises providing an audio message.

17. A terminal, comprising:
a memory storing instructions, the instructions being part of a computer program; and
a processor configured to executed the instructions to perform operations comprising:
initiating the computer program;
detecting that a condition has occurred by:
determining an identity of a user based on facial changes associated with speech of the user; and
determining that the user is a disabled user;

automatically activating, based on the detection that the condition occurred, a gesture-based operation mode of the computer program;
receiving first gesture data generated by a sensor, the first gesture data representing a first one of a plurality of gestures;
determining that the first gesture does not correspond to a command; and
performing a task based on the first gesture data, where

Assignees

• Capital One Services Llc

Inventors

• Franklin David
• Sedita Van Shea
• Simpson Stephen

Classifications

• G06F3/038

  Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry · CPC title

• G06F3/0304

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title

• G06Q40/00

  Finance; Insurance; Tax strategies; Processing of corporate or income taxes · CPC title

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• H04N21/42204

  User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor (constructive details of casings for the remote control device H01H9/0235; user interfaces for controlling a tuning device of a television receiver through a remote control H03J9/00; remote control of peripheral devices connected to a television receiver through the remote control device of the television receiver H04B1/205) · CPC title