Systems, devices, and methods for gesture identification

Patent metadata

Field	Value
Publication number	US-10684692-B2
Application number	US-201715852196-A
Country	US
Kind code	B2
Filing date	Dec 22, 2017
Priority date	Jun 19, 2014
Publication date	Jun 16, 2020
Grant date	Jun 16, 2020

Abstract

Systems, devices, and methods adapt established concepts from natural language processing for use in gesture identification algorithms. A gesture identification system includes sensors, a processor, and a non-transitory processor-readable memory that stores data and/or instructions for performing gesture identification. A gesture identification system may include a wearable gesture identification device. The gesture identification process involves segmenting signals from the sensors into data windows, assigning a respective “window class” to each data window, and identifying a user-performed gesture based on the corresponding sequence of window classes. Each window class exclusively characterizes at least one data window property and is analogous to a “letter” of an alphabet. Under this model, each gesture is analogous to a “word” made up of a particular combination of window classes.

First claim

The invention claimed is:

1. A method of operating a gesture identification system to identify a user-performed gesture, the gesture identification system including at least one sensor responsive to user-performed gestures and a processor communicatively coupled to the at least one sensor, the method comprising:
providing at least one signal from the at least one sensor to the processor;
segmenting the at least one signal into data windows;
for each i th data window in at least a subset of the data windows:
determining a window class for the i th data window by the processor, the window class selected by the processor from a library of window classes, wherein each window class in the library of window classes exclusively characterizes at least one data window property;
determining, by the processor, a respective probability that each gesture in a gesture library is the user-performed gesture based on the window class for the i th data window, wherein determining, by the processor, a respective probability that each gesture in a gesture library is the user-performed gesture includes:
determining, by the processor, at least one respective transition model for each gesture in the gesture library; and
determining, by the processor and for each gesture in the gesture library, a probability that the gesture is the user-performed gesture based at least in part on the at least one transition model for the gesture; and

identifying a highest-probability gesture for the i th data window by the processor, the highest-probability gesture corresponding to the gesture in the gesture library that has a highest probability of being the user-performed gesture for the i th data window; and

identifying the user-performed gesture by the processor based on the highest-probability gesture for at least one data window in the at least a subset of data windows.

2. The method of claim 1 wherein providing at least one signal from the at least one sensor to the processor includes providing at least one substantially continuous data stream from the at least one sensor to the processor, and wherein segmenting the at least one signal into data windows includes segmenting the at least one substantially continuous data stream into data windows in real-time by the processor.

3. The method of claim 1 wherein each window class in the library of window classes exclusively characterizes a respective range of values for the same at least one data window property.

4. The method of claim 3 wherein each window class in the library of window classes exclusively characterizes a respective range of values for at least one Root Mean Square (“RMS”) value for the i th data window.

5. The method of claim 1 wherein the at least one sensor includes a plurality of sensors, and wherein:
providing at least one signal from the at least one sensor to the processor includes providing a respective signal from each respective sensor in the plurality of sensors to the processor; and
segmenting the at least one signal into data windows includes segmenting the respective signal from each respective sensor in the plurality of sensors into the data windows, wherein each data window includes a respective portion of the signal from each respective sensor in the plurality of sensors.

6. The method of claim 5 wherein the plurality of sensors includes N sensors, and wherein each window class in the library of window classes exclusively characterizes a respective region in an N-dimensional hyperspace and each dimension of the N-dimensional hyperspace represents a property of the signal from a respective one of the N sensors.

7. The method of claim 6 wherein each region of the N-dimensional hyperspace represents a respective combination of ranges for Root Mean Square (“RMS”) values of the signals from the N sensors.

8. The method of claim 6 wherein, for each window class in the library of window classes, the corresponding region in the N-dimensional hyperspace is exclusively characterized by at least one respective angle formed in the N-dimensional hyperspace.

9. The method of claim 1 wherein the at least one sensor includes at least one muscle activity sensor selected from the group consisting of: an electromyography (EMG) sensor and a mechanomyography (MMG) sensor, and wherein:
providing at least one signal from the at least one sensor to the processor includes providing at least one signal from the at least one muscle activity sensor to the processor.

10. The method of claim 1 wherein the at least one sensor includes at least one inertial sensor selected from the group consisting of: an accelerometer, a gyroscope, and an inertial measurement unit (IMU), and wherein:
providing at least one signal from the at least one sensor to the processor includes providing at least one signal from the at least one inertial sensor to the processor.

11. The method of claim 1 wherein the gesture identification system further comprises a non-transitory processor-readable storage medium that stores processor-executable gesture identification instructions, the non-transitory processor-readable storage medium communicatively coupled to the processor, and the method further comprising: executing by the processor the gesture identification instructions to cause the processor to:
determine the window class for each i th data window;
determine the respective probability that each gesture in the gesture library is the user-performed gesture for each i th data window;
identify the highest-probability gesture for each i th data window; and
identify the user-performed gesture.

12. A gesture identification system comprising:
at least one sensor responsive to physical gestures performed by a user of the gesture identification system, wherein in response to a physical gesture performed by the user the at least one sensor provides at least one signal;
a processor communicatively coupled to the at least one sensor; and
a non-transitory processor-readable storage medium communicatively coupled to the processor, wherein the non-transitory processor-readable storage medium stores processor-executable gesture identification instructions that, when executed by the processor, cause the gesture identification device to:
segment the at least one signal into data windows;
for each i th data window in at least a subset of the data windows:
determine a window class for the i th data window, the window class selected from a library of window classes, wherein each window class in the library of window classes exclusively characterizes at least one data window property;
determine a respective probability that each gesture in a gesture library is the physical gesture performed by the user based on the window class for the i th data window, wherein the processor-executable gesture identification instructions that, when executed by the processor, cause the gesture identification device to determine a respective probability that each gesture in a gesture library is the physical gesture performed by the user, cause the gesture identification device to:
determine at least one respective transition model for each gesture in the gesture library; and
determine, for each gesture in the gesture library, a probability that the gesture is the physical gesture performed by the user based at least in part on the at least one transition model for the gesture; and

identify a highest-probability gesture for the i th data window, the highest-probability gesture corresponding to the gesture in the gesture library that has a highest probability of being the physical gesture performed by the user for the i th data window; and

identify the physical gesture per

Assignees

• Facebook Tech Llc

Inventors

• Aleem Idris S
• Ataee Pedram

Classifications

• G06F18/2415

  based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate · CPC title

• G06F2218/12

  Classification; Matching · CPC title

• G06V40/20

  Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) · CPC title

• G06F1/163

  Wearable computers, e.g. on a belt · CPC title

• G06F3/015

  Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection · CPC title