Methods and systems for computerized recognition of hand gestures

Patent metadata

Field	Value
Publication number	US-11227151-B2
Application number	US-202016809939-A
Country	US
Kind code	B2
Filing date	Mar 5, 2020
Priority date	Mar 5, 2020
Publication date	Jan 18, 2022
Grant date	Jan 18, 2022

Abstract

Methods, systems, and computer readable media for recognizing one or more hand gestures of a hand-based signal or conversation are described. Some implementations include obtaining one or more unprocessed images of the hand-based signal or conversation including images of at least one of the one or more hand gestures extracting one or more spectral features from the one or more unprocessed images using a Gabor filter bank, receiving the one or more unprocessed images of the one or more hand gestures and the extracted one or more spectral features by a Convolution Neural Network (CNN), and outputting a classification for the at least one of the one or more hand gestures using the Convolution Neural Network (CNN). In some implementations, at least one of the extracted one or more spectral features and at least one of the one or more unprocessed images of the one or more hand gestures are concatenated and input to the Convolution Neural Network (CNN).

First claim

The invention claimed is:

1. A computerized method for recognizing one or more hand gestures of a hand-based signal or conversation, the method comprising:
obtaining one or more unprocessed images of the hand-based signal or conversation, wherein the one or more unprocessed images include images of at least one of the one or more hand gestures, and wherein the one or more unprocessed images are being processed to determine a meaning of the hand-based signal or conversation;
extracting a plurality of spectral features from the one or more unprocessed images using a Gabor filter bank, wherein the Gabor filter bank performs Gabor filtering on the one or more unprocessed images, and wherein imaginary, phase, magnitude, and real Gabor responses are extracted from the plurality of spectral features;
concatenating the plurality of the extracted spectral features and the at least one of the one or more unprocessed images of the one or more hand gestures to generate an input layer of n feature maps of size 28×28, where 1≤n≤17;
receiving the one or more unprocessed images of the one or more hand gestures and the extracted one or more spectral features by a Convolution Neural Network (CNN);
filtering the input layer by a first convolution layer followed by a first rectified linear (ReLU) activation function to generate 128 feature maps with a size of 26×26;
subsampling the 128 feature maps with a size of 26×26 with a first 2×2 max-pooling window with a stride of 2 pixels to generate 128 feature maps with a size of 13×13;
filtering the 128 feature maps with a size of 13×13 by a second convolution layer followed by a second ReLU activation function and subsampling with a second 2×2 max-pooling window with a stride of 2 pixels to generate 128 feature maps with a size of 11×11;
filtering the 128 feature maps with a size of 11×11 by a third convolution layer followed by a third ReLU activation function and subsampling with a third 2×2 max-pooling window with a stride of 2 pixels to generate 128 feature maps with a size of 9×9;
performing a batch normalization on the 128 feature maps with a size of 9×9 to generate 128 feature maps with a size of 4×4;
applying a drop-out layer to the 128 feature maps with a size of 4×4 to generate 1024 feature maps with a size of 1×1;
applying the 1024 feature maps with a size of 1×1 to a first fully connected layer (FC) of 1024 neurons followed by a fourth ReLu activation function to generate 1024 first fully connected feature maps with a size of 1×1;
applying the 1024 first fully connected feature maps with a size of 1×1 to a second FC layer consisting of a number of neurons of a training dataset to generate 1024 second fully connected feature maps with a size of 1×1;
applying the second fully connected feature maps to a softmax classifier;
outputting a classification for the at least one of the one or more hand gestures using the softmax classifier, wherein the classification is selected by the CNN from among a plurality of candidate classifications to determine the meaning of the hand-based signal or conversation.

2. The method of claim 1 , wherein the Gabor filter bank performs Gabor filtering with a single scale and orientation to the one or more unprocessed images of the at least one of the one or more hand gestures, such that n=1.

3. The method of claim 1 , wherein the Gabor filter bank performs Gabor filtering with four scales and four orientations to the one or more unprocessed images of the at least one of the one or more hand gestures, such that n=16.

4. The method of claim 1 , wherein the first convolution layer has an architecture that includes 128 kernels of size 5×5 with a stride of 1 pixel.

5. The method of claim 1 , wherein the hand-based signal or conversation includes a sign language that comprises at least one of an American MNIST replacement sign language alphabet (MNIST ASL), an Arabic sign language alphabet (ArSL), or a Massey University ASL dataset (MUASL).

6. The method of claim 5 , further comprising:
creating the training dataset from the sign language, wherein the training dataset includes at least one of cropped hand gesture images, hand gesture images with noise added thereto, or the cropped hand gesture images with the noise added thereto.

7. A system for recognizing one or more hand gestures of a hand-based signal or conversation, the system comprising:
one or more processors and a non-transitory computer readable storage having software instructions stored thereon configured to cause the one or more processors to:
obtain one or more unprocessed images of the hand-based signal or conversation, wherein the one or more unprocessed images include images of at least one of the one or more hand gestures, and wherein the one or more unprocessed images are being processed to determine a meaning of the hand-based signal or conversation;
extract a plurality of spectral features from the one or more unprocessed images using a Gabor filter bank, wherein the Gabor filter bank performs Gabor filtering on the one or more unprocessed images, and wherein imaginary, phase, magnitude, and real Gabor responses are extracted from the plurality of spectral features;
concatenate the plurality of the extracted spectral features and the at least one of the one or more unprocessed images of the one or more hand gestures to generate an input layer of n feature maps of size 28×28, where 1≤n≤17;
receive the one or more unprocessed images of the one or more hand gestures and the extracted one or more spectral features by a Convolution Neural Network (CNN);
filter the input layer by a first convolution layer followed by a first rectified linear (ReLU) activation function to generate 128 feature maps with a size of 26×26;
subsample the 128 feature maps with a size of 26×26 with a first 2×2 max-pooling window with a stride of 2 pixels to generate 128 feature maps with a size of 13×13;
filter the 128 feature maps with a size of 13×13 by a second convolution layer followed by a second ReLU activation function and subsampling with a second 2×2 max-pooling window with a stride of 2 pixels, to generate 128 feature maps with a size of 11×11;
filter the 128 feature maps with a size of 11×11 by a third convolution layer followed by a third ReLU activation function and subsampling with a third 2×2 max-pooling window with a stride of 2 pixels to generate 128 feature maps with a size of 9×9;
perform a batch normalization on the 128 feature maps with a size of 9×9 to generate 128 feature maps with a size of 4×4;
apply a drop-out layer to the 128 feature maps with a size of 4×4 to generate 1024 feature maps with a size of 1×1;
apply the 1024 feature maps with a size of 1×1 to a first fully connected layer (FC) of 1024 neurons followed by a fourth ReLu activation function to generate 1024 fully connected feature maps with a size of 1×1;
apply the 1024 fully connected feature maps with a size of 1×1 to a second FC layer consisting of a number of neurons of a training database to generate 1024 second fully connected feature maps with a size of 1×1;
apply the second fully connected feature maps to a softmax classifier;
output a classification for the at least one of the one or more hand gestures using the softmax classifier, wherein the classification is selected by the CNN from among a plurality of candidate classifications to determine the meaning of the hand-based signal or conversation.

8. The system of claim 7 , wherein the Gabor filter bank performs Gabor filtering with a single scale and orientation to the one or more unprocessed images of the at least one of the one or more hand gestures, such that n=1.

9. The system of claim 7 , wherein the Gabor filter bank performs Gabor filtering with four scales

Assignees

• Univ King Fahd Pet & Minerals

Inventors

• Luqman Hamzah
• El-Alfy El-Sayed M
• Binmakhashen Galal M

Classifications

• G06V10/774

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/776

  Validation; Performance evaluation · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F18/2431

  Multiple classes · CPC title