Systems and method for action recognition using micro-doppler signatures and recurrent neural networks

Patent metadata

Field	Value
Publication number	US-11295119-B2
Application number	US-201816626591-A
Country	US
Kind code	B2
Filing date	Jul 2, 2018
Priority date	Jun 30, 2017
Publication date	Apr 5, 2022
Grant date	Apr 5, 2022

Abstract

The present disclosure may be embodied as systems and methods for action recognition developed using a multimodal dataset that incorporates both visual data, which facilitates the accurate tracking of movement, and active acoustic data, which captures the micro-Doppler modulations induced by the motion. The dataset includes twenty-one actions and focuses on examples of orientational symmetry that a single active ultrasound sensor should have the most difficulty discriminating. The combined results from three independent ultrasound sensors are encouraging, and provide a foundation to explore the use of data from multiple viewpoints to resolve the orientational ambiguity in action recognition. In various embodiments, recurrent neural networks using long short-term memory (LSTM) or hidden Markov models (HMMs) are disclosed for use in action recognition, for example, human action recognition, from micro-Doppler signatures.

First claim

What is claimed is:

1. A system for action recognition of sensed motion, comprising:
a first micro-Doppler sensor configured to produce a spectrogram of a sensed motion;
a splitter configured to receive the spectrogram from the first micro-Doppler sensor and segment the spectrogram into a sequence of spectrogram slices, each spectrogram slice of the sequence of spectrogram slices having a pre-determined time period;
an artificial neural network (“ANN”) for classifying sensed motion, the ANN comprising a recognition layer configured to receive the sequence of spectrogram slices from the splitter as an input vector.

2. The system of claim 1 , wherein the ANN is a recurrent neural network (“RNN”).

3. The system of claim 2 , wherein the recognition layer comprises a plurality of hidden Markov models (“HMM”), each HMM corresponding to a hidden sequence of skeletal poses and a visible sequence of spectrogram slice prototypes.

4. The system of claim 3 , wherein the recognition layer further comprises a Viterbi module configured to receive a plurality of spectrogram slice prototypes and to find a most likely sequence of skeletal pose prototypes.

5. The system of claim 2 , wherein the recognition layer comprises one or more long short-term memory layers.

6. The system of claim 1 , wherein the ANN is a convolutional deep belief network (“CDBN”).

7. The system of claim 1 , wherein the micro-Doppler sensor is an ultrasound sensor.

8. The system of claim 1 , further comprising one or more additional micro-Doppler sensors, each additional micro-Doppler sensor configured to produce a spectrogram of the sensed motion.

9. The system of claim 8 , wherein the splitter is further configured to concatenate the spectrograms from the first micro-Doppler sensor and the one or more additional micro-Doppler sensors; and wherein the recognition layer is configured to receive the concatenated spectrograms as the input vector.

10. The system of claim 8 , further comprising one or more additional ANNs, each ANN corresponding to an additional micro-Doppler sensor of the one or more additional micro-Doppler sensors and having a recognition layer configured to receive a plurality of spectrogram slices as an input vector.

11. The system of claim 1 , further comprising a motion sensor configured to produce RGB-D data of a sensed motion, and where the input vector further comprises a sequence of RGB-D data.

12. A method for action recognition of sensed motion, comprising:
capturing a spectrogram of a sensed motion using a micro-Doppler sensor;
segmenting the captured spectrogram into a sequence of spectrogram slices, each spectrogram slice of the sequence of spectrogram slices having a pre-determined time period; and
classifying the sequence of spectrogram slices as an action using an artificial neural network (“ANN”) having a recognition layer configured to receive the sequence of spectrogram slices as an input vector.

13. The method of claim 12 , further comprising translating each spectrogram slice of the sequence of spectrogram slices into a spectrogram slice prototype based on the Euclidean distance between each spectrogram slice and each spectrogram slice prototype.

14. The method of claim 13 , wherein the recognition layer comprises a plurality of hidden Markov models (“HMM”), each HMM corresponding to a hidden sequence of skeletal poses and a visible sequence of spectrogram slice prototypes; and classifying the sequence of spectrogram slices further comprises:
computing, using each HMM, a most likely sequence of hidden skeletal poses;
computing a log-likelihood of each of the most likely sequences of hidden sequence of skeletal poses and the visible sequence of spectrogram slice prototypes; and
selecting the action based on the computed log-likelihoods.

15. The method of claim 14 , wherein a Viterbi algorithm is used to compute a maximum a posteriori (MAP) estimate.

16. The method of claim 12 , further comprising capturing additional spectrograms of the sensed motion using additional micro-Doppler sensors.

17. The method of claim 16 , further comprising concatenating the spectrogram and the additional spectrograms.

Assignees

• Univ Johns Hopkins

Inventors

• Andreou Andreas G
• Sanni Kayode
• Murray Thomas S
• Mendat Daniel R
• Pouliquen Philippe O

Classifications

• G01S7/539 Primary

  using analysis of echo signal for target characterisation; Target signature; Target cross-section · CPC title

• G06V10/34

  Smoothing or thinning of the pattern; Morphological operations; Skeletonisation · CPC title

• G06V10/84

  using probabilistic graphical models from image or video features, e.g. Markov models or Bayesian networks · CPC title

• G06V40/25

  Recognition of walking or running movements, e.g. gait recognition · CPC title

• G06F2218/20

  by applying autoregressive analysis · CPC title