Motor task analysis system and method

Patent metadata

Field	Value
Publication number	US-10776423-B2
Application number	US-201515508618-A
Country	US
Kind code	B2
Filing date	Sep 3, 2015
Priority date	Sep 9, 2014
Publication date	Sep 15, 2020
Grant date	Sep 15, 2020

Abstract

Video processing for motor task analysis is described. In various examples, a video of a person carrying out a motor task, such as placing the forefinger on the nose, is input to a trained machine learning system to classify the motor task into one of a plurality of classes. In an example, motion descriptors such as optical flow are computed from pairs of frames of the video and the motion descriptors are input to the machine learning system. The motor task analysis may be used to assess or evaluate neurological conditions such as multiple sclerosis and/or Parkinson's.

First claim

The invention claimed is:

1. A computer-implemented method comprising:
providing a machine learning system;
training the machine learning system using a plurality of individuals to find location-dependent local motion features of videos which discriminate between a plurality of classes of motor tasks, the location-dependent local motion features including data relating to one or more discriminative spatio-temporal regions;
receiving a video depicting at least part of a person or animal performing at least one of the motor tasks and including data relating to one or more discriminative spatio-temporal regions of the video;
inputting the video to the trained machine learning system;
receiving, from the trained machine learning system, data about which of the plurality of classes of motor tasks the at least one motor task is predicted to belong to; and
evaluating a neurological disease or condition of the person or animal performing the at least one of the motor tasks based on the received data including data relating to the one or more discriminative spatio-temporal regions of the video.

2. A method as claimed in claim 1 wherein the local motion features comprise velocity or acceleration features.

3. A method as claimed in claim 2 comprising calculating the acceleration features by taking into account frequency of change of direction or rate of change of optical flow values of a sub-volume of the video.

4. A method as claimed in claim 3 comprising disregarding changes of direction of the rate of change of the optical flow values, where a magnitude of the optical flow values is below a threshold.

5. A method as claimed in claim 1 comprising calculating, for pairs of frames of the video, motion descriptors, and wherein inputting the video to the trained machine learning system comprises inputting the motion descriptors.

6. A method as claimed in claim 1 comprising pre-processing the video prior to inputting the video to the trained machine learning system, by scaling, centering and carrying out foreground extraction.

7. A method as claimed in claim 1 comprising training the machine learning system using videos of people performing the at least one of the motor tasks, where the videos are labeled with labels indicating which of the plurality of possible classes the at least one of the motor tasks belongs to, and where the videos are of different lengths.

8. A method as claimed in claim 1 comprising inputting the video to a trained machine learning system comprising any of a random decision forest, a jungle of directed acyclic graphs, an ensemble of support vector machines.

9. A method as claimed in claim 1 , wherein the neurological disease or condition of the person or animal performing the at least one of the motor tasks comprises one or more of Parkinson's Disease, Huntington's disease, Amyotrophic Lateral Sclerosis (ALS), cerebral palsy, Rheumatoid Arthritis, muscle wasting, chronic obstructive pulmonary disease (COPD), autism, schizophrenia, Alzheimer's disease, tremors, multiple sclerosis, depression, dementia, mania, bipolar disorder, obsessive-compulsive disorder, congestive heart failure, cardiac arrhythmia, gastrointestinal disorder, and incontinence.

10. A computer-implemented method comprising:
providing a machine learning system;
training the machine learning system using a plurality of individuals to find location-dependent local motion features of videos which discriminate between a plurality of classes of motor tasks, the location-dependent local motion features including data relating to one or more discriminative spatio-temporal regions;
receiving a video depicting at least part of a person or animal performing at least one of the motor tasks and including data relating to one or more discriminative spatio-temporal regions of the video;
inputting the video to the trained machine learning system;
receiving, from the trained machine learning system, data about which of the plurality of classes of motor tasks the at least one motor task is predicted to belong to; and
evaluating a neurological disease or condition of the person or animal performing the at least one of the motor tasks based on the received data including data relating to the one or more discriminative spatio-temporal regions of the video,
wherein the machine learning system is trained using videos of people performing the at least one of the motor tasks, where the videos are labeled with labels indicating which of the plurality of possible classes the at least one of the motor tasks belongs to.

11. A method as claimed in claim 10 wherein the local motion features comprise velocity or acceleration features.

12. A method as claimed in claim 11 comprising calculating the acceleration features by taking into account frequency of change of direction or rate of change of optical flow values of a sub-volume of the video.

13. A method as claimed in claim 12 comprising disregarding changes of direction of the rate of change of the optical flow values, where a magnitude of the optical flow values is below a threshold.

14. A method as claimed in claim 10 comprising calculating, for pairs of frames of the video, motion descriptors, and wherein inputting the video to the trained machine learning system comprises inputting the motion descriptors.

15. A method as claimed in claim 10 comprising pre-processing the video prior to inputting the video to the trained machine learning system, by scaling, centering and carrying out foreground extraction.

16. A method as claimed in claim 10 wherein the videos of people performing the at least one of the motor tasks are of different lengths.

17. A method as claimed in claim 10 comprising inputting the video to a trained machine learning system comprising any of a random decision forest, a jungle of directed acyclic graphs, an ensemble of support vector machines.

18. A method as claimed in claim 10 , wherein the neurological disease or condition of the person or animal performing the at least one of the motor tasks comprises one or more of Parkinson's Disease, Huntington's disease, Amyotrophic Lateral Sclerosis (ALS), cerebral palsy, Rheumatoid Arthritis, muscle wasting, chronic obstructive pulmonary disease (COPD), autism, schizophrenia, Alzheimer's disease, tremors, multiple sclerosis, depression, dementia, mania, bipolar disorder, obsessive-compulsive disorder, congestive heart failure, cardiac arrhythmia, gastrointestinal disorder, and incontinence.

Assignees

• Novartis Ag

Inventors

• Kontschieder Peter
• Dorn Jonas
• Zikic Darko
• Criminsi Antonio
• Dahlke Frank Kurt

Classifications

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F18/24155

  Bayesian classification · CPC title

• G06F18/24

  Classification techniques · CPC title

• G06V40/23

  Recognition of whole body movements, e.g. for sport training · CPC title

• G06V20/46

  Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames · CPC title