Electronic device and a related method for detecting and counting an action

Patent metadata

Field	Value
Publication number	US-12087093-B2
Application number	US-202117349853-A
Country	US
Kind code	B2
Filing date	Jun 16, 2021
Priority date	Jul 17, 2020
Publication date	Sep 10, 2024
Grant date	Sep 10, 2024

Abstract

An electronic device includes memory circuitry, and processor circuitry having an action detection circuitry configured to operate according to an action detection model for detecting an action based on a machine-learning scheme. The processor circuitry being configured to obtain sensor data; generate, based on the sensor data, a set of features associated with a frame; determine, based on the set, using the action detection model, whether the frame corresponds to a sub-action; apply a nondeterministic finite automaton, NFA, scheme, to the determined sub-action for the frame, wherein the NFA scheme has a set of states associated with corresponding sub-actions and is configured to output one or more action classes; determine, using the NFA scheme, an action class; detect the action based on the action class; and increment an action counter based on the detected action.

First claim

What is claimed is:

1. An electronic device comprising:
memory circuitry; and
processor circuitry comprising an action detection circuitry configured to operate according to an action detection model for detecting an action based on a machine-learning scheme, wherein the action comprises one or more sub-actions;
the processor circuitry being configured to:
obtain sensor data;
generate, based on the sensor data, a set of features associated with a frame;
determine, based on the set of features associated with the frame, using the action detection model, whether the frame corresponds to a sub-action, wherein the processor circuitry is configured to perform determinations for a number of consecutive frames above a threshold number of consecutive frames as corresponding to the sub-action before determining whether the frame corresponds to the sub-action;
apply a nondeterministic finite automaton, NFA, scheme, to the determined sub-action for the frame, wherein the NFA scheme has a set of states associated with corresponding sub-actions and is configured to output one or more action classes;
perform, using the NFA scheme, a state transition from a state associated with the sub-action to another state associated with another sub-action;
determine, using the NFA scheme, an action class;
detect the action based on the action class; and
increment an action counter based on the detected action.

2. The electronic device according to claim 1 , wherein the determination of the action class comprises a labelling of the frame with the action class.

3. The electronic device according to claim 1 , wherein the processor circuitry is configured to initialize the NFA scheme.

4. The electronic device according to claim 1 , wherein the machine-learning scheme is trained using a supervised scheme on the features for sub-action determination.

5. The electronic device according to claim 1 , wherein the machine-learning scheme comprises a supervised machine learning scheme.

6. The electronic device according to claim 5 , wherein the supervised machine learning scheme comprises one or more of:
a Convolutional Neural Network;
a Graph Convolutional Network;
a random forest scheme;
a gradient boosting scheme;
linear models; and/or
a Recurrent Neural Network.

7. The electronic device according to claim 1 , wherein the processor circuitry is configured to repeat for each of a series of frames:
the determination, based on the set of features associated with the frame, using the action detection model, of whether the frame corresponds to a sub-action, and
the application of the NFA scheme to the determined sub-action for the frame.

8. The electronic device according to claim 1 , wherein the determination of the sub-action is performed after expiry of a time period.

9. The electronic device according to claim 1 , the electronic device comprising associated sensor circuitry configured to generate the sensor data.

10. The electronic device according to claim 1 , wherein the electronic device is an electronic user device.

11. The electronic device according to claim 1 , wherein the electronic device is a server device.

12. The electronic device according to claim 1 , wherein the set of features associated with the frame is generated using a sliding window with a number of frames.

13. The electronic device according to claim 12 , wherein the processor circuitry is configured to update the sliding window.

14. The electronic device according to claim 1 , wherein the determining, based on the set of features associated with the frame, using the action detection model, whether the frame corresponds to the sub-action is based on a detection of a start of the sub-action or an end of the sub-action using the set and the action detection model.

15. A method, performed by an electronic device, for counting an action, wherein the action comprises one or more sub-actions, the method comprising:
obtaining sensor data;
generating, based on the sensor data, a set of features associated with a frame,
determining, based on the set, using an action detection model, whether the frame corresponds to a sub-action, wherein determinations for a number of consecutive frames above a threshold number of consecutive frames as corresponding to the sub-action are performed before determining whether the frame corresponds to the sub-action, wherein the action detection model is based on a machine-learning scheme,
applying a nondeterministic finite automaton, NFA, scheme to the determined sub-action for the frame, wherein the NFA scheme has a set of states associated with corresponding sub-actions and is configured to output one or more action classes;
performing, using the NFA scheme, a state transition from a state associated with the sub-action to another state associated with another sub-action;
determining, using the NFA scheme, an action class;
detecting the action based on the action class; and
incrementing an action counter based on the detected action.

16. The method according to claim 15 , wherein determining of the action class comprises labelling of the frame with the action class.

17. The method according to claim 15 , wherein the method comprises initializing the NFA scheme.

18. The method according to claim 16 , wherein the method comprises training the machine-learning scheme using a supervised scheme on the features for sub-action determination.

19. The method according to claim 16 , wherein the machine-learning scheme comprises a supervised machine learning scheme.

20. The electronic device according to claim 1 , wherein the processor circuitry is configured to perform the determinations for a number of consecutive frames as corresponding to the sub-action above a threshold number of two or more consecutive frames before determining whether the frame corresponds to the sub-action.

21. The electronic device according to claim 20 , wherein:
the processor circuitry is configured to perform the determinations as predictions for the number of consecutive frames above the threshold number of two or more consecutive frames as corresponding to the sub-action before determining whether the frame corresponds to the sub-action.

22. The electronic device according to claim 1 , wherein the processor circuitry is configured to perform the determinations for a number of consecutive frames as corresponding to the sub-action above a configurable threshold based on one or more of:
a sampling rate;
a false positive requirement; and/or
a false negative requirement.

Assignees

• Sony Group Corp

Inventors

• Huang Sangxia
• Strömbäck David

Classifications

• G06F18/24

  Classification techniques · CPC title

• G06F18/21

  Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation · CPC title

• A63B2220/17

  Counting, e.g. counting periodical movements, revolutions or cycles, or including further data processing to determine distances or speed · CPC title

• A63B2220/05

  Image processing for measuring physical parameters · CPC title

• A63B24/0062

  Monitoring athletic performances, e.g. for determining the work of a user on an exercise apparatus, the completed jogging or cycling distance · CPC title