Method for recognizing emotion, training method, apparatuses, device, storage medium and product

Patent metadata

Field	Value
Publication number	US-2024404267-A1
Application number	US-202218697413-A
Country	US
Kind code	A1
Filing date	Sep 29, 2022
Priority date	Feb 9, 2022
Publication date	Dec 5, 2024
Grant date	—

Abstract

A method for recognizing emotion, a training method, apparatuses, a device, a storage medium and a product. The method for recognizing emotion includes: acquiring to-be-recognized spiking sequences corresponding to video information; and recognizing the to-be-recognized spiking sequences by using a spiking neural network emotion recognition model, so as to obtain a corresponding emotion category.

First claim

1 . A method for recognizing emotion, comprising:
acquiring to-be-recognized spiking sequences corresponding to video information; and recognizing the to-be-recognized spiking sequences by using a spiking neural network emotion recognition model, so as to obtain a corresponding emotion category.

2 . The method for recognizing emotion as claimed in claim 1 , wherein before recognizing the to-be-recognized spiking sequences by using the spiking neural network emotion recognition model, the method further comprises:
training a pre-established spiking neural network emotion recognition model, to obtain a trained spiking neural network emotion recognition model.

3 . The method for recognizing emotion as claimed in claim 2 , wherein training the pre-established spiking neural network emotion recognition model, to obtain the trained spiking neural network emotion recognition model, comprises:
acquiring test sets of a plurality of emotion categories; and performing test training on the pre-established spiking neural network emotion recognition model by using the test sets, to obtain a trained spiking neural network emotion recognition model.

4 . The method for recognizing emotion as claimed in claim 2 , wherein training the pre-established spiking neural network emotion recognition model, to obtain the trained spiking neural network emotion recognition model, comprises:
pre-establishing an emotion recognition-based dynamic visual data set; and training the pre-established spiking neural network emotion recognition model by using the dynamic visual data set, to obtain a trained spiking neural network emotion recognition model.

5 . The method for recognizing emotion as claimed in claim 4 , wherein the process of pre-establishing the emotion recognition-based dynamic visual data set comprises: acquiring emotion recognition-based raw visual data;
performing simulation processing on the raw visual data by using a dynamic visual sensor simulation method, to obtain a plurality of spiking sequences corresponding to the raw visual data; or directly acquiring a plurality of spiking sequences corresponding to the raw visual data by using a dynamic visual camera; and establishing an emotion recognition-based dynamic visual data set on the basis of the plurality of spiking sequences.

6 . The method for recognizing emotion as claimed in claim 5 , wherein the process of performing simulation processing on the raw visual data by using the dynamic visual sensor simulation method, to obtain the plurality of spiking sequences corresponding to the raw visual data, comprises:
sequentially traversing N frames of video frame images in the raw dynamic video data, wherein N represents a total number of video frame images contained in the raw visual data; when traversing to a current ith frame, converting a video frame image of the current ith frame from an RGB color space to a grayscale space, and taking the converted video frame data as current video frame data, wherein the numerical range of i is from 1 to N; and when the value of i is equal to 1, assigning all floating-point data of the current video frame data to a first output channel of a first time step of simulation data, to obtain a spiking sequence composed of the first output channel, and taking the current video frame data as a previous video frame.

7 . The method for recognizing emotion as claimed in claim 6 , wherein the process of performing simulation processing on the raw visual data by using the dynamic visual sensor simulation method, to obtain corresponding spiking sequences, further comprises:
when i is not equal to 1, respectively assigning the first output channel and a second output channel according to a preset threshold and a grayscale difference value between the current video frame and the previous video frame, and taking the current video frame data as the previous video frame; updating the value of i by adding 1; and when a updated i is less than N, executing the step of converting the video frame image of the current ith frame from the RGB color space to the grayscale space, and taking the converted video frame data as the current video frame data.

8 . The method for recognizing emotion as claimed in claim 7 , wherein the process of performing simulation processing on the raw visual data by using the dynamic visual sensor simulation method, to obtain corresponding spiking sequences, further comprises:
when the updated i is not less than N, completing traversing of the N frames of video frame images in the raw dynamic video data, to obtain spiking sequences composed of the first output channel and the second output channel.

9 . The method for recognizing emotion as claimed in claim 7 , wherein respectively assigning the first output channel and the second output channel according to the grayscale difference value between the current video frame and the previous video frame and the preset threshold, comprises:
calculating, for each pixel, a grayscale difference value between the current video frame and the previous video frame at the pixel; assigning 1 to a position corresponding to the first output channel when the grayscale difference value is greater than the preset threshold; or assigning 1 to a position corresponding to the second output channel when the grayscale difference value is less than the preset threshold.

10 . The method for recognizing emotion as claimed in claim 4 , wherein the spiking neural network comprises a voting neuronal population component;
the process of training the pre-established spiking neural network emotion recognition model by using the dynamic visual data set, to obtain the trained spiking neural network emotion recognition model, comprises: initializing a parameter weight of the pre-established spiking neural network emotion recognition model; using the dynamic visual data set as an input to a current spiking neural network in the spiking neural network emotion recognition model, and obtaining an output frequency of a voting neuronal population of each emotion category via forward propagation of the current spiking neural network; calculating, regarding each emotion category, an error between the output frequency of the voting neuronal population of the emotion category and a real label of a corresponding emotion category; calculating a gradient corresponding to the parameter weight according to the error, and updating the parameter weight of the current spiking neural network by using the gradient; judging whether the current spiking neural network after updating the parameter weight converges; and when it is judged that the current spiking neural network after updating the parameter weight has converged, stopping training, to obtain a trained spiking neural network emotion recognition model.

11 . The method for recognizing emotion as claimed in claim 10 , wherein the process of training the pre-established spiking neural network emotion recognition model by using the dynamic visual data set, to obtain the trained spiking neural network emotion recognition model, further comprises:
when it is judged that the current spiking neural network after updating the parameter weight has not converged, returning to execute the step of using the dynamic visual data set as the input to the current spiking neural network in the spiking neural network emotion recognition model, and obtaining the output frequency of the voting neuronal population of each emotion category via forward propagation of the current spiking neural network.

12 . The method for recognizing emotion as claimed in claim 10 , wherein judging whether the current spiking neural network after updating t

Assignees

• Suzhou Metabrain Intelligent Technology Co Ltd

Inventors

• Zhao Yaqian
• Wang Binqiang
• Dong Gang
• Li Rengang

Classifications

• G06F18/00

  Pattern recognition · CPC title

• G06V10/776

  Validation; Performance evaluation · CPC title

• G06V10/82 Primary

  using neural networks · CPC title

• G06V20/40 Primary

  in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44) · CPC title

• G06N3/049 Primary

  Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs · CPC title