Engagement level determination and dissemination

Patent metadata

Field	Value
Publication number	US-11908345-B2
Application number	US-201816234255-A
Country	US
Kind code	B2
Filing date	Dec 27, 2018
Priority date	Dec 27, 2018
Publication date	Feb 20, 2024
Grant date	Feb 20, 2024

Abstract

Various systems and methods for engagement dissemination. A face detector detects a face in video data. A context filterer determines a student is on-platform and a section type. An appearance monitor selects an emotional and a behavioral classifiers. Emotional and behavioral components are classified based on the detected face. A context-performance monitor selects an emotional and a behavioral classifiers specific to the section type, Emotional and behavioral components are classified based on the log data. A fuser combines the emotional components into an emotional state of the student based on confidence values of the emotional components. The fuser combines the behavioral components a behavioral state of the student based on confidence values of the behavioral components. The user determine an engagement level of the student based on the emotional state and the behavioral state of the student.

First claim

What is claimed is:

1. An apparatus for engagement dissemination, the apparatus comprising:
a hardware processor; and
memory including instructions, which when executed by the hardware processor, cause the hardware processor to:
detect in video data, a face of a student of a plurality of students in a classroom;
determine that the student is on-platform based on log data;
determine a section type for the student based on the log data, wherein the section type is one of an instructional section type or an assessment section type;
select a first emotional classifier specific to the section type;
classify, using the first emotional classifier, a first emotional component based on the detected face;
select a first behavioral classifier specific to the section type;
classify, using the first behavioral classifier, a first behavioral component based on the detected face;
select a second emotional classifier specific to the section type;
classify, using the second emotional classifier, a second emotional component based on the log data and the section type by using input derived from the log data to extract a set of features, the set of features including at least one of: a number of failed first attempts to a number of questions, a number of hints used within a time period, or an amount of time spent on a first attempt on answering questions in a topic;
select a second behavioral classifier specific to the section type;
classify, using the second behavioral classifier, a second behavioral component based on the log data and the section type;
combine the classified first emotional component and the second emotional component into an emotional state of the student based on confidence values of the first emotional component and the second emotional component;
combine the classified first behavioral component and the second behavioral component into a behavioral state of the student based on confidence values of the first behavioral component and the second behavioral component;
determine an engagement level of the student based on the emotional state of the student and the behavioral state of the student; and
display a student avatar representing the student in a user interface, the user interface including a plurality of student avatars that includes the student avatar, each of the plurality of student avatars representing a respective student in the classroom, wherein each of the plurality of student avatars are colored with an engagement color indicating a respective engagement level of the respective student.

2. The apparatus of claim 1 , wherein the engagement level is one of engaged, may be engaged, or not engaged.

3. The apparatus of claim 2 , further comprising instructions, which when executed by the hardware processor, cause the hardware processor to:
determine the engagement color based on the engagement level of the student.

4. The apparatus of claim 3 , further comprising instructions, which when executed by the hardware processor, cause the hardware processor to display an intervention alert for the student based on the engagement level of the student.

5. The apparatus of claim 3 , wherein to determine the engagement color, the instructions, which when executed by the hardware processor, cause the hardware processor to:
determine a number of engagement levels for a previous time period;
determine a number of engagement level for each possible engagement level of the student for a previous time period;
calculate a ratio using the number of engagement levels and the number of engagement level for each possible engagement level;
determine a largest ratio from the calculated ratios; and
determine the engagement color based on the largest ratio.

6. The apparatus of claim 1 , wherein the behavior state is one of on-task or off-task.

7. The apparatus of claim 4 , wherein the emotional state is one of satisfied, bored, or confused.

8. The apparatus of claim 1 , further comprising instructions, which when executed by the hardware processor, cause the hardware processor to:
calculate first features from the detected face based on a proper subset of face animation parameters; and
calculate second features from the detected face based on movement of the detected face from previous video data, wherein the first features and the second features are used by the first emotional classifier and the second behavioral classifier.

9. The apparatus of claim 1 , further comprising instructions, which when executed by the hardware processor, cause the hardware processor to:
determine a number of questions accessed in a previous context time period; and
determine a number of questions where a hint associated with a question was accessed in the previous context time period, wherein the number of questions and the number of questions where a hint associated with a question was accessed are used by the first behavioral classifier and the second behavioral classifier.

10. A method for engagement dissemination, the method comprising operations performed using an electronic processor, the operations comprising:
detecting, in video data, a face of a student of a plurality of students in a classroom;
determining that the student is on-platform based on log data;
determining a section type for the student based on the log data, wherein the section type is one of an instructional section type or an assessment section type;
selecting, using an appearance monitor (APM), a first emotional classifier specific to the section type;
classifying, using the first emotional classifier, a first emotional component based on the detected face;
selecting, using the APM, a first behavioral classifier specific to the section type;
classifying, using the first behavioral classifier, a first behavioral component based on the detected face;
selecting, using a context-performance monitor (CPM), a second emotional classifier specific to the section type;
classifying, using the second emotional classifier, a second emotional component based on the log data and the section type;
selecting, using the CPM, a second behavioral classifier specific to the section type;
classifying, using the second behavioral classifier, a second behavioral component based on the log data and the section type by using input derived from the log data to extract a set of features, the set of features including at least one of: a number of failed first attempts to a number of questions, a number of hints used within a time period, or an amount of time spent on a first attempt on answering questions in a topic;
combining the classified first emotional component and the second emotional component into an emotional state of the student based on confidence values of the first emotional component and the second emotional component;
combining the classified first behavioral component and the second behavioral component into a behavioral state of the student based on confidence values of the first behavioral component and the second behavioral component;
determining an engagement level of the student based on the emotional state of the student and the behavioral state of the student; and
displaying a student avatar representing the student in a user interface, the user interface including a plurality of student avatars that includes the student avatar, each of the plurality of student avatars representing a respective student in the classroom, wherein each of the plurality of student avatars are colored with an engagement color indicating a respective engagement level of the respective student.

11. The method of claim 10 , wherein the engagement level is one of engaged, may be engaged, or not engaged.

12

Assignees

• Intel Corp

Inventors

• Alyuz Civitci Nese
• Tanriover Cagri Cagatay
• Aslan Sinem
• Okur Kavil Eda
• Arslan Esme Asli
• Genc Ergin

Classifications

• G09B5/08 Primary

  providing for individual presentation of information to a plurality of student stations · CPC title

• G06F18/256

  of results relating to different input data, e.g. multimodal recognition · CPC title

• G06V40/161

  Detection; Localisation; Normalisation · CPC title

• G06V40/171

  Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships · CPC title

• G06V40/172

  Classification, e.g. identification · CPC title