Systems and methods for adaptive assessment

Patent metadata

Field	Value
Publication number	US-2021343175-A1
Application number	US-202016866117-A
Country	US
Kind code	A1
Filing date	May 4, 2020
Priority date	May 4, 2020
Publication date	Nov 4, 2021
Grant date	—

Abstract

Systems and methods for adaptive assessment may utilize test deliver policies derived from a multi-armed bandit approach in combination with an item response theory model. A monotonic policy may involve defining a difficulty range according to which test items are selected for delivery during an assessment, where the difficulty range is updated following delivery of each test item based on test-taker performance. A multi-stage policy may implement several stages for test item delivery, with a different initial difficulty range for test item selection being defined for each stage and the difficulty ranges being updated based on test-taker performance within each respective stage. A probability matching policy may involve defining an item difficulty probability distribution according to which test items are selected for delivery to a test taker, where the probability distribution is initialized based on test taker skill level and updated based on test taker performance.

First claim

The invention claimed is:

1 . A system comprising:
a server that is in electronic communication with a user device associated with a user account, the server comprising:
a processor; and
a memory device configured to store computer-readable instructions, which, when executed, cause the processor to:
initiate an assessment;
select a first test item from a test item bank based on first test item selection parameters;
cause a client device to deliver the first test item;
receive first response data from the client device;
perform analysis of the first response data;
produce second test item selection parameters by modifying the first test item selection parameters based on the analysis of the first response data;
select a second test item from the test item bank based on the second test item selection parameters;
cause the client device to deliver the second test item;
determine that an end condition has been met; and
responsive to determining that the end condition has been met, end the assessment.

2 . The system of claim 1 , wherein the first response data comprises recorded speech data, wherein performing analysis of the first response data comprises executing a speech recognition algorithm to identify and extract words from the recorded speech data.

3 . The system of claim 1 , wherein performing analysis of the first response data comprises:
generating a score based on the first response data; updating an item response theory model based on the score; updating a confidence level associated with the item response theory model; responsive to updating the confidence level, determining a change in the confidence level; and generating a reward value based on the change in the confidence level, wherein the second test item selection parameters are generated based on the score and the reward value.

4 . The system of claim 3 , wherein the first test item selection parameters comprise a first difficulty range, wherein the second test item selection parameters comprise a second difficulty range, and wherein the computer-readable instructions, when executed, further cause the processor to:
generate a random number; determine that the reward value exceeds a predetermined threshold; determine that the random number exceeds a predetermined probability threshold; and responsive to determining that the reward value exceeds the predetermined threshold and that the random number exceeds the predetermined probability threshold, increasing the first difficulty range of the first test item selection parameters to the second difficulty range of the second test item selection parameters.

5 . The system of claim 4 , wherein selecting the second test item comprises:
randomly selecting the second test item from a group of test items of the test item bank, wherein the group of test items includes only test items having difficulty values within the second difficulty range, wherein the difficulty values of the test items of the group of test items are calculated using the item response theory model.

6 . The system of claim 3 , wherein the first test item selection parameters comprise a first probability distribution, wherein the second test item selection parameters comprise a second probability distribution, wherein updating the item response theory model comprises updating a user skill level of a user to which the test is being delivered based on the score, and wherein the computer-readable instructions, when executed, further cause the processor to:
responsive to updating the user skill level, generate the second probability distribution based on the update user skill level and the reward value.

7 . The system of claim 6 , wherein selecting the second test item comprises:
selecting the second test item from a group of test items of the test item bank according to the second probability distribution, such that a probability of selecting a given test item of the group of test items having a difficulty value determined by the item response theory mode is defined by the second probability distribution.

8 . The system of claim 1 , wherein determining that the end condition has been met comprises determining that a predetermined number of test items have been delivered.

9 . A system comprising:
a server that is in electronic communication with a user device associated with a user account, the server comprising:
a processor; and
a memory device configured to store computer readable instructions, which, when executed, cause the processor to:
initiate an assessment;
generate a random number;
select a first test item based on test item selection parameters and the random number, the first test item selection parameters defining a first difficulty range, wherein the first test item has a first difficulty value that is within the difficulty range;
cause a client device to deliver the first test item;
receive first response data from the client device;
perform analysis of the first response data;
update the test item selection parameters by increasing the first difficulty range to a second difficulty range based on the analysis of the first response data;
generate a second random number;
select a second test item having a second difficulty value within the second difficulty range based on the second random number;
cause the client device to deliver the second test item;
determine that a first end condition has been met; and
end the assessment.

10 . The system of claim 9 , wherein the first response data comprises recorded speech data, wherein performing analysis of the first response data comprises executing a speech recognition algorithm to identify and extract words from the recorded speech data.

11 . The system of claim 9 , wherein performing analysis of the first response data comprises:
generating a score based on the first response data; updating an item response theory model based on the score, wherein the first difficulty value and the second difficulty value are determined based on the item response theory model; updating a confidence level associated with the item response theory model; responsive to updating the confidence level, determining a change in the confidence level; and generating a reward value based on the change in the confidence level, wherein the second test item selection parameters are generated based on the score and the reward value.

12 . The system of claim 11 , wherein the computer-readable instructions, when executed, further cause the processor to:
determine that the reward value exceeds a predetermined threshold; and determine that the random number exceeds a predetermined probability threshold, wherein the second difficulty range is generated responsive to determining that the reward value exceeds the predetermined threshold and that the random number exceeds the predetermined probability threshold.

13 . The system of claim 9 , wherein determining that the first end condition has been met comprises determining that a predetermined number of test items have been delivered.

14 . The system of claim 9 , wherein the computer-readable instructions, when executed, further cause the processor to:
determine that the first end condition has been met by determining that a first predetermined number of test items have been delivered during a first stage, wherein the first difficulty range has a predefined association with the first stage; responsive to determining that the first predetermined number of test items have been delivered:
end the first stage,
initiate a second stage, and
update the item selection parameters to include a third diff

Assignees

• Pearson Education Inc

Inventors

• Wang Zhaodong

Classifications

• G09B7/04 Primary

  characterised by modifying the teaching program in response to a wrong answer, e.g. repeating the question or supplying a further explanation · CPC title

• G09B19/04

  Speaking (with audible presentation of the material to be studied G09B5/04) · CPC title