Non-scorable response filters for speech scoring systems

Patent metadata

Field	Value
Publication number	US-9704413-B2
Application number	US-201514665258-A
Country	US
Kind code	B2
Filing date	Mar 23, 2015
Priority date	Mar 25, 2011
Publication date	Jul 11, 2017
Grant date	Jul 11, 2017

Abstract

A method for scoring non-native speech includes receiving a speech sample spoken by a non-native speaker and performing automatic speech recognition and metric extraction on the speech sample to generate a transcript of the speech sample and a speech metric associated with the speech sample. The method further includes determining whether the speech sample is scorable or non-scorable based upon the transcript and speech metric, where the determination is based on an audio quality of the speech sample, an amount of speech of the speech sample, a degree to which the speech sample is off-topic, whether the speech sample includes speech from an incorrect language, or whether the speech sample includes plagiarized material. When the sample is determined to be non-scorable, an indication of non-scorability is associated with the speech sample. When the sample is determined to be scorable, the sample is provided to a scoring model for scoring.

First claim

It is claimed:

1. A computer-implemented method of scoring non-native speech, comprising:
receiving a speech sample spoken by a non-native speaker to be scored by a scoring model using a processing system, wherein the scoring model generates scores for speech samples based on one or more speech metrics;
performing automatic speech recognition on the speech sample to generate a transcript of the speech sample;
processing the speech sample to generate a plurality of speech metrics and a confidence score associated with one of the scoring model speech metrics using the processing system;
applying a plurality of non-scorable response filters to the plurality of speech metrics and the confidence score using the processing system;
determining whether the speech sample is scorable or non-scorable based upon the transcript and a collective application of the non-scorable response filters to the plurality of speech metrics and the confidence score using the processing system, wherein the determining is based on assessment of audio quality of the speech sample, an amount of speech of the speech sample, a degree to which the speech sample is off-topic, and whether the speech sample includes speech from an incorrect language;
associating an indication of non-scorability with the speech sample when the sample is determined to be non-scorable using the processing system; and
providing the sample to a scoring model for scoring when the sample is determined to be scorable using the processing system.

2. The method of claim 1 , further comprising:
identifying a non-speech metric associated with a speaker of the non-native speech;
wherein determining whether the speech sample is scorable or non-scorable is based in part on the non-speech metric.

3. The method of claim 2 , wherein the non-speech metric is a demographic metric.

4. The method of claim 3 , wherein the demographic metric is a sex, a nationality, or a region associated with the speaker.

5. The method of claim 2 , wherein the non-speech metric is a score for the speaker on a different test item.

6. The method of claim 5 , wherein the different test item is a reading item or a writing item.

7. The method of claim 1 , wherein the confidence score is associated with a speech metric generated by an automatic speech recognizer.

8. The method of claim 1 , wherein the confidence score is associated with a module that generates one of the one or more speech metrics.

9. The method of claim 1 , wherein the speech sample is not provided to the scoring model when the speech sample is determined to be non-scorable.

10. The method of claim 1 , wherein the speech sample is determined to be scorable or non-scorable based upon an audio-quality filter.

11. The method of claim 10 , wherein the audio-quality filter identifies a likelihood that the speech sample is scorable or non-scorable based upon at least one of a spectrum metric, a power metric, a signal-to-noise ratio metric, a peak-speech-level metric, a frequency-contour metric, a jitter metric, a shimmer metric, or a pitch metric.

12. The method of claim 1 , wherein the speech sample is determined to be scorable or non-scorable based upon an insufficient-speech filter.

13. The method of claim 12 , wherein the insufficient-speech filter identifies a likelihood that the speech sample contains an insufficient amount of spoken speech based upon a spectrum metric, a voice activity detection metric, a silence metric, a pitch metric, a power metric, or an automated speech recognition metric.

14. The method of claim 1 , wherein the speech sample is determined to be scorable or non-scorable based upon an incorrect-language filter.

15. The method of claim 14 , wherein the incorrect-language filter identifies, based upon language-identification-based metrics, a likelihood that the speech sample is non-scorable because the sample includes speech of a language different than an expected language.

16. The method of claim 14 , wherein the incorrect-language filter uses a language identification method that is a combination of a phone recognizer and a plurality of language models, wherein each of the language models is associated with a different language.

17. The method of claim 1 , wherein the speech sample is determined to be scorable or non-scorable based upon a plagiarism detection filter.

18. The method of claim 17 , wherein the plagiarism detection filter identifies a likelihood that the speech sample is non-scorable based upon an automatic-speech-recognizer confidence metric or a similarity of the speech sample to a particular text.

19. A system for scoring non-native speech, comprising:
a processing system;
one or more non-transitory computer-readable storage mediums containing instructions configured to cause the processing system to perform operations including:
receiving a speech sample spoken by a non-native speaker to be scored by a scoring model, wherein the scoring model generates scores for speech samples based on one or more speech metrics;
performing automatic speech recognition on the speech sample to generate a transcript of the speech sample;
processing the speech sample to generate a plurality of speech metrics and a confidence score associated with one of the scoring model speech metrics;
applying a plurality of non-scorable response filters to the plurality of speech metrics and the confidence score;
determining whether the speech sample is scorable or non-scorable based upon the transcript and a collective application of the non-scorable response filters to the plurality of speech metrics and the confidence score, wherein the determining is based on assessment of audio quality of the speech sample, an amount of speech of the speech sample, a degree to which the speech sample is off-topic, and whether the speech sample includes speech from an incorrect language;
associating an indication of non-scorability with the speech sample when the sample is determined to be non-scorable; and
providing the sample to a scoring model for scoring when the sample is determined to be scorable.

20. A non-transitory computer program product for scoring non-native speech, tangibly embodied in a machine-readable non-transitory storage medium, including instructions configured to cause a processing system to execute steps including:
receiving a speech sample spoken by a non-native speaker to be scored by a scoring model, wherein the scoring model generates scores for speech samples based on one or more speech metrics;
performing automatic speech recognition on the speech sample to generate a transcript of the speech sample;
processing the speech sample to generate a plurality of speech metrics and a confidence score associated with one of the scoring model speech metrics;
applying a plurality of non-scorable response filters to the plurality of speech metrics and the confidence score;
determining whether the speech sample is scorable or non-scorable based upon the transcript and a collective application of the non-scorable response filters to the plurality of speech metrics and the confidence score, wherein the determining is based on assessment of audio quality of the speech sample, an amount of speech of the speech sample, a degree to which the speech sample is off-topic, and whether the speech sample includes speech from an incorrect language;
associating an indication of non-scorability with the speech sample when the sample is determined to be non-scorable; and
providing the sample to a scoring model for scoring when the

Assignees

• Educational Testing Service

Inventors

• Yoon Su-Youn
• Higgins Derrick
• Zechner Klaus
• Xie Shasha
• Jeon Je Hun
• Evanini Keelan
• Ling Guangming
• Bejar Isaac

Classifications

• G10L15/26

  Speech to text systems (G10L15/08 takes precedence) · CPC title

• G10L15/005

  Language recognition · CPC title

• G10L25/60

  for measuring the quality of voice signals · CPC title

• G10L25/90

  Pitch determination of speech signals · CPC title

• G10L25/78

  Detection of presence or absence of voice signals (switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10) · CPC title