Generation of training data for verbal harassment detection

Patent metadata

Field	Value
Publication number	US-11620987-B2
Application number	US-202017135015-A
Country	US
Kind code	B2
Filing date	Dec 28, 2020
Priority date	Dec 31, 2019
Publication date	Apr 4, 2023
Grant date	Apr 4, 2023

Abstract

In some cases, one or more heuristics can be automatically generated using a small dataset of segments previously labeled by one or more domain experts. The generated one or more heuristics along with one or more patterns can be used to assign training labels to a large unlabeled dataset of segments. A subset of segments representing an occurrence of verbal harassment can be selected using the assigned training labels. Randomly selected segments can be used as being indicative of a non-occurrence of verbal harassment. The selected subset of segments and randomly selected segments can be used to train one or more machine learning models for verbal harassment detection.

First claim

The invention claimed is:

1. A computer-implemented method of training a machine learning model for detection of verbal harassment, the method comprising:
by one or more hardware processors:
determining a plurality of verbal harassment heuristics using a first plurality of speech segments, the segments of the first plurality of speech segments previously labeled with an occurrence of verbal harassment or a non-occurrence of verbal harassment;
determining a plurality of labels for a second plurality of speech segments by applying the plurality of verbal harassment heuristics and a plurality of verbal harassment patterns, the segments of the second plurality of speech segments not previously labeled with the occurrence or the non-occurrence of verbal harassment;
aggregating the plurality of labels into a plurality of likelihoods for the occurrence of verbal harassment;
selecting a subset of segments from the second plurality of speech segments based on comparing the plurality of likelihoods to at least one threshold; and
training a machine learning model for verbal harassment detection using the subset of segments from the second plurality of speech segments and a plurality of randomly selected segments, wherein the plurality of randomly selected segments comprises only training data indicative of the non-occurrence of verbal harassment.

2. The method of claim 1 , wherein at least one of the first plurality of speech segments, the second plurality of speech segments, or the plurality of randomly selected segments comprise text data.

3. The method of claim 2 , wherein text data has been obtained by applying automatic speech recognition to audio data.

4. The method of claim 1 , wherein a number of segments in the second plurality of speech segments is larger than a number of segments in the first plurality of speech segments.

5. The method of claim 1 , wherein determining the plurality of labels for the second plurality of speech segments comprises determining more than one label for at least one segment of the second plurality of speech segments.

6. The method of claim 5 , wherein aggregating the plurality of labels comprises selecting a single label for the at least one segment of the second plurality of speech segments.

7. The method of claim 1 , wherein the first and second plurality of speech segments comprise speech segments recorded in one or more vehicles, and wherein the machine learning model is configured to detect verbal harassment detection in a vehicle.

8. The method of claim 1 , wherein the subset of segments from the second plurality of speech segments comprises only training data indicative of the occurrence of verbal harassment.

9. The method of claim 1 , wherein the at least one threshold is equal to or greater than 0.9.

10. The method of claim 1 , wherein the segments of the first plurality of speech segments comprise manually-generated labels.

11. The method of claim 1 , wherein the subset of segments from the second plurality of speech segments represents training data indicative of the occurrence of verbal harassment.

12. The method of claim 1 , wherein the machine learning model for verbal harassment detection comprises a text classification machine learning model.

13. The method of claim 12 , wherein the text classification machine learning model comprises at least one of hierarchical attention model, a fastText model, or a convolutional neural network model.

14. The method of claim 1 , the segments of the first plurality of speech segments have been previously manually labeled with the occurrence of verbal harassment or the non-occurrence of verbal harassment.

15. A non-transitory computer readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform a method comprising:
determining a plurality of verbal harassment heuristics using a first plurality of speech segments, the segments of the first plurality of speech segments previously labeled with an occurrence of verbal harassment or a non-occurrence of verbal harassment;
determining a plurality of labels for a second plurality of speech segments by applying the plurality of verbal harassment heuristics and a plurality of verbal harassment patterns, the segments of the second plurality of speech segments not previously labeled with the occurrence or the non-occurrence of verbal harassment;
aggregating the plurality of labels into a plurality of likelihoods for the occurrence of verbal harassment;
selecting a subset of segments from the second plurality of speech segments based on comparing the plurality of likelihoods to at least one threshold; and
training a machine learning model for verbal harassment detection using the subset of segments from the second plurality of speech segments and a plurality of randomly selected segments, wherein the plurality of randomly selected segments comprises only training data indicative of the non-occurrence of verbal harassment.

16. The medium of claim 15 , wherein determining the plurality of labels for the second plurality of speech segments comprises determining more than one label for at least one segment of the second plurality of speech segments.

17. The medium of claim 16 , wherein aggregating the plurality of labels comprises selecting a single label for the at least one segment of the second plurality of speech segments.

18. The medium of claim 15 , wherein the segments of the first plurality of speech segments comprise manually-generated labels.

19. The medium of claim 15 , wherein the subset of segments from the second plurality of speech segments represents training data indicative of the occurrence of verbal harassment.

20. The medium of claim 15 , wherein the machine learning model for verbal harassment detection comprises a text classification machine learning model.

Assignees

• Beijing Didi Infinity Technology & Dev Co Ltd

Inventors

• Lyu Ying
• Han Kun

Classifications

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06F40/30

  Semantic analysis · CPC title

• G06F40/216 Primary

  using statistical methods · CPC title