Enhanced collection of training data for machine learning to improve worksite safety and operations

Patent metadata

Field	Value
Publication number	US-11636359-B2
Application number	US-202016902107-A
Country	US
Kind code	B2
Filing date	Jun 15, 2020
Priority date	Jun 15, 2020
Publication date	Apr 25, 2023
Grant date	Apr 25, 2023

Abstract

Systems and methods for enhanced collection of training data for machine learning to improve worksite safety and operations. One embodiment is a system with an interface to instruct a user device of a user to capture first sensor data of an object performing one or more known actions. The system also includes a training controller to allocate points as an award to the user of the user device for correctly capturing the object performing the one or more known actions, and to generate an instruction for the user to capture second sensor data of the object performing one or more unknown actions if the user has exceeded a threshold of awarded points. The training controller trains a machine learning function based on the second sensor data of the object performing the one or more unknown actions as identified by the user having exceeded the threshold of awarded points.

First claim

What is claimed is:

1. A system for training a machine learning function, the system comprising:
a processor that implements a training controller configured to generate training data for training the machine learning function to automatically detect safety hazards in a worksite environment; and
an interface configured to instruct a user via a user device to capture first sensor data of an object performing one or more known actions at the worksite environment, and to receive the first sensor data captured by the user device;
the training controller configured to determine whether the first sensor data correctly depicts the object performing the one or more known actions with an artificial intelligence engine, and to allocate points as an award to the user of the user device for correctly capturing the object performing the one or more known actions;
in response to determining that the user has not exceeded a threshold of awarded points, the training controller is configured to instruct the user via the user device to continue to capture the first sensor data related to known actions;
in response to determining that the user has exceeded the threshold of awarded points, the training controller is configured to instruct the user via the user device to capture second sensor data of the object performing one or more unknown actions at the worksite environment, and to classify the one or more unknown actions;
the interface is configured to receive, as user-classified data, the second sensor data captured and a classification from the user of the one or more unknown actions performed by the object; and
the training controller is configured to refine the training data with the user-classified data, and to train the machine learning function using the training data as refined with the user-classified data.

2. The system of claim 1 wherein:
the training controller is configured to calculate a rating of the user in correctly capturing the object performing the one or more known actions in comparison to other users, and to instruct the user via the user device to capture the second sensor data based on the rating of the user.

3. The system of claim 2 wherein:
the training controller is configured to generate an instruction for the user to capture the second sensor data based on the rating of the user, wherein the instruction requests the user to capture the second sensor data with a target sensor.

4. The system of claim 1 wherein:
the interface is configured to receive a user confidence level of correctly classifying the one or more unknown actions, and
the training controller is configured to determine whether to refine the training data with the user-classified data based on the user confidence level of the user.

5. The system of claim 1 wherein:
the training controller is configured to determine whether the user belongs to an expert user domain associated with correctly capturing the object performing the one or more known actions, and to instruct the user via the user device to capture the second sensor data based on the user belonging to the expert user domain.

6. The system of claim 1 wherein:
the training controller is configured to determine a correlation between a location of acquiring the first sensor data and performance of the user in correctly capturing the object performing the one or more known actions, and to instruct the user via the user device to capture the second sensor data based on the correlation.

7. A method of training a machine learning function to automatically detect safety hazards in a worksite environment, the method comprising:
generating training data for training the machine learning function by:
instructing a user via a user device to capture first sensor data of an object performing one or more known actions at the worksite environment;
receiving the first sensor data captured by the user device;
determining whether the first sensor data correctly depicts the object performing the one or more known actions with an artificial intelligence engine;
allocating points as an award to the user of the user device for correctly capturing the object performing the one or more known actions;
in response to determining that the user has not exceeded a threshold of awarded points, instructing the user via the user device to continue to capture the first sensor data related to known actions;
in response to determining that the user has exceeded the threshold of awarded points, instructing the user via the user device to capture second sensor data of the object performing one or more unknown actions at the worksite environment, and to classify the one or more unknown actions;
receiving, as user-classified data, the second sensor data captured by the user device and a classification from the user of the one or more unknown actions performed by the object; and
refining the training data with the user-classified data; and

training the machine learning function using the training data as refined with the user-classified data.

8. The method of claim 7 further comprising:
calculating a rating of the user in correctly capturing the object performing the one or more known actions in comparison to other users; and
instructing the user via the user device to capture the second sensor data based on the rating of the user.

9. The method of claim 8 further comprising:
generating an instruction for the user to capture the second sensor data based on the rating of the user, wherein the instruction requests the user to capture the second sensor data with a target sensor.

10. The method of claim 7 further comprising:
receiving a user confidence level of correctly classifying the one or more unknown actions; and
determining whether to refine the training data with the user-classified data based on the user confidence level of the user.

11. The method of claim 7 further comprising:
determining whether the user belongs to an expert user domain associated with correctly capturing the object performing the one or more known actions; and
instructing the user via the user device to capture the second sensor data based on the user belonging to the expert user domain.

12. The method of claim 7 further comprising:
determining a correlation between a location of acquiring the first sensor data and performance of the user in correctly capturing the object performing the one or more known actions; and
instructing the user via the user device to capture the second sensor data based on the correlation.

13. The method of claim 7 further comprising:
determining a correlation between a timing of acquiring the first sensor data and performance of the user in correctly capturing the object performing the one or more known actions; and
instructing the user via the user device to capture the second sensor data based on the correlation.

14. The method of claim 7 further comprising:
determining a correlation between a sensor type of the first sensor data and performance of the user in correctly capturing the object performing the one or more known actions; and
instructing the user via the user device to capture the second sensor data based on the correlation.

15. A non-transitory computer readable medium embodying programmed instructions executed by a processor, wherein the instructions direct the processor to perform a method of training a machine learning function to automatically detect safety hazards in a worksite environment, the method comprising:
generating training data for training the machine learning function by:
instructing a user via a user device to capture first s

Assignees

• Boeing Co

Inventors

• Laughlin Brian D
• Kelsey William D

Classifications

• G06Q10/105 Primary

  Human resources · CPC title

• G06Q10/06398

  Performance of employee with respect to a job function · CPC title

• G06Q50/2057

  Career enhancement or continuing education service · CPC title

• G09B19/00

  Teaching not covered by other main groups of this subclass (teaching or practice apparatus for gun-aiming or gun-laying F41G3/26) · CPC title

• F16P3/14

  the means being photocells or other devices sensitive without mechanical contact · CPC title