Guided operator VR training

Patent metadata

Field	Value
Publication number	US-11928307-B2
Application number	US-202217693202-A
Country	US
Kind code	B2
Filing date	Mar 11, 2022
Priority date	Mar 11, 2022
Publication date	Mar 12, 2024
Grant date	Mar 12, 2024

Abstract

Aspects disclosed herein provide an operator training system that can train an operator about basic machine movements, operations, and applications, such as asphalt compactor rolling patterns, paving-by-numbers, milling-by-numbers, etc. with the use of a virtual reality headset at any point or time on a work site. The operator training system uses data received from the virtual reality headset and associated hand controls to guide the operator via training scenarios. The operator training system also includes providing immediate feedback on the outcome of the training scenario and may also identify one or more areas of improvement.

First claim

What is claimed is:

1. An operator training system for operator training at a work site on the operation of a machine, comprising:
a head mountable display device including a display screen;
at least one hand control including at least one navigation control; and
a controller in communication with the head mountable display device and the at least one hand control, the controller configured to:
generate a first set of graphical user interfaces associated with setup of an operator training application;
display the first set of graphical user interfaces via the display screen;
receive, from the at least one hand control, data based on at least one virtual object displayed via the first set of graphical user interfaces;
generate, based in part on the data, a second set of graphical user interfaces associated with at least one task associated with a training scenario corresponding to operating the machine on the work site, the second set of graphical user interfaces including one or more representations of at least a portion of the machine, the at least one task requires the operator to utilize the at least one hand control to interact with the one or more representations of at least a portion of the machine, and the second set of graphical user interfaces depict the operator's interactions with the one or more representations of at least a portion of the machine as they occur via the at least one hand control; and
display the second set of graphical user interfaces via the display screen.

2. The operator training system of claim 1 , wherein the machine includes an asphalt compactor.

3. The operator training system of claim 1 , wherein the head mountable display device includes a virtual reality headset and the at least one hand control includes a virtual reality hand control.

4. The operator training system of claim 1 , wherein the first set of graphical user interfaces include a plurality of virtual objects that are selectable via the at least one hand control.

5. The operator training system of claim 4 , wherein the at least one virtual object displayed via the first set of graphical user interfaces corresponds to the training scenario.

6. The operator training system of claim 1 , wherein the at least one task associated with the training scenario includes at least one of a machine walkaround task, a machine settings task, and a path overlay task.

7. The operator training system of claim 1 , wherein the controller is further configured to:
receive, from the at least one hand control, second data associated with the at least one task;
generate an assessment based in part on the second data; and
display a third set of graphical user interfaces including the assessment via the display screen.

8. The operator training system of claim 7 , wherein the assessment includes at least one of a score assessment and feedback.

9. The operator training system of claim 1 , wherein the training scenario includes at least one of an asphalt compactor rolling pattern scenario, a paving-by-numbers scenario, and a milling-by-numbers scenario.

10. A computer-implemented method for operator training at a work site on the operation of a machine, comprising:
generating, at a controller, a first set of graphical user interfaces associated with setup of an operator training application;
displaying, via a display screen of a head mountable display device in communication with the controller, the first set of graphical user interfaces;
receiving, from at least one hand control in communication with the controller, data based on at least one virtual object displayed via the first set of graphical user interfaces;
generating, based in part on the data, a second set of graphical user interfaces associated with at least one task associated with a training scenario corresponding to operating the machine on the work site, the second set of graphical user interfaces including one or more representations of at least a portion of the machine, the at least one task requires the operator to utilize the at least one hand control to interact with the one or more representations of at least a portion of the machine, and the second set of graphical user interfaces depict the operator's interactions with the one or more representations of at least a portion of the machine as they occur via the at least one hand control; and
displaying the second set of graphical user interfaces via the display screen.

11. The computer-implemented method of claim 10 , wherein the machine includes an asphalt compactor.

12. The computer-implemented method of claim 10 , wherein the head mountable display device includes a virtual reality headset and the at least one hand control includes a virtual reality hand control.

13. The computer-implemented method of claim 10 , wherein the first set of graphical user interfaces include a plurality of virtual objects that are selectable via the at least one hand control.

14. The computer-implemented method of claim 13 , wherein the at least one virtual object displayed via the first set of graphical user interfaces corresponds to the training scenario.

15. The computer-implemented method of claim 10 , wherein the at least one task associated with the training scenario includes at least one of a machine walkaround task, a machine settings task, and a path overlay task.

16. The computer-implemented method of claim 10 , further including:
receiving, from the at least one hand control, second data associated with the at least one task;
generating an assessment based in part on the second data, the assessment including at least one of a score assessment and feedback; and
displaying a third set of graphical user interfaces including the assessment via the display screen.

17. The computer-implemented method of claim 10 , wherein the training scenario includes at least one of an asphalt compactor rolling pattern scenario, a paving-by-numbers scenario, and a milling-by-numbers scenario.

18. A non-transitory computer-readable medium having instructions stored therein, wherein the instructions, when executed, cause a processor to:
generate, at a controller, a first set of graphical user interfaces associated with setup of an operator training application;
display, via a display screen of a head mountable display device in communication with the controller, the first set of graphical user interfaces;
receive, from at least one hand control in communication with the controller, data based on at least one virtual object displayed via the first set of graphical user interfaces;
generate, based in part on the data, a second set of graphical user interfaces associated with at least one task associated with a training scenario corresponding to operating a machine on a work site, the second set of graphical user interfaces including one or more representations of at least a portion of the machine, the at least one task requires the operator to utilize the at least one hand control to interact with the one or more representations of at least a portion of the machine, and the second set of graphical user interfaces depict the operator's interactions with the one or more representations of at least a portion of the machine as they occur via the at least one hand control;
display the second set of graphical user interfaces via the display screen; and
receive data from the at least one hand control indicating the operator's interaction with the one or more representations of at least a portion of the machine.

Assignees

• Caterpillar Paving Products Inc

Inventors

• King David A
• Mansell Todd Willis
• Remboldt Eric
• Peasley Matt

Classifications

• G06F3/04815 Primary

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title

• G02B27/017 Primary

  Head mounted · CPC title

• G06F3/014

  Hand-worn input/output arrangements, e.g. data gloves · CPC title

• G09B9/052

  characterised by provision for recording or measuring trainee's performance (devices for psychotechnics, e.g. for vehicle drivers A61B5/16, A61B5/18) · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title