System and Method for Managing Mixed Fleet Worksites Using Video and Audio Analytics

Patent metadata

Field	Value
Publication number	US-2016292933-A1
Application number	US-201514676243-A
Country	US
Kind code	A1
Filing date	Apr 1, 2015
Priority date	Apr 1, 2015
Publication date	Oct 6, 2016
Grant date	—

Abstract

Systems and methods for managing and optimizing mixed fleet worksite operations based on video and or audio data are disclosed. One method includes receiving one or more models relating to a fleet of machines at the worksite, wherein the fleet of machines comprises an in-network machine and an out-of-network machine, receiving first sensor data associated with the out-of-network machine at the worksite, receiving second sensor data associated with the in-network machine at the worksite, determining a machine state of each of the in-network machine and the out-of-network machine based at least on the first sensor data and the second sensor data, comparing the determined machine states to a modeled machine state represented by the received one or more models to classify site operations and/or detect an irregularity in site operations or an inefficiency in site operations, and generating a response based at least on the detected irregularity or inefficiency.

First claim

We claim:

1 . A method comprising:
receiving, by one or more processors, one or more models relating to a worksite or a fleet of machines at the worksite, wherein the fleet of machines comprises an in-network machine and an out-of-network machine; receiving, by the one or more processors, first sensor data associated with the out-of-network machine at the worksite, the first sensor data comprising one or more of image data and audio data; receiving, by the one or more processors, second sensor data associated with the in-network machine at the worksite; determining, by the one or more processors, a machine state of each of the in-network machine and the out-of-network machine based at least on the first sensor data and the second sensor data; comparing the determined machine states to a modeled machine state represented by the received one or more models to detect an irregularity in site operations or an inefficiency in site operations; and generating a response based at least on the detected irregularity or inefficiency.

2 . The method of claim 1 , further comprising transmitting the response to one or more of an operator and a central station.

3 . The method of claim 1 , wherein the one or more models comprises a worksite model and further comprising updating the worksite model based at least on the generated response.

4 . The method of claim 1 , wherein the first sensor data is received via an onsite sensor and the second sensor data is received via an offsite sensor.

5 . The method of claim 1 , wherein the first sensor data is received via an offsite sensor and the second sensor data is received via an onsite sensor.

6 . The method of claim 1 , wherein one or more of the determined machine state and the modeled machine state comprise a characteristic of a material load.

7 . The method of claim 1 , wherein the determining the machine state comprises generating a signature representing the machine state.

8 . The method of claim 1 , wherein the response comprises one or more of an audible indicator and a visual indicator.

9 . A system comprising:
a processor; and a memory bearing instructions that, upon execution by the processor, cause the system at least to:
receive one or more models relating to a worksite or a fleet of machines at the worksite, wherein the fleet of machines comprises an in-network machine and an out-of-network machine;
receiving, by the one or more processors, first sensor data associated with the out-of-network machine at the worksite, the first sensor data comprising one or more of image data and audio data;
receive second sensor data associated with the in-network machine at the worksite;
determine a machine state of each of the in-network machine and the out-of-network machine based at least on the first sensor data and the second sensor data; and
compare the determined machine states to a modeled machine state represented by the received one or more models to classify one or more site operations.

10 . The system of claim 9 , wherein the one or more models comprises a worksite model and further comprising instructions that, upon execution by the processor, cause the system at least to update the worksite model based at least on the classified one or more site operations.

11 . The system of claim 9 , wherein the first sensor data is received via an onsite sensor and the second sensor data is received via an offsite sensor.

12 . The system of claim 9 , wherein the first sensor data is received via an offsite sensor and the second sensor data is received via an onsite sensor.

13 . The system of claim 9 , wherein one or more of the determined machine state and the modeled machine state comprise a characteristic of a material load.

14 . The system of claim 9 , wherein the determining the machine state comprises generating a signature representing the machine state.

15 . A computer readable storage medium bearing instructions that, upon execution by one or more processors, effectuate operations comprising:
receiving, by the one or more processors, one or more models relating to a worksite or a fleet of machines at the worksite, wherein the fleet of machines comprises an in-network machine and an out-of-network machine; receiving, by the one or more processors, first sensor data associated with the out-of-network machine at the worksite, the first sensor data comprising one or more of image data and audio data; receiving, by the one or more processors, second sensor data associated with the in-network machine at the worksite; determining, by the one or more processors, a machine state of each of the in-network machine and the out-of-network machine based at least on the first sensor data and the second sensor data; comparing the determined machine states to a modeled machine state represented by the received one or more models to detect an irregularity in site operations or an inefficiency in site operations; and generating a response based at least on the detected irregularity or inefficiency.

16 . The computer readable storage medium of claim 15 , wherein the first sensor data is received via an onsite sensor and the second sensor data is received via an offsite sensor.

17 . The computer readable storage medium of claim 15 , wherein the first sensor data is received via an offsite sensor and the second sensor data is received via an onsite sensor.

18 . The computer readable storage medium of claim 15 , wherein one or more of the determined machine state and the modeled machine state comprise a characteristic of a material load.

19 . The computer readable storage medium of claim 15 , wherein the detecting an irregularity in site operations or an inefficiency in site operations comprises comparing the determined machine state to one or more models relating to the worksite.

20 . The computer readable storage medium of claim 15 , further comprising transmitting the response to one or more of an operator and a central station.

Assignees

• Caterpillar Inc

Inventors

• Sprock Christopher
• Baumann Ryan
• Talmaki Sanat
• Halepatali Praveen

Classifications

• G06Q10/0639

  Performance analysis of employees; Performance analysis of enterprise or organisation operations · CPC title

• G07C5/008 Primary

  communicating information to a remotely located station (transmission systems for measured values G08C) · CPC title

• G07C5/0866

  the electronic data carrier being a digital video recorder in combination with video camera · CPC title

• G06Q50/08

  Construction · CPC title