System and method for managing mixed fleet worksites using video and audio analytics

Patent metadata

Field	Value
Publication number	US-9685009-B2
Application number	US-201514676243-A
Country	US
Kind code	B2
Filing date	Apr 1, 2015
Priority date	Apr 1, 2015
Publication date	Jun 20, 2017
Grant date	Jun 20, 2017

Abstract

Systems and methods for managing and optimizing mixed fleet worksite operations based on video and or audio data are disclosed. One method includes receiving one or more models relating to a fleet of machines at the worksite, wherein the fleet of machines comprises an in-network machine and an out-of-network machine, receiving first sensor data associated with the out-of-network machine at the worksite, receiving second sensor data associated with the in-network machine at the worksite, determining a machine state of each of the in-network machine and the out-of-network machine based at least on the first sensor data and the second sensor data, comparing the determined machine states to a modeled machine state represented by the received one or more models to classify site operations and/or detect an irregularity in site operations or an inefficiency in site operations, and generating a response based at least on the detected irregularity or inefficiency.

First claim

We claim:

1. A method for managing mixed-fleet worksites, comprising:
receiving, by one or more processors of a central station, one or more models relating to a worksite or a fleet of machines at the worksite, the fleet of machines including an in-network machine configured to communicate with the central station, and an out-of-network machine that is not configured to communicate with the central station;
receiving, by the one or more processors of the central station, first sensor data associated with the out-of-network machine at the worksite, the first sensor data comprising one or more of image data and audio data;
receiving, by the one or more processors of the central station, second sensor data associated with the in-network machine at the worksite;
determining, by the one or more processors, a machine state of each of the in-network machine and the out-of-network machine, the machine state of the out-of-network machine based at least on a comparison of a feature of an object detected in the image data of the first sensor data to a signature that represents a known or learned machine state, the machine state of the in-network machine based at least on the second sensor data, the machine state one of a full load, an empty load, a payload material type, a payload to air ratio, a payload placement, a payload compaction, a payload water content, a material amount moved, a drop placement, an excavator position, an idle state, a swing state, a dump position, a deformation of machine, an operator characteristic and a ground crew location relative to machine;
comparing the determined machine states to a modeled machine state represented by the received one or more models to detect a non-failure irregularity in site operations or an inefficiency in site operations;
generating a response based at least on the detected irregularity or inefficiency, the response including a warning or a remote reconfiguration of operational parameters of the in-network machine; and
transmitting the response to an operator, a display at the worksite, or to the in-network machine,
wherein the one or more models include a machine model, a payload model or a worksite model
wherein the worksite includes a mining site or a construction site,
wherein the out-of-network machine is an out-of-network digging machine, an out-of-network loading machine or an out-of-network hauling machine,
wherein the in-network machine is an in-network digging machine, an in-network loading machine or an in-network hauling machine,
wherein the out-of-network hauling machine and in-network hauling machine are each configured to carry excavated materials between different locations at the worksite.

2. A computer program product comprising a non-transitory storage medium having a computer readable program code embodied therein, the computer readable program code adapted to be executed to implement a method for managing a mixed-fleet worksite, the method comprising:
receiving, by the one or more processors, one or more models relating to a worksite or a fleet of machines at the worksite, the fleet of machines including an in-network machine configured to communicate with a central station and an out-of-network machine that is not configured to communicate with the central station;
receiving, by the one or more processors, first sensor data associated with the out-of-network machine at the worksite, the first sensor data comprising one or more of image data and audio data;
receiving, by the one or more processors, second sensor data associated with the in-network machine at the worksite;
determining, by the one or more processors, a machine state of each of the in-network machine and the out-of-network machine, the machine state of the out-of-network machine based at least on a comparison of a feature of an object detected in the image data of the first sensor data to a signature that represents a known or learned machine state, the machine state of the in-network machine based at least on the second sensor data, the machine state one of a full load, an empty load, a payload material type, a payload to air ratio, a payload placement, a payload compaction, a payload water content, a material amount moved, a drop placement, an excavator position, an idle state, a swing state, a dump position, a deformation of machine, an operator characteristic and a ground crew location relative to machine;
comparing the determined machine states to a modeled machine state represented by the received one or more models to detect a non-failure irregularity in site operations or an inefficiency in site operations;
generating a response based at least on the detected irregularity or inefficiency, the response including a warning or a remote reconfiguration of operational parameters of the in-network machine; and
transmitting the response to an operator, a display at the worksite, or to the in-network machine,
wherein the one or more models include a machine model, a payload model or a worksite model
wherein the worksite includes a mining site or a construction site,
wherein the out-of-network machine is an out-of-network digging machine, an out-of-network loading machine or an out-of-network hauling machine,
wherein the in-network machine is an in-network digging machine, an in-network loading machine or an in-network hauling machine,
wherein the out-of-network hauling machine and in-network hauling machine are each configured to carry excavated materials between different locations on the worksite.

3. The method of claim 1 , wherein the one or more models comprises a worksite model and the method further comprises updating the worksite model based at least on the generated response.

4. The method of claim 1 , wherein the first sensor data is received via an onsite sensor and the second sensor data is received via an offsite sensor.

5. The method of claim 1 , wherein the first sensor data is received via an offsite sensor and the second sensor data is received via an onsite sensor.

6. The method of claim 1 , wherein one or more of the determined machine state and the modeled machine state comprise a characteristic of a material load.

7. The method of claim 1 , wherein the determining the machine state comprises generating a current signature representing the machine state.

8. The method of claim 1 , wherein the response comprises one or more of an audible indicator and a visual indicator.

9. A system comprising:
a processor configured to
receive one or more models relating to a worksite or a fleet of machines at the worksite, the fleet of machines including an in-network machine configured to communicate with a central station and an out-of-network machine that is not configured to communicate with the central station;
receive first sensor data associated with the out-of-network machine at the worksite, the first sensor data comprising one or more of image data and audio data;
receive second sensor data associated with the in-network machine at the worksite;
determine a machine state of each of the in-network machine and the out-of-network machine, the machine state of the out-of-network machine based at least on a comparison of a feature of an object detected in the image data of the first sensor data to a signature that represents a known or learned machine state, the machine state of the in-network machine based at least on the second sensor data, the machine state one of a full load, an empty load, a payload material type, a payload to air ratio, a payload placement, a payload compaction, a payload water content, a material amount moved, a drop placement, an excavator position, an idle state, a swing state, a dump position, a deformation of machine, an operator characteristic and a g

Assignees

• Caterpillar Inc

Inventors

• Sprock Christopher
• Baumann Ryan
• Talmaki Sanat
• Halepatali Praveen

Classifications

• G07C5/0866

  the electronic data carrier being a digital video recorder in combination with video camera · CPC title

• G07C5/008 Primary

  communicating information to a remotely located station (transmission systems for measured values G08C) · CPC title

• G06Q10/0639

  Performance analysis of employees; Performance analysis of enterprise or organisation operations · CPC title

• G06Q50/08

  Construction · CPC title