Integration of optical area monitoring with industrial machine control

Patent metadata

Field	Value
Publication number	US-9256944-B2
Application number	US-201414553431-A
Country	US
Kind code	B2
Filing date	Nov 25, 2014
Priority date	May 19, 2014
Publication date	Feb 9, 2016
Grant date	Feb 9, 2016

Abstract

An industrial safety system is provided that integrates optical safety monitoring with machine control. The safety system includes an imaging sensor device supporting pixel array processing functions that allow time-of-flight (TOF) analysis to be performed on selected portions of the pixel array, while two-dimensional imaging analysis is performed on the remaining portions of the array, reducing processing load and response time relative to performing TOF analysis for all pixels of the array. The portion of the pixel array designated for TOF analysis can be pre-defined through configuration of the imaging sensor device, or can be dynamically selected based on object detection and classification by the two-dimensional imaging analysis. The imaging sensor device can also implement a number of safety and redundancy functions to achieve a high degree of safety integrity.

First claim

What is claimed is:

1. An optical safety system, comprising:
a memory that stores computer-executable components;
a processor, operatively coupled to the memory, that executes the computer-executable components, the computer-executable components comprising:
a pixel array component configured to, for one or more pixel arrays of one or more images of a hazardous industrial area captured by one or more imaging sensor devices, group pixels of the one or more pixel arrays to yield a subset of the pixels on which two-dimensional (2D) analysis is to be performed;
an image analysis component configured to perform 2D analysis on the subset of the pixels;
a distance determination component configured to perform 3D analysis on point cloud data received from the one or more imaging sensor devices; and
a hazard analysis and decision component configured to classify a first object in the one or more images as motorized industrial equipment and a second object in the one or more images as a human based on results of the 2D analysis and the 3D analysis, to correlate the results of the 2D analysis and the 3D analysis to yield correlated information relating to a relationship between the human and the motorized industrial equipment, and to generate a control output that controls the motorized industrial equipment based on the correlated information.

2. The system of claim 1 , wherein the point cloud data comprises distance information determined for selected pixels of the one or more pixel arrays by the one or more imaging sensor devices, and the distance determination component is configured to aggregate the point cloud data from the one or more imaging sensor devices to yield three-dimensional image data for the hazardous industrial area.

3. The system of claim 2 , wherein the hazard analysis and decision component is configured to classify the first object and the second object based on analysis of the three-dimensional image data and additional information comprising at least one of passive infrared data, localization data, object tagging data, or temperature sensing data.

4. The system of claim 3 , wherein the hazard analysis and decision component is configured to
estimate the first object and the second object as respective three-dimensional geometric objects within a three-dimensional space represented by the three-dimensional image data, wherein the geometric objects respectively encapsulate the first object and the second object, and
determine a minimum distance between the geometric objects to yield an estimated distance between the first object and the second object.

5. The system of claim 3 , wherein the hazard analysis and decision component is configured to
project the three-dimensional representations of the first object and the second object defined by the three-dimensional image data onto a two-dimensional plane to yield a first projection and a second projection; and
determine a minimum distance between the first projection and the second projection to yield an estimated distance between the first object and the second object.

6. The system of claim 5 , wherein the hazard analysis and decision component is further configured to
encapsulate the first projection and the second projection within respective geometrical objects, and
estimate the minimum distance as a distance between the geometrical objects.

7. The system of claim 1 , wherein the hazard analysis and decision component is further configured to identify a location of the human relative to the motorized industrial equipment based on at least one of the 2D analysis or the 3D analysis.

8. The system of claim 1 , further comprising a safety component configured to monitor one or more internal components of the one or more imaging sensor devices to detect a fault condition, and to at least one of generate a safety output or modify at least one of the results of the 2D analysis and the 3D analysis to compensate for the fault condition in response to detection of the fault condition based on one or more safety algorithms.

9. The system of claim 1 , wherein the fault condition comprises at least one of a temperature condition, a power condition, a vibration condition, or a component fault condition.

10. The system of claim 2 , wherein the correlated information comprises at least one of a location of the human relative to the motorized industrial equipment, a velocity of the human, a velocity of the motorized industrial equipment, an acceleration of the human, an acceleration of the motorized industrial equipment, a trajectory of the human, a trajectory of the motorized industrial equipment, or an anticipated intersection between the trajectory of the human and the trajectory of the motorized industrial equipment.

11. The system of claim 10 , wherein the hazardous analysis and decision component is further configured to predict a future location of at least one of the human or a component of the motorized industrial equipment within the hazardous industrial area based the correlated information, and to generate the control output based on the future location.

12. The system of claim 1 , further comprising an interface component configured to exchange data with an industrial controller over a network.

13. A method for controlling motorized industrial equipment, comprising:
collecting, by a safety system comprising at least one processor, image data from one or more imaging sensor devices that monitor a hazardous industrial zone;
performing two-dimensional (2D) imaging analysis on a set of pixels of the image data;
performing three-dimensional (3D) analysis on point cloud data comprising distance information for selected pixels of the image data calculated by the one or more imaging sensor devices;
classifying a first object in the image data as a human based on at least one of the 2D imaging analysis or the 3D analysis;
classifying a second object in the image data as motorized industrial equipment based on at least one of the 2D imaging analysis or the 3D analysis;
correlating results of the 2D imaging analysis and the 3D analysis to yield correlated information relating to a relationship between the human and the motorized equipment within the hazardous industrial zone; and
sending an instruction to alter operation of the motorized industrial equipment based on the correlated information.

14. The method of claim 13 , further comprising identifying a location of the human relative to the motorized industrial equipment based on at least one of the 2D imaging analysis or the 3D analysis.

15. The method of claim 13 , further comprising generating three-dimensional image data for the hazardous industrial zone based on aggregation of the point cloud data from the one or more imaging sensor devices.

16. The method of claim 13 , further comprising:
monitoring one or more internal components of the one or more imaging sensor devices to detect a fault condition, and
generating a safety output in response to detection of the fault condition based on one or more safety algorithms.

17. The method of claim 13 , wherein the correlating comprises determining, as the correlated information, at least one of a location of the human relative to the motorized industrial equipment, a velocity of the human, a velocity of the motorized industrial equipment, an acceleration of the human, an acceleration of the motorized industrial equipment, a trajectory of the human, a trajectory of the motorized industrial equipment, or an anticipated intersection between the trajectory of the human and the trajectory of the motorized industr

Assignees

• Rockwell Automation Tech Inc

Inventors

• Galera Richard
• Bowlby Anne
• Jones Derek W
• Pradhan Nilesh
• Leard Francis L

Classifications

• F16P3/142 Primary

  using image capturing devices · CPC title

• G06V10/80

  Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level (multimodal speaker identification or verification G10L17/10) · CPC title

• G06F18/24

  Classification techniques · CPC title

• G06F18/25

  Fusion techniques · CPC title

• G06T2207/30232

  Surveillance · CPC title