Method and device for detecting containers which have fallen over and/or are damaged in a container mass flow

Patent metadata

Field	Value
Publication number	US-12564867-B2
Application number	US-202017756744-A
Country	US
Kind code	B2
Filing date	Nov 16, 2020
Priority date	Dec 3, 2019
Publication date	Mar 3, 2026
Grant date	Mar 3, 2026

Abstract

Method for detecting containers which have fallen over and/or are damaged in a container mass flow, wherein the containers in the container mass flow are transported vertically on a transporter, wherein the container mass flow is captured as an image data stream using at least one camera, and wherein the image data stream is evaluated by an image processing unit, wherein the image data stream is evaluated by the image processing unit using a deep neural network in order to detect and locate the containers which have fallen over and/or are damaged.

First claim

The invention claimed is:

1 . A method for detecting containers which have fallen over and/or are damaged in a container mass flow, wherein the containers of the container mass flow are transported vertically on a transporter, wherein the container mass flow is captured using at least one camera as an image data flow, and wherein the image data flow is evaluated by an image processing unit,
wherein the image data flow is evaluated by the image processing unit using a deep neural network in order to detect and locate the containers which have fallen over and/or are damaged, the deep neural network is trained with a training data set comprising images of fallen over and/or damaged containers, such that the deep neural network develops a model based on the training data set to distinguish the fallen over and/or damaged containers from each other, the fallen over and/or damaged containers are characterized or marked in the images of the training data set and/or in metadata of the images, the images of the training data set are automatically duplicated to create further images with additional combinations of fallen over and/or damaged containers, image segments are generated during the duplication, each of the image segments comprising a vertical or fallen over and/or damaged container, and the image segments are individually rotated and/or enlarged during the duplication.

2 . The method according to claim 1 , wherein the training data set comprises images of the fallen over and/or damaged containers with different container types.

3 . The method according to claim 2 , wherein at least one of the images of the training data set comprises a combination of different container types.

4 . The method according to claim 1 , wherein the training data set comprises images of the fallen over and/or damaged containers with different ambient conditions.

5 . The method according to claim 1 , wherein in the images of the training data set and/or in the metadata of the images the fallen over and/or damaged containers are characterized by at least one surrounding box.

6 . The method according to claim 1 , wherein at least one exposure parameter is changed in the image segments during the duplication.

7 . The method according to claim 1 , wherein the fallen over and/or damaged containers are automatically separated from vertically transported containers of the container mass flow after detection and localization by the deep neural network.

8 . The method according to claim 1 , wherein the image data flow is continuously captured and divided into individual images by means of a sliding window, and wherein the individual images are subsequently evaluated with the deep neural network.

9 . The method according to claim 1 , wherein the transporter is configured as a mass transporter with which the containers are transported in multiple rows.

10 . A device for detecting containers which have fallen over and/or are damaged in a container mass flow for carrying out the method according to claim 9 , with
a transporter for the vertical transport of the containers of the container mass flow, at least one camera to capture the container mass flow as an image data stream, and with an image processing unit to evaluate the image data stream, wherein the image processing unit comprises a deep neural network for evaluating the image data stream in order to detect and locate the containers which have fallen over and/or are damaged, the deep neural network is trained with a training data set comprising images of fallen over and/or damaged containers, such that the deep neural network develops a model based on the training data set to distinguish the fallen over and/or damaged containers from each other, the fallen over and/or damaged containers are characterized or marked in the images of the training data set and/or in metadata of the images, the images of the training data set are automatically duplicated to create further images with additional combinations of fallen over and/or damaged containers, image segments are generated during the duplication, each of the image segments comprising a vertical or fallen over and/or damaged container, and the image segments are individually rotated and/or enlarged during the duplication.

11 . The device according to claim 10 , wherein the image processing unit comprises a storage medium containing machine instructions that, when executed with the image processing unit, evaluate the image data stream with the deep neural network.

12 . The method according to claim 4 , wherein the ambient conditions are illumination conditions.

13 . The method according to claim 7 , wherein the fallen over and/or damaged containers are automatically separated from the vertically transported containers of the container mass flow after detection and localization by the deep neural network with a gripper arm or with a switch.

Assignees

• Krones Ag

Inventors

• Awiszus Stefan
• Bayer Udo
• Paukert Josef
• Siddiqui Aurangzaib Ahmed

Classifications

• G06T2207/30108

  Industrial image inspection · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/10152

  Varying illumination · CPC title

• G06T2207/10144

  Varying exposure · CPC title