Vision system for leg detection

Patent metadata

Field	Value
Publication number	US-11832582-B2
Application number	US-201716092333-A
Country	US
Kind code	B2
Filing date	Aug 17, 2017
Priority date	Aug 17, 2016
Publication date	Dec 5, 2023
Grant date	Dec 5, 2023

Abstract

A leg ( 205 ) detection system comprising: a robotic arm ( 200 ) comprising a gripping portion ( 208 ) for holding a teat cup ( 203, 210 ) for attaching to a teat ( 1102, 1104, 1106, 1108, 203 S, 203) of a dairy livestock ( 200, 202, 203 ); an imaging system coupled to the robotic arm ( 200 ) and configured to capture a first three-dimensional (3D) image ( 138, 2400, 2500 ) of a rearview of the dairy livestock ( 200, 202, 203 ) in a stall ( 402 ), the imaging system comprising a 3D camera ( 136, 138 ) or a laser ( 132 ), wherein each pixel of the first 3D image ( 138, 2400, 2500 ) is associated with a depth value; one or more memory ( 104 ) devices configured to store a reference (3D) 3D image ( 138, 2400, 2500 ) of the stall ( 402 ) without any dairy livestock ( 200, 202, 203 ); and a processor ( 102 ) communicatively coupled to the imaging system and the one or more memory ( 104 ) devices, the processor ( 102 ) configured to: access the first 3D image ( 138, 2400, 2500 ) and the reference (3D) 3D image ( 138, 2400, 2500 ); subtract the first 3D image ( 138, 2400, 2500 ) from the reference (3D) 3D image ( 138, 2400, 2500 ) to produce a second 3D image ( 138, 2400, 2500 ); perform morphological image ( 138, 2400, 2500 ) processing on the second 3D image ( 138, 2400, 2500 ) to produce a third 3D image ( 138, 2400, 2500 ); perform image ( 138, 2400, 2500 ) thresholding on the third 3D image ( 138, 2400, 2500 ) to produce a fourth 3D image ( 138, 2400, 2500 ); cluster ( 2616, 2618, 2626, 2628 ) data from the fourth 3D image ( 138, 2400, 2500 ); identify, using the clustered data from the fourth 3D image ( 138, 2400, 2500 ), one or more legs ( 205 ) of the dairy livestock ( 200, 202, 203 ); and provide instructions for movements of the robotic arm ( 200 ) to avoid the identified one or more legs ( 205 ) while attaching the teat cup ( 203, 210 ) to the teat ( 1102, 1104, 1106, 1108, 203 S, 203 ) of the dairy livestock ( 200, 202, 203 ).

First claim

The invention claimed is:

1. A leg detection system comprising:
a robotic arm comprising a gripping portion for holding a teat cup for attaching to a teat of a dairy livestock;
an imaging system coupled to the robotic arm and configured to capture a first three-dimensional (3D) image of a stall with a rearview of the dairy livestock in the stall, the imaging system comprising a 3D camera or a laser, wherein each pixel of the first 3D image is associated with a depth value;
one or more memory devices configured to store a reference 3D image of the stall without any dairy livestock; and
a processor communicatively coupled to the imaging system and the one or more memory devices, the processor configured to:
access the first 3D image and the reference 3D image;
subtract the first 3D image from the reference 3D image to produce a second 3D image of the rearview of the dairy livestock;
perform morphological image processing on the second 3D image to produce a third 3D image;
perform image thresholding on the third 3D image to produce a fourth 3D image;
cluster data from the fourth 3D image;
identify, using the clustered data from the fourth 3D image, one or more legs of the dairy livestock; and
provide instructions for movements of the robotic arm to avoid the identified one or more legs while attaching the teat cup to the teat of the dairy livestock.

2. The leg detection system of claim 1 , wherein performing morphological image processing on the second 3D image to produce the third 3D image comprises using an image erosion algorithm.

3. The leg detection system of claim 1 , wherein performing image thresholding on the third 3D image to produce the fourth 3D image comprises using adaptive thresholding.

4. The leg detection system of claim 1 , wherein performing image thresholding on the third 3D image to produce the fourth 3D image comprises:
identifying a plurality of pixels in the third 3D image;
for each particular pixel of the plurality of pixels:
if a value of the particular pixel is less than a threshold value, set the value of the particular pixel to a background value; else
set the value of the particular pixel to a foreground value.

5. The leg detection system of claim 1 , wherein clustering data from the fourth 3D image comprises using k-means clustering.

6. The leg detection system of claim 1 , wherein clustering data from the fourth 3D image comprises:
forming a plurality of pixel vectors;
performing clustering on each row of the fourth 3D image using the plurality of pixel vectors; and
performing clustering on each column of the fourth 3D image using the plurality of pixel vectors.

7. The leg detection system of claim 6 , wherein identifying the one or more legs of the dairy livestock comprises:
dividing the fourth 3D image into a left side and a right side;
identifying a left largest cluster within the left side;
identifying a right largest cluster within the right side;
identifying a location of a left leg of the dairy livestock as corresponding to the left largest cluster; and
identifying a location of a right leg of the dairy livestock as corresponding to the right largest cluster.

8. The leg detection system of claim 1 , the processor further configured to remove any pixels from the second 3D image that are not within a predetermined distance of the 3D camera, the pixels being removed prior to performing the morphological image processing on the second 3D image.

9. A leg detection method, comprising:
accessing, by a processor, a first 3D image of a stall with a rearview of a dairy livestock in the stall, wherein each pixel of the first 3D image is associated with a depth value;
accessing, by the processor, a reference 3D image of the stall without any dairy livestock;
subtracting, by the processor, the first 3D image from the reference 3D image to produce a second 3D image of the rearview of the dairy livestock;
performing, by the processor, morphological image processing on the second 3D image to produce a third 3D image;
performing, by the processor, image thresholding on the third 3D image to produce a fourth 3D image;
clustering, by the processor, data from the fourth 3D image; and
identifying, by the processor using the clustered data from the fourth 3D image, one or more legs of the dairy livestock.

10. The leg detection method of claim 9 , wherein performing morphological image processing on the second 3D image to produce the third 3D image comprises using an image erosion algorithm.

11. The leg detection method of claim 9 , wherein performing image thresholding on the third 3D image to produce the fourth 3D image comprises using adaptive thresholding.

12. The leg detection method of claim 9 , wherein performing image thresholding on the third 3D image to produce the fourth 3D image comprises:
identifying a plurality of pixels in the third 3D image;
for each particular pixel of the plurality of pixels:
if a value of the particular pixel is less than a threshold value, set the value of the particular pixel to a background value; else
set the value of the particular pixel to a foreground value.

13. The leg detection method of claim 9 , wherein clustering data from the fourth 3D image comprises using k-means clustering.

14. The leg detection method of claim 9 , wherein clustering data from the fourth 3D image comprises:
forming a plurality of pixel vectors;
performing clustering on each row of the fourth 3D image using the plurality of pixel vectors; and
performing clustering on each column of the fourth 3D image using the plurality of pixel vectors.

15. The leg detection method of claim 14 , wherein identifying the one or more legs of the dairy livestock comprises:
dividing the fourth 3D image into a left side and a right side;
identifying a left largest cluster within the left side;
identifying a right largest cluster within the right side;
identifying a location of a left leg of the dairy livestock as corresponding to the left largest cluster; and
identifying a location of a right leg of the dairy livestock as corresponding to the right largest cluster.

16. The leg detection method of claim 9 , further comprising removing, by the processor, any pixels from the second 3D image that are not within a predetermined distance of the 3D camera, the pixels being removed prior to performing the morphological image processing on the second 3D image.

17. One or more computer-readable non-transitory storage media comprising software that is operable when executed by one or more processors to:
access a first 3D image of a stall with a rearview of a dairy livestock in the stall, wherein each pixel of the first 3D image is associated with a depth value;
access a reference 3D image of the stall without any dairy livestock;

subtract the first 3D image from the reference 3D image to produce a second 3D image of the rearview of the dairy livestock;
perform morphological image processing on the second 3D image to produce a third 3D image;
perform image thresholding on the third 3D image to produce a fourth 3D image;
cluster data from the fourth 3D image; and
identify, using the clustered data from the fourth 3D image, one or more legs of the dairy livestock.

18. The one or more computer-readable non-transitory storage media of claim 17 , wherein:
performing morphological image processing on the second 3D image to produce the third 3D image comprises using an image erosion algorithm;
performing image thresholding on the third 3D image to produce the fourth 3D image comp

Assignees

• Technologies Holdings Corp

Inventors

• Foresman Mark A
• Prevost Bradley J
• Van Aart Marijn
• Van Der Sluis Peter Willem
• Janani Alireza

Classifications

• A01J5/007 Primary

  Monitoring milking processes; Control or regulation of milking machines · CPC title

• A01J5/017

  Automatic attaching or detaching of clusters · CPC title

• A01J5/0175

  Attaching of clusters · CPC title

• G06T7/75

  involving models · CPC title

• G06V10/25

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title