System and method for identification of plant species

Patent metadata

Field	Value
Publication number	US-2022327815-A1
Application number	US-202017640742-A
Country	US
Kind code	A1
Filing date	Sep 3, 2020
Priority date	Sep 5, 2019
Publication date	Oct 13, 2022
Grant date	—

Abstract

A computer-implemented method, computer program product and computer system (100) for identifying weeds in a crop field using a dual task convolutional neural network (120) having a topology with an intermediate module (121) to execute a classification task being associated with a first loss function (LF1), and with a semantic segmentation module (122) to execute a segmentation task with a second different loss function (LF2). The intermediate module and the segmentation module are being trained together, taking into account the first and second loss functions (LF1, LF2). The system executes a method including receiving a test input (91) comprising an image showing crop plants of a crop species in an agricultural field and showing weed plants of one or more weed species among said crop plants; predicting the presence of one or more weed species (11, 12, 13) which are present in the respective tile; outputting a corresponding intermediate feature map to the segmentation module as output of the classification task; generating a mask for each weed species class as segmentation output of the second task by extracting multiscale features and context information from the intermediate feature map and concatenating the extracted information to perform semantic segmentation; and generating a final image (92) indicating for each pixel if it belongs to a particular weed species, and if so, to which weed species it belongs.

First claim

1 . A computer-implemented method ( 1000 ) for identifying weeds in a crop field using a dual task convolutional neural network ( 120 ) having a topology with:
an intermediate module ( 121 ) configured for executing a first task in determining weed species ( 11 , 12 , 13 ) which are present on a test input image ( 91 ), the first task being associated with a first loss function (LF 1 ), and a semantic segmentation module ( 122 ) configured for executing a second task in segmenting the test input image ( 91 ) to determine a class for each pixel of the test input image ( 91 ), the classes comprising the determined weed species, the second task associated with a second different loss function (LF 2 ), wherein the intermediate module and the segmentation module being trained together, taking into account the first and second loss functions (LF 1 , LF 2 ); the method comprising:
receiving ( 1100 ) a test input ( 91 ) comprising an image showing crop plants of a crop species in an agricultural field and showing weed plants of one or more weed species among said crop plants;
extracting ( 1200 ) tiles from the test input image, the tiles having the dimensions of the input shape of the intermediate module;
for each extracted tile:
the intermediate module ( 121 ) predicting ( 1300 ) the presence of one or more weed species which are present in the respective tile;
the intermediate module ( 121 ) outputting ( 1400 ) a corresponding intermediate feature map ( 121 - o 1 ) to the segmentation module ( 122 ) as output of the first task; and
the segmentation module generating ( 1700 ) a mask for each weed species class as segmentation output of the second task by extracting ( 1500 ) multiscale features and context information from the intermediate feature map and concatenating ( 1600 ) the extracted information to perform semantic segmentation, the mask being an image having the same size as a tile where each pixel on the mask is associated with a value representing the probability for said pixel to belong to the associated class; and

combining ( 1800 ) the generated masks into a final image indicating for each pixel if it belongs to a particular weed species, and if so, to which weed species it belongs.

2 . The method of claim 1 , wherein the intermediate module is implemented by a classification neural network.

3 . The method of claim 1 , wherein the first loss function (FL 1 ) is “weighted binary cross-entropy” where each sample pixel is weighted depending on the class it belongs to, the intermediate module ( 121 ) using “sigmoid” as last activation layer to support the presence of multiple classes simultaneously.

4 . The method of claim 1 , wherein the intermediate module is implemented by a regression neural network.

5 . The method of claim 4 , wherein the first loss function (FL 1 ) is “mean squared error” or “mean average error”, the intermediate module ( 121 ) using “linear” or “sigmoid” as last activation layer to support the detection of a presence of multiple classes simultaneously.

6 . The method of claim 1 , wherein the second loss function (FL 2 ) associated with the segmentation module is “weighted categorical cross-entropy”.

7 . The method of claim 1 , wherein an image training data set used for training the intermediate module together with the segmentation module comprises:
a first data subset with images ( 41 - a ) showing real world situations in an agricultural field with crop plants of a crop species ( 10 ) and weed plants of one or more weed species ( 11 , 12 , 13 ) amongst the crop plants, the first data subset with manual pixel annotations ( 41 - 1 , 41 - 2 , 41 - 3 ) indicating the species to which the pixels of the training images belong; and at least one of:
a second data subset with images ( 42 - a ) showing a plurality of weed plants of different weed species obtained from single plant images with each single plant image showing a plant of a single species, and the single plants pasted into a soil background image, the second data subset with automatic annotations ( 42 - 1 , 42 - 3 , 42 - 4 ) of the pixels belonging to the single weed species; or
a third data subset with images ( 43 - a ) showing a plurality of weed plants of a single weed species, the third data subset with automatic annotations ( 42 - 3 , 43 - 1 , 43 - 4 ) of the pixels belonging to the single weed species.

8 . The method of claim 1 , wherein the segmentation module ( 122 ) is implemented by a pyramid pooling module, and wherein the pyramid pooling module is based on a PSPNet topology.

9 . (canceled)

10 . The method of claim 8 , wherein the pyramid pooling module comprises four separate filters ( 122 - 2 ) with different receptive fields which scan the intermediate feature map ( 12101 ) and create four arrays for multi-scale feature detection to integrate information of different scales and sizes.

11 . The method of claim 10 , wherein the pyramid pooling module further comprises a plurality of up-sampling layers configured to restore the image size of each array to the size of the intermediate feature map ( 121 o 1 ) using bilinear interpolation.

12 . The method of claim 11 , wherein the pyramid pooling module further comprises convolutional layers ( 122 - 4 ) configured to extract contextual information from the four separate filters and concatenate the contextual information with the information of different scales and sizes to generate a final feature map ( 122 - 4 o ) of the same size as the intermediate feature map ( 121 - o 1 ).

13 . The method of claim 12 , wherein the pyramid pooling module further comprises fully-connected layers ( 122 - 5 ) to compute final pixel-wise predictions as the generated masks with a last activation layer “softmax”.

14 . The method of claim 1 , further comprising:
if a particular weed is identified, outputting a signal for operating, controlling and/or monitoring farming machinery wherein the signal is configured to trigger spraying or applying a herbicide or another crop protection agent targeting the particular weed.

15 . A non-transitory computer-readable medium having instructions encoded thereon that, when loaded into a memory of a computing device and executed by at least one processor of the computing device, cause the at least one processor to execute the steps of the computer-implemented method according to claim 1 .

16 . A computer system for identifying weeds in a crop field, the computer system comprising a memory and at least one processor and further comprising software modules that, when executed by the at least one processor, cause the computer system to perform the steps of the computer-implemented method according to claim 1 .

17 . A computer-implemented method ( 1000 ) for identifying plant species in an agricultural field using a dual task convolutional neural network ( 120 ) having a topology with:
an intermediate module ( 121 ) configured for executing a first task in determining plant species ( 11 , 12 , 13 ) which are present on a test input image ( 91 ), the first task being associated with a first loss function (LF 1 ), and a semantic segmentation module ( 122 ) configured for executing a second task in segmenting the test input image ( 91 ) to determine a class for each pixel of the test input image ( 91 ), the classes comprising the determined plant species, the second task associated with a second different loss function (LF 2 ), wherein the intermediate module and the segmentation module being trained together, taking into account the first and second loss functions (LF 1 , LF 2

Assignees

• Basf Se

Inventors

• Picon Ruiz Artzai
• Linares De La Puerta Miguel
• Klukas Christian
• Eggers Till
• Oberst Rainer
• Contreras Gallardo Juan Manuel
• Romero Rodriguez Javier
• Gad Hikal Khairy Shohdy
• Kraemer Gerd
• Echazarra Huguet Jone
• Navarra-Mestre Ramon
• Gonzalez San Emeterio Miguel

Classifications

• G06V20/188

  Vegetation · CPC title

• A01M21/00 Primary

  Apparatus for the destruction of unwanted vegetation, e.g. weeds (control of undesirable vegetation on roads or permanent ways of railways E01H11/00) · CPC title

• G06T7/001

  using an image reference approach · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/10024

  Color image · CPC title