License plate detection and recognition system

Patent metadata

Field	Value
Publication number	US-11783589-B2
Application number	US-202318150171-A
Country	US
Kind code	B2
Filing date	Jan 4, 2023
Priority date	Feb 11, 2018
Publication date	Oct 10, 2023
Grant date	Oct 10, 2023

Abstract

A license plate detection and recognition system receives training data comprising images of license plates. The system prepares ground truth data from the training data based predefined parameters. The system trains a first machine learning algorithm based on the ground truth data to generate a license plate detection model. The license plate detection model is configured to detect one or more regions in the images. The one or more regions contains a candidate for a license plate. The LPDR system generates a bounding box for each region. The LPDR system trains a second machine learning algorithm based on the ground truth data and the license plate detection model to generate a license plate recognition model. The license plate recognition model generates a sequence of alphanumeric characters with a level of recognition confidence for the sequence.

First claim

What is claimed is:

1. A computer-implemented method comprising:
receiving an input image;
detecting one or more license plate (LP) regions in the input image using a license plate detection model, wherein each LP region includes a license plate;
for each detected LP region, determining coordinates of a corresponding bounding box in the input image, a corresponding type of LP, and a corresponding detection confidence value;
determining a rotation angle for each LP region using a third machine learning model, the rotation angle configured to align each LP region to a horizontally aligned state;
rotating each LP region by a corresponding rotation angle; and
determining a sequence of alphanumeric characters of a LP identifier in each rotated LP region with a recognition confidence value corresponding to the sequence, using a license plate recognition model.

2. The computer-implemented method of claim 1 , further comprising:
receiving training data comprising images of license plates;
preparing ground truth data from the training data based on predefined parameters;
training a first machine learning algorithm based on the ground truth data to generate the license plate detection model, the license plate detection model configured to detect one or more regions in the images, the one or more regions containing a candidate for a license plate, and to generate a bounding box for each license plate; and
training a second machine learning algorithm based on the ground truth data and the license plate detection model to generate the license plate recognition model, the license plate recognition model configured to generate a sequence of alphanumeric characters in the bounding box with a level of recognition confidence for the sequence.

3. The computer-implemented method of claim 2 ,
wherein the predefined parameters comprise a combination of a number of rows for each license plate in the images, a minimum size for each license plate in the images, a minimum ratio of width-to-height for each license plate in the images, a minimum visibility for each license plate in the images, a template for each license, fonts, and text identifier from each license plate in the images.

4. The computer-implemented method of claim 1 , further comprising:
determining a bounding box corresponding to an image of a license plate in the input image by applying the license plate detection model to the input image; and
generating a string identifier with a corresponding confidence level for the license plate in the bounding box by applying the license plate recognition model to the image of the license plate within the bounding box.

5. The computer-implemented method of claim 4 , wherein generating the string identifier comprises:
determining a rotation angle of the bounding box using a third machine learning algorithm corresponding to the third machine learning model;
aligning the image of the license plate in the bounding box to the horizontally aligned state by performing a rotation of the image of the license plate in the bounding box with the rotation angle; and
performing a segmentation of the aligned image of the license plate to determine the string identifier corresponding to the license plate.

6. The computer-implemented method of claim 1 , further comprising:
determining a bounding box corresponding to an image of each candidate license plate in the input image by applying the license plate detection model to the input image, the bounding box indicating a license plate type and a corresponding confidence level;
forming a first set of candidates based on the determined bounding boxes;
first invalidating one or more candidates in the first set of candidates in response to detecting that a first feature of the one or more candidates in the first set of candidates exceeds a first feature threshold;
forming a second set of candidates of remaining candidates after the first invalidating;
generating a string identifier with a corresponding confidence level for each license plate in each bounding box by applying the license plate recognition model to the second set of candidates;
second invalidating one or more candidates in the second set of candidates in response to detecting that a second feature of the one or more candidates in the second set of candidates exceeds a second feature threshold;
forming a third set of candidates of remaining candidates after the second invalidating; and
generating an array of license plate information comprising an image of a license plate from the input image, a corresponding string identifier, a corresponding bounding box coordinates, and a corresponding confidence level.

7. The computer-implemented method of claim 2 ,
wherein the first machine learning algorithm includes a first convolution neural network (CNN) based algorithm,
wherein the second machine learning algorithm includes a second convolution neural network (CNN) based algorithm, and
wherein the type of the license plate is selected from at least one of: a single row license plate comprising one or more characters in a single row or a stacked form, and a multiple row license plate.

8. The computer-implemented method of claim 2 ,
wherein training the first machine learning algorithm is based on the ground truth data,
wherein the ground truth data comprises historical data regarding detection of a single row license plate of stacked letters from one or more images, detection of a multiple row license plate from one or more images, standard size and dimensions of each type of license plate, bounding box points of one or more license plates, and number of corresponding license plate identifier text rows for each type of bounding box.

9. The computer-implemented method of claim 2 ,
wherein training the second machine learning algorithm is based on the ground truth data,
wherein the ground truth data comprises historical data regarding one or more images of a template of a license plate without identifier, a mask image with information about how license plate identifier data is put on a template image, and alphanumeric identifiers of one or more license plates visible in corresponding images.

10. The computer-implemented method of claim 1 , further comprising:
validating each detected LP region before performing the license plate recognition, wherein the validation is performed based on an expected size of the license plate, mutual arrangement, and a detection confidence value;
validating the LP identifier of the detected LP region, based on a minimal and maximal number of characters of the license plate identifier, an expected license plate template, and a level of recognition confidence value; and
using the LP identifier of a current input image to update the LP identifier of a previous input image, in response to the current input image being a part of an input video stream.

11. A computing apparatus, the computing apparatus comprising:
a processor; and
a memory storing instructions that, when executed by the processor, configure the apparatus to perform operations comprising:
receiving an input image;
detecting one or more license plate (LP) regions in the input image using a license plate detection model, wherein each LP region includes a license plate;
for each detected LP region, determining coordinates of a corresponding bounding box in the input image, a corresponding type of LP, and a corresponding detection confidence value;
determining a rotation angle for each LP region using a third machine learning model, the rotation angle configured to align each LP region to a horizontally aligned state;
rotating each LP region by a corresponding rotation angle; and
determining a

Assignees

• Nortek Security & Control Llc
• Nice North America Llc

Inventors

• Popov Ilya
• Yashunin Dmitry
• Budenkov Semen
• Khadloya Krishna

Classifications

• G06V20/54 Primary

  of traffic, e.g. cars on the road, trains or boats · CPC title

• G06F18/24

  Classification techniques · CPC title

• G06T5/002

  Physics · mapped topic

• G06T5/20

  using local operators · CPC title

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title