Image recognition device, operating method of image recognition device, and computing device including image recognition device

Patent metadata

Field	Value
Publication number	US-11449720-B2
Application number	US-202016870412-A
Country	US
Kind code	B2
Filing date	May 8, 2020
Priority date	May 10, 2019
Publication date	Sep 20, 2022
Grant date	Sep 20, 2022

Abstract

Provided is an image recognition device. The image recognition device includes a frame data change detector that sequentially receives a plurality of frame data and detects a difference between two consecutive frame data, an ensemble section controller that sets an ensemble section in the plurality of frame data, based on the detected difference, an image recognizer that sequentially identifies classes respectively corresponding to a plurality of section frame data by applying different neural network classifiers to the plurality of section frame data in the ensemble section, and a recognition result classifier that sequentially identifies ensemble classes respectively corresponding to the plurality of section frame data by combining the classes in the ensemble section.

First claim

What is claimed is:

1. An image recognition device comprising:
a frame data change detector configured to sequentially receive a plurality of frame data and to detect a difference between two consecutive frame data;
an ensemble section controller configured to set an ensemble section in the plurality of frame data, based on the detected difference, the ensemble section including a multiplicity of frame data that are determined from the plurality of frame data based on the detected difference;
an image recognizer including a plurality of neural network classifiers that are learned in different ways and configured to sequentially identify classes respectively corresponding to the multiplicity of frame data by applying the multiplicity of frame data to different neural network classifiers among the plurality of neural network classifiers; and
a recognition result classifier configured to sequentially identify ensemble classes respectively corresponding to the multiplicity of frame data by combining the classes of the multiplicity of frame data in the ensemble section.

2. The image recognition device of claim 1 , wherein, when the difference between the two consecutive frame data is greater than a threshold, the two consecutive frame data are incorporated into different ensemble sections by the ensemble section controller.

3. The image recognition device of claim 2 , wherein the ensemble section controller sets preceding frame data of the two consecutive frame data as last frame data of a current ensemble section and sets subsequent frame data of the two consecutive frame data as start frame data of a subsequent ensemble section.

4. The image recognition device of claim 1 , wherein, when the difference between the two consecutive frame data is less than a threshold, the ensemble section controller includes the two consecutive frame data in a current ensemble section.

5. The image recognition device of claim 1 , wherein the image recognizer includes neural network classifiers learned in different ways and sequentially applies the neural network classifiers to the multiplicity of frame data in the ensemble section.

6. The image recognition device of claim 5 , wherein the neural network classifiers share operation resources of the image recognizer and are activated alternately.

7. The image recognition device of claim 5 , wherein the recognition result classifier alternately receives the classes from the neural network classifiers.

8. The image recognition device of claim 1 , wherein, when a specific class is repeatedly detected from first frame data to nth frame data (where n is a positive integer greater than 1) among the multiplicity of frame data, the recognition result classifier sums probabilities of the specific class detected from the first frame data to the nth frame data, divides the summed result by n, and includes the divided result in an ensemble class of the nth frame data.

9. The image recognition device of claim 1 , wherein, when a specific class is detected from nth frame data (where n is a positive integer greater than 2) among the multiplicity of frame data, and when the specific class is detected k times (where k is a positive integer greater than 1 and less than n) from first frame data to the nth frame data, the recognition result classifier sums probabilities of the specific class detected from the first frame data to the nth frame data, divides the summed result by k, and includes the divided result in an ensemble class of the nth frame data.

10. The image recognition device of claim 1 , wherein, when a specific class is detected from nth frame data (where n is a positive integer greater than 1) among the multiplicity of frame data, and when the specific class is first detected from first frame data to the nth frame data, the recognition result classifier includes the specific class in an ensemble class of the nth frame data.

11. The image recognition device of claim 1 , wherein, when a specific class is detected from first frame data among the multiplicity of frame data, the recognition result classifier includes the specific class in an ensemble class of the first frame data.

12. The image recognition device of claim 1 , wherein the ensemble section controller is configured to further receive the classes, and the ensemble section controller corrects the ensemble section, based on the received classes.

13. The image recognition device of claim 12 , wherein, when a difference between the classes corresponding to a plurality of consecutive frame data in the ensemble section is greater than a threshold, the ensemble section controller incorporates the plurality of consecutive frame data into different ensemble sections.

14. A method of operating an image recognition device, the method comprising:
sequentially receiving a plurality of frame data;
setting an ensemble section, based on a difference between previous frame data and received frame data among the plurality of frame data, the ensemble section including a multiplicity of frame data that are determined from the plurality of frame data based on the difference;
identifying a class in the received frame data by using a first neural network classifier different from a second neural network classifier applied to the previous frame data in the ensemble section, the first neural network classifier and the second neural network classifier learned in different ways; and
identifying an ensemble class by combining a preceding class identified in the previous frame data in the ensemble section and the class identified in the received frame data.

15. The method of claim 14 , wherein the identifying of the ensemble class includes:
combining preceding classes identified in all of a plurality of previous frame data in the ensemble section and the class identified in the received frame data.

16. The method of claim 14 , wherein the setting of the ensemble section includes:
when the difference between the previous frame data and the received frame data is less than a threshold, incorporating the received frame data into the ensemble section.

17. The method of claim 14 , further comprising:
correcting the ensemble section by using the class identified in the received frame data.

18. A computing device comprising:
a processor;
a memory configured to store a plurality of frame data; and
an image recognition device including a plurality of neural network classifiers that are learned in different ways and configured to perform an image recognition of the plurality of frame data stored in the memory, in response to a request of the processor, and
wherein the image recognition device is configured to:
set an ensemble section, based on differences of the plurality of frame data, the ensemble section including a multiplicity of frame data that are determined from the plurality of frame data based on the differences;
identify classes of the multiplicity of frame data by applying different neural network classifiers to the multiplicity of frame data, and
identify an ensemble class of a specific time by combining a class of current frame data of the specific time and classes of a plurality of previous frame data in the ensemble section.

19. The computing device of claim 18 , wherein the image recognition device compares the class of the current frame data and the classes of the plurality of previous frame data in the ensemble section and corrects the ensemble section depending on the comparison result.

20. The computing device of claim 18 , wherein, when the num

Assignees

• Electronics & Telecommunications Res Inst

Inventors

• Kim Ju-Yeob
• Kim Byung Jo
• Kim Seong Min
• Kim Jin Kyu
• Park Ki Hyuk
• Lee Mi Young
• Lee Joo Hyun
• Jeon Young-Deuk
• Cho Min-Hyung

Classifications

• G06V20/41 Primary

  Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49) · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/809

  of classification results, e.g. where the classifiers operate on the same input data · CPC title

• G06F18/2431

  Multiple classes · CPC title

• G06F18/254

  of classification results, e.g. of results related to same input data · CPC title