Image recognition device, operating method of image recognition device, and computing device including image recognition device

Patent metadata

Field	Value
Publication number	US-2020356804-A1
Application number	US-202016870412-A
Country	US
Kind code	A1
Filing date	May 8, 2020
Priority date	May 10, 2019
Publication date	Nov 12, 2020
Grant date	—

Abstract

Provided is an image recognition device. The image recognition device includes a frame data change detector that sequentially receives a plurality of frame data and detects a difference between two consecutive frame data, an ensemble section controller that sets an ensemble section in the plurality of frame data, based on the detected difference, an image recognizer that sequentially identifies classes respectively corresponding to a plurality of section frame data by applying different neural network classifiers to the plurality of section frame data in the ensemble section, and a recognition result classifier that sequentially identifies ensemble classes respectively corresponding to the plurality of section frame data by combining the classes in the ensemble section.

First claim

What is claimed is:

1 . An image recognition device comprising:
a frame data change detector configured to sequentially receive a plurality of frame data and to detect a difference between two consecutive frame data; an ensemble section controller configured to set an ensemble section in the plurality of frame data, based on the detected difference; an image recognizer configured to sequentially identify classes respectively corresponding to a plurality of section frame data by applying different neural network classifiers to the plurality of section frame data in the ensemble section; and a recognition result classifier configured to sequentially identify ensemble classes respectively corresponding to the plurality of section frame data by combining the classes in the ensemble section.

2 . The image recognition device of claim 1 , wherein, when the difference between the two consecutive frame data is more than a threshold, the two consecutive frame data are incorporated into different ensemble sections by the ensemble section controller.

3 . The image recognition device of claim 2 , wherein the ensemble section controller sets preceding frame data of the two consecutive frame data as last frame data of a current ensemble section and sets subsequent frame data of the two consecutive frame data as start frame data of a subsequent ensemble section.

4 . The image recognition device of claim 1 , wherein, when the difference between the two consecutive frame data is less than a threshold, the ensemble section controller includes the two consecutive frame data in a current ensemble section.

5 . The image recognition device of claim 1 , wherein the image recognizer includes neural network classifiers learned in different ways and sequentially applies the neural network classifiers to the plurality of section frame data in the ensemble section.

6 . The image recognition device of claim 5 , wherein the neural network classifiers share operation resources of the image recognizer and are activated alternately.

7 . The image recognition device of claim 5 , wherein the recognition result classifier alternately receives the classes from the neural network classifiers.

8 . The image recognition device of claim 1 , wherein, when a specific class is repeatedly detected from first frame data to nth frame data (where n is a positive integer greater than 1) among the plurality of section frame data, the recognition result classifier sums probabilities of the specific class detected from the first frame data to the nth frame data, divides the summed result by n, and includes the divided result in an ensemble class of the nth frame data.

9 . The image recognition device of claim 1 , wherein, when a specific class is detected from nth frame data (where n is a positive integer greater than 2) among the plurality of section frame data, and when the specific class is detected k times (where k is a positive integer greater than 1 and less than n) from first frame data to the nth frame data, the recognition result classifier sums probabilities of the specific class detected from the first frame data to the nth frame data, divides the summed result by k, and includes the divided result in an ensemble class of the nth frame data.

10 . The image recognition device of claim 1 , wherein, when a specific class is detected from nth frame data (where n is a positive integer greater than 1) among the plurality of section frame data, and when the specific class is first detected from first frame data to the nth frame data, the recognition result classifier includes the specific class in an ensemble class of the nth frame data.

11 . The image recognition device of claim 1 , wherein, when a specific class is detected from first frame data among the plurality of section frame data, the recognition result classifier includes the specific class in an ensemble class of the first frame data.

12 . The image recognition device of claim 1 , wherein the ensemble section controller is configured to further receive the classes, and the ensemble section controller corrects the ensemble section, based on the received classes.

13 . The image recognition device of claim 12 , wherein, when a difference between the classes corresponding to a plurality of consecutive frame data in the ensemble section is more than a threshold, the ensemble section controller incorporates the plurality of consecutive frame data into different ensemble sections.

14 . A method of operating an image recognition device, the method comprising:
receiving frame data; setting an ensemble section, based on a change of between previous frame data and the received frame data; identifying a class in the received frame data by using a neural network classifier different from a neural network classifier applied to the previous frame data in the ensemble section; and identifying an ensemble class by combining a preceding class identified in the previous frame data in the ensemble section and the class identified in the received frame data.

15 . The method of claim 14 , wherein the identifying of the ensemble class by combining the preceding class identified in the previous frame data and the class identified in the received frame data in the ensemble section includes;
combining preceding classes identified in all of a plurality of previous frame data in the ensemble section and the class identified in the received frame data.

16 . The method of claim 14 , wherein the setting of the ensemble section, based on the change of between the previous frame data and the received frame data includes;
when the change of between the previous frame data and the received frame data is less than a threshold, incorporating the received frame data into the same ensemble section as the previous frame data.

17 . The method of claim 14 , further comprising:
correcting the ensemble section by using the class identified in the received frame data.

18 . A computing device comprising:
a processor; a memory configured to store a plurality of frame data; and an image recognition device configured to perform an image recognition of the plurality of frame data stored in the memory, based on a request of the processor, and wherein the image recognition device is configured to: set an ensemble section, based on a change of the plurality of frame data; identify classes by applying different neural network classifiers to a plurality of section frame data in the ensemble section, and identify an ensemble class of a specific time by combining a class of frame data of the specific time and classes of a plurality of previous frame data in the ensemble section.

19 . The computing device of claim 18 , wherein the image recognition device compares the class of the specific time and the classes of the plurality of previous frame data in the ensemble section and corrects the ensemble section depending on the comparison result.

20 . The computing device of claim 18 , wherein, when the number of the plurality of frame data included in the ensemble section reaches a threshold, the image recognition device ends the ensemble section and starts a new ensemble section from a subsequent frame data.

Assignees

• Electronics & Telecommunications Res Inst

Inventors

• Kim Ju-Yeob
• Kim Byung Jo
• Kim Seong Min
• Kim Jin Kyu
• Park Ki Hyuk
• Lee Mi Young
• Lee Joo Hyun
• Jeon Young-Deuk
• Cho Min-Hyung

Classifications

• G06V20/41 Primary

  Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49) · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/809

  of classification results, e.g. where the classifiers operate on the same input data · CPC title

• G06F18/2431

  Multiple classes · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title