System and method for deep network fusion for fast and robust object detection

Patent metadata

Field	Value
Publication number	US-10657364-B2
Application number	US-201715434880-A
Country	US
Kind code	B2
Filing date	Feb 16, 2017
Priority date	Sep 23, 2016
Publication date	May 19, 2020
Grant date	May 19, 2020

Abstract

A method and apparatus are provided. The method includes receiving an image, detecting an object in the image, determining, by a primary object detector, a primary confidence detection score of the object, determining, by a classification network, a confidence scaling factor of the object, and adjusting the primary confidence detection score based on multiplying the primary confidence detection score by the confidence scaling factor.

First claim

What is claimed is:

1. A method, comprising:
receiving an image;
detecting an object in the image;
determining, by a primary object detector, a primary confidence detection score of the object;
determining, by a classification network, a confidence scaling factor of the object;
adjusting the primary confidence detection score based on multiplying the primary confidence detection score by the confidence scaling factor;
generating a semantic segmentation mask based on classifying each pixel into one of pre-determined classes;
determining an amount of overlap, if any, between a bounding box of the object and the semantic segmentation mask; and
adjusting the primary confidence detection score and eliminating a secondary object detection from the semantic segmentation mask if there is no overlap between the bounding box of the object and the semantic segmentation mask,
wherein the semantic segmentation mask is a binary mask generated with one color showing activated pixels in the image and another color showing a background in the image.

2. The method of claim 1 , further comprising:
scaling the primary confidence detection score based on multiplying the primary confidence detection score by a product of a plurality of confidence scaling factors of the object from a plurality of classification networks.

3. The method of claim 2 , further comprising:
setting an object detection to a fixed size using the scaled primary confidence detection score;
determining random crops with the fixed size from the scaled primary confidence detection score; and
labelling a class of the random crops with respect to ground truth to train the classification network.

4. The method of claim 1 , wherein the confidence scaling factor is based on a ratio of a classification probability of the object to a desired maximum classification probability.

5. The method of claim 1 , wherein the confidence scaling factor is set to at least a desired minimum value.

6. The method of claim 1 , further comprising determining a secondary confidence detection score of the object;
maintaining the primary confidence detection score if the secondary confidence detection score is greater than or equal to a threshold; and
adjusting the primary confidence detection score if the secondary confidence detection score is less than the threshold.

7. The method of claim 6 , wherein adjusting the primary confidence detection score if the secondary confidence detection score is less than the threshold comprises multiplying the primary confidence detection score by a product of the secondary confidence detection score and a plurality of confidence scaling factors.

8. The method of claim 6 , wherein adjusting the primary confidence detection score if the secondary confidence detection score is less than the threshold comprises multiplying the adjusted primary confidence detection score by a product of the secondary confidence detection score and a plurality of confidence scaling factors.

9. The method of claim 1 , further comprising adjusting the primary confidence detection score in proportion to an amount of overlap if the amount of overlap between a bounding box of the object and a semantic segmentation mask is less than a threshold.

10. The method of claim 1 , further comprising maintaining the primary confidence detection score if an amount of overlap between a bounding box of the object and a semantic segmentation mask is greater than or equal to a threshold.

11. The method of claim 1 , further comprising operating the primary object detector and the classification network in parallel.

12. An apparatus, comprising:
an image sensor;
a primary object detector;
a classification network; and
a processor configured to:
capture an image from the image sensor,
detect an object in the image,
determine a primary confidence detection score of the object using the primary object detector,
determine a confidence scaling factor of the object using the classification network,
adjust the primary confidence detection score based on multiplying the primary confidence detection score by the confidence scaling factor,
generate a semantic segmentation mask based on classifying each pixel into one of pre-determined classes,
determine an amount of overlap, if any, between a bounding box of the object and the semantic segmentation mask; and
adjust the primary confidence detection score and eliminate a secondary object detection from the semantic segmentation mask if there is no overlap between the bounding box of the object and the semantic segmentation mask,
wherein the semantic segmentation mask is a binary mask generated with one color showing activated pixels in the image and another color showing a background in the image.

13. The apparatus of claim 12 , wherein the processor is further configured to scale the primary confidence detection score based on multiplying the primary confidence detection score by a product of a plurality of confidence scaling factors of the object from a plurality of classification networks.

14. The apparatus of claim 12 , wherein the confidence scaling factor is based on a ratio of a classification probability of the object to a desired maximum classification probability.

15. The apparatus of claim 12 , wherein the confidence scaling factor is set to at least a desired minimum value.

16. The apparatus of claim 12 , wherein the processor is further configured to determine a secondary confidence detection score of the object,
maintain the primary confidence detection score if the secondary confidence detection score is greater than or equal to a threshold, and
adjust the primary confidence detection score if the secondary confidence detection score is less than the threshold.

17. The apparatus of claim 16 , wherein adjusting the primary confidence detection score if the secondary confidence detection score is less than the threshold comprises multiplying the primary confidence detection score by a product of the secondary confidence detection score and a plurality of confidence scaling factors.

18. The apparatus of claim 16 , wherein adjusting the primary confidence detection score if the secondary confidence detection score is less than the threshold comprises multiplying the adjusted primary confidence detection score by a product of the secondary confidence detection score and a plurality of confidence scaling factors.

19. The apparatus of claim 12 , wherein the processor is further configured to adjust the primary confidence detection score in proportion to an amount of overlap if the amount of overlap between a bounding box of the object and a semantic segmentation mask is less than a threshold.

20. The apparatus of claim 12 , wherein the processor is further configured to maintain the primary confidence detection score if an amount of overlap between a bounding box of the object and a semantic segmentation mask is greater than or equal to a threshold.

21. The apparatus of claim 12 , wherein the processor is further configured to operate the primary object detector and the classification network in parallel.

22. The apparatus of claim 12 , wherein the processor is further configured to:
set an object detection to a fixed size using the scaled primary confidence detection score,
determine random crops with the fixed size from the scaled primary confidence detection score, and
label a class of the random crops with respect to ground truth to train the classification network.

Assignees

• Samsung Electronics Co Ltd

Inventors

• El-Khamy Mostafa
• Du Xianzhi
• Lee Jungwon

Classifications

• G06F30/398

  Design verification or optimisation, e.g. using design rule check [DRC], layout versus schematics [LVS] or finite element methods [FEM] (optical proximity correction [OPC] design processes G03F1/36) · CPC title

• G06T2207/10004

  Still image; Photographic image · CPC title

• G06T2207/30196

  Human being; Person · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T7/11

  Region-based segmentation · CPC title