Object localization framework for unannotated image data

Patent metadata

Field	Value
Publication number	US-11004236-B2
Application number	US-201916401695-A
Country	US
Kind code	B2
Filing date	May 2, 2019
Priority date	May 2, 2019
Publication date	May 11, 2021
Grant date	May 11, 2021

Abstract

A system is provided for object localization in image data. The system includes an object localization framework comprising a plurality of object localization processes. The system is configured to receive an image comprising unannotated image data having at least one object in the image, access a first object localization process of the plurality of object localization processes, determine first bounding box information for the image using the first object localization process, wherein the first bounding box information comprises at least one first bounding box annotating at least a first portion of the at least one object in the image, and receive first feedback regarding the first bounding box information determined by the first object localization process. The system is further configured to persist the image with the first bounding box information or access a second object localization process based on the first feedback.

First claim

What is claimed is:

1. A system for object localization in image data, the system comprising:
an object localization framework comprising a plurality of object localization processes, wherein the plurality of object localization processes for the object localization framework comprise four separate object localization processes, and wherein the object localization framework comprises a data processing pipeline that proceeds in order through each of the four separate object localization processes until success of one of the four separate object localization processes;
wherein the system is configured to:
receive an image comprising unannotated image data having at least one object in the image;
access a first object localization process of the plurality of object localization processes;
determine first bounding box information for the image using the first object localization process, wherein the first bounding box information comprises at least one first bounding box annotating at least a first portion of the at least one object in the image;
receive first feedback regarding the first bounding box information determined by the first object localization process;
if the received first feedback indicates success, persist the image with the first bounding box information determined by the first object localization process so that it can be applied for object detection in other images; and
if the received first feedback indicates failure, access a second object localization process of the plurality of object localization processes.

2. The system of claim 1 , wherein the first object localization process comprises generating the first bounding box information using convolution layer heatmaps of the image determined through a convoluted neural network.

3. The system of claim 2 , wherein the second object localization process comprises generating second bounding box information using basic contours in the image with Haar cascades, and wherein the system is further configured to:
determine the second bounding box information for the image using the second object localization process, wherein the second bounding box information comprises at least one second bounding box annotating at least a second portion of the at least one object in the image;
receive second feedback regarding the second bounding box information determined by the second object localization process;
if the received second feedback indicates success, persist the image with the second bounding box information determined by the second object localization process so that it can be applied for object detection in other images; and
if the received second feedback indicates failure, access a third object localization process of the plurality of object localization processes.

4. The system of claim 3 , wherein the third object localization process comprises converting the image using canny edge detection and generating third bounding box information using basic contours in the converted image with the Haar cascades, and wherein the system is further configured to:
determine the third bounding box information for the image using the third object localization process, wherein the third bounding box information comprises at least one third bounding box annotating at least a third portion of the at least one object in the image;
receive third feedback regarding the third bounding box information determined by the third object localization process;
if the received third feedback indicates success, persist the image with the third bounding box information determined by the third object localization process so that it can be applied for object detection in other images; and
if the received third feedback indicates failure, access a fourth object localization process of the plurality of object localization processes.

5. The system of claim 4 , wherein the fourth object localization process comprises applying image erosion to the image, highlighting margins among the at least one object in the image after the image erosion, and generating fourth bounding box information comprising at least one fourth bounding box annotating at least a fourth portion of the at least one object in the image using contour generation.

6. The system of claim 5 , wherein the at least one object comprises a plurality of objects in the image, wherein one of the second bounding box information, the third bounding box information, or the fourth bounding box information is selected based on the plurality of objects.

7. The system of claim 1 , wherein the first bounding box information is determined unsupervised without annotated data of the image generated from user input, and wherein the system is further configured to:
provide the first bounding box information for the first feedback.

8. A method for object localization in image data, the method comprising:
receiving an image comprising unannotated image data having at least one object in the image;
accessing a first object localization process of a plurality of object localization processes for an object localization framework, wherein the first object localization process comprises generating the first bounding box information using convolution layer heatmaps of the image determined through a convoluted neural network;
determining first bounding box information for the image using the first object localization process, wherein the first bounding box information comprises at least one first bounding box annotating at least a portion of the at least one object in the image;
receiving first feedback regarding the first bounding box information determined by the first object localization process;
if the received first feedback indicates success, persisting the image with the first bounding box information determined by the first object localization process so that it can be applied for object detection in other images;
if the received first feedback indicates failure, accessing a second object localization process of the plurality of object localization processes, wherein the second object localization process comprises generating second bounding box information using basic contours in the image with Haar cascades;
determining the second bounding box information for the image using the second object localization process, wherein the second bounding box information comprises at least one second bounding box annotating at least a second portion of the at least one object in the image;
receiving second feedback regarding the second bounding box information determined by the second object localization process;
if the received second feedback indicates success, persisting the image with the second bounding box information determined by the second object localization process so that it can be applied for object detection in other images; and
if the received second feedback indicates failure, accessing a third object localization process of the plurality of object localization processes.

9. The method of claim 8 , wherein the third object localization process comprises converting the image using canny edge detection and generating third bounding box information using basic contours in the converted image with the Haar cascades, and wherein the method further comprises:
determining the third bounding box information for the image using the third object localization process, wherein the third bounding box information comprises at least one third bounding box annotating at least a third portion of the at least one object in the image;
receiving third feedback regarding the third bounding box information determined by the third object localization process;
if the received third feedback indicates success, persisting the image with the third bounding box

Assignees

• Salesforce Com Inc

Inventors

• Mustafi Joy
• Kumar Lakshya
• Dua Rajdeep Singh

Classifications

• G06V10/446

  using Haar-like filters, e.g. using integral image techniques · CPC title

• G06V10/82

  using neural networks · CPC title

• G06T7/11 Primary

  Region-based segmentation · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06T7/75 Primary

  involving models · CPC title