Identifying objects within an image

Patent metadata

Field	Value
Publication number	US-9754177-B2
Application number	US-201313923820-A
Country	US
Kind code	B2
Filing date	Jun 21, 2013
Priority date	Jun 21, 2013
Publication date	Sep 5, 2017
Grant date	Sep 5, 2017

Abstract

One or more aspects of the subject disclosure are directed towards identifying objects within an image via image searching/matching. In one aspect, an image is processed into bounding boxes, with the bounding boxes further processed to each surround a possible object. A sub-image of pixels corresponding to the bounding box is featurized for matching with tagged database images. The information (tags) associated with any matched images is processed to identify/categorize the sub-image and thus the object corresponding thereto.

First claim

What is claimed is:

1. A computer-implemented method for processing an image, the method comprising:
creating a first bounding box around a first region of interest in an image;
creating a second bounding box around a second region of interest in the image;
creating a third bounding box around the first region of interest in the image;
determining that the first bounding box and the third bounding box meet a variance threshold;
determining that the second bounding box does not meet the variance threshold;
determining the first bounding box and the third bounding box overlap;
based on the determination that the first bounding box and the third bounding box overlap, merging the first bounding box and the third bounding box into a single fourth bounding box;
extracting a sub-image corresponding to the fourth bounding box;
querying a remote image-to-image search engine using the sub-image to find a plurality of matched images for the sub-image, the plurality of matched images comprising a first matched image, the first matched image comprising a second object that corresponds to a first object, the second object having a first identifier that provides a category for the second object;
determining a first matching value based on matching the sub-image with the first matched image;
determining that the first matching value is greater than a matching value threshold;
based on determining that the first matching value is greater than the matching value threshold, labeling the sub-image with a sub-image tag that corresponds to an identity of the second object; and
outputting the image with the labeled sub-image.

2. The method of claim 1 further comprising:
segmenting the image into segments; and
creating the first and the second bounding boxes from the segments.

3. The method of claim 2 further comprising, grouping a fifth and sixth bounding box together based upon similarity.

4. The method of claim 1 further comprising extracting features from the sub-image.

5. The method of claim 4 wherein extracting the features comprises using multi-orientation steerable filters.

6. The method of claim 1 , wherein labeling the sub-image comprises:
integrating the identity of the second object; and
pruning the identity of the second object to fit to the sub-image.

7. The method of claim 6 , wherein pruning the identity of the second object comprises determining what identifying information of the one or more matched images is semantically coherent.

8. The method of claim 6 , wherein pruning the identity of the second object comprises pruning noisy tags.

9. The method of claim 1 , wherein the identity of the second object is a species of the category of the second object.

10. A system comprising:
a memory including an input image; and
a processor configured to:
create a first bounding box around a first region of interest in the input image;
create a second bounding box around a second region of interest in the input image;
create a third bounding box around the first region of interest in the input image;
determine that the first bounding box and the third bounding box meet a variance threshold;
determine that the second bounding box does not meet the variance threshold;
determine the first bounding box and the third bounding box overlap;
based on the determination that the first bounding box and the third bounding box overlap, merge the first bounding box and the third bounding box into a single fourth bounding box;
select a sub-image corresponding to the fourth bounding box;
query an image-to-image search engine using the sub-image to obtain identifying information of a plurality of matched images, the plurality of matched images comprising a first matched image, the first matched image comprising a second object that corresponds to a first object, the second object having a first identifier that provides a category for the second object;
process the identifying information, the process comprising:
determining a first matching value based on matching the sub-image with the first matched image;
determining that the first matching value is greater than a matching value threshold; and
based on determining that the first matching value is greater than the matching value threshold, generating an identifier for the sub-image, the identifier corresponding to the category of the second object.

11. The system of claim 10 wherein the processor is incorporated into one or more of the following: a search engine, a mobile computing device, a personal computing device, and a cloud service.

12. The system of claim 10 wherein the processor is further configured to output a parsed image labeled with the identifier.

13. The system of claim 10 wherein the processor is further configured to extract features from the sub-image, and to query the image-to-image search engine with at least some of the features to obtain the identifying information of the one or more matched images.

14. The system of claim 13 wherein the features comprise global features.

15. The system of claim 10 , wherein selecting the sub-image corresponding to the fourth bounding box further comprises determining the fourth bounding box includes a color of interest.

16. One or more computer-readable hardware storage devices storing computer executable instructions, which upon execution perform operations, comprising:
inputting an image;
creating a first bounding box around a first region of interest in the image;
creating a second bounding box around a second region of interest in the image;
creating a third bounding box around the first region of interest in the image;
determining that the first bounding box and the third bounding box meet a variance threshold;
determining that the second bounding box does not meet the variance threshold;
determining the first bounding box and the third bounding box overlap;
based on the determination that the first bounding box and the third bounding box overlap, merging the first bounding box and the third bounding box into a single fourth bounding box;
extracting a sub-image corresponding to the fourth bounding box;
determining sub-image features of the sub-image;
using the sub-image, querying an image-to-image search engine including a plurality of tagged images with features corresponding to the sub-image features of the sub-image, the plurality of tagged images comprising a first tagged image, the first tagged image comprising a second object that corresponds to a first object, the second object having a first identifier that provides a category for the second object;
determining a first matching value based on matching the sub-image with the first tagged image;
determining that the first matching value is greater than a matching value threshold; and
based on determining that the first matching value is greater than the matching value threshold, matching the first object corresponding to the sub-image with the first tagged image.

17. The one or more computer-readable hardware storage devices of claim 16 , wherein matching the first object comprises processing information associated with the first tagged image to categorize or identify the first object.

18. The one or more computer-readable hardware storage devices of claim 16 , wherein generating the first, second, and third bounding boxes comprises segmenting the image into segments and processing the segments.

19. The one or more computer-readable hardware storage devices of claim 18 , wherein processing the segments comprises determining tha

Assignees

• Microsoft Corp
• Microsoft Technology Licensing Llc

Inventors

• Liu Ce
• Weiss Yair
• Torralba Barriuso Antonio

Classifications

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F2218/12

  Classification; Matching · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06F18/2411

  based on the proximity to a decision surface, e.g. support vector machines · CPC title

• G06V10/25 Primary

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title