Content-based detection and three dimensional geometric reconstruction of objects in image and video data

Patent metadata

Field	Value
Publication number	US-9779296-B1
Application number	US-201615234969-A
Country	US
Kind code	B1
Filing date	Aug 11, 2016
Priority date	Apr 1, 2016
Publication date	Oct 3, 2017
Grant date	Oct 3, 2017

Abstract

Systems, computer program products, and techniques for detecting objects depicted in digital image data are disclosed, according to various exemplary embodiments. The inventive concepts uniquely utilize internal features to accomplish object detection, thereby avoiding reliance on detecting object edges and/or transitions between the object and other portions of the digital image data, e.g. background textures or other objects. The inventive concepts thus provide an improvement over conventional object detection since objects may be detected even when edges are obscured or not depicted in the digital image data. In one aspect, a computer-implemented method of detecting an object depicted in a digital image includes: detecting a plurality of identifying features of the object, wherein the plurality of identifying features are located internally with respect to the object; and projecting a location of one or more edges of the object based at least in part on the plurality of identifying features.

First claim

What is claimed is:

1. A computer-implemented method of detecting an object depicted in a digital image, the method comprising:
detecting, using a hardware processor, a plurality of identifying features of the object, wherein the plurality of identifying features are located internally with respect to the object; and
projecting, using the hardware processor, a location of one or more edges of the object based at least in part on the plurality of identifying features; and
outputting the projected location of the one or more edges of the object to at least one of: a display of a computer, and a non-transitory computer readable medium.

2. The computer-implemented method as recited in claim 1 , wherein detecting the plurality of identifying features comprises analyzing a plurality of feature vectors each corresponding to pixels within a patch of the digital image to determine whether the patch includes a sharp transition in intensity.

3. The computer-implemented method as recited in claim 1 , wherein detecting the plurality of identifying features comprises automatic feature zone discovery; and
wherein automatic feature zone discovery comprises:
matching a plurality of pixels in the digital image to a plurality of corresponding pixels in a plurality of reference images to form a set of matching pairs, each matching pair including one pixel from the digital image and one pixel from one of the plurality of reference images; and
determining a subset of the matching pairs exhibiting a frequency within the set of matching pairs that is greater than a predetermined frequency threshold.

4. The computer-implemented method as recited in claim 1 , comprising transforming the digital image to display the projected location of the one or more edges of the object.

5. The computer-implemented method as recited in claim 4 , wherein at least portions of at least one of the one or more edges displayed in the transformed digital image are missing from the digital image.

6. The computer-implemented method as recited in claim 4 , wherein the digital image is characterized by a complex background comprising a plurality of sharp intensity transitions not corresponding to edges of the object.

7. The computer-implemented method as recited in claim 1 , wherein the plurality of identifying features comprise boilerplate content.

8. The computer-implemented method as recited in claim 1 , comprising identifying a plurality of distinctive pixels within the plurality of identifying features of the object, wherein the distinctive pixels are located at positions within the digital image characterized by a sharp transition in intensity.

9. The computer-implemented method as recited in claim 1 , comprising matching the digital image depicting the object to one of a plurality of reference images each depicting a known object type, wherein the matching comprises determining whether the object includes distinctive pixels that correspond to distinctive pixels present in one or more of the plurality of reference images.

10. The computer-implemented method as recited in claim 1 , comprising:
matching the digital image depicting the object to one of a plurality of reference images each depicting a known object type; and
designating as an outlier a candidate match between a distinctive pixel in the digital image and one or more candidate corresponding distinctive pixels present in one of the plurality of reference images;
wherein the outlier is designated in response to determining a distance ratio is greater than a predetermined distance ratio threshold, wherein the distance ratio is a ratio describing:
a first distance between the distinctive pixel in the digital image and a first of the one or more candidate corresponding distinctive pixels; and
a second distance between the distinctive pixel in the digital image and a second of the one or more candidate corresponding distinctive pixels.

11. The computer-implemented method as recited in claim 1 , comprising:
matching the digital image depicting the object to one of a plurality of reference images each depicting a known object type; and
designating as an outlier a candidate match between a distinctive pixel in the digital image and a candidate corresponding distinctive pixel present in one of the plurality of reference images in response to determining the candidate match is not unique.

12. The computer-implemented method as recited in claim 1 , wherein at least a portion of one or more edges of the object for which the location is projected is missing in the digital image.

13. The computer-implemented method as recited in claim 1 , wherein projecting the location of the one or more edges of the object is based on a mapping of key points within some or all of the plurality of identifying features to key points of a reference image depicting an object belonging to a same class as the object depicted in the digital image.

14. The computer-implemented method as recited in claim 1 , comprising cropping the digital image based at least in part on the projected location of the one or more edges of the object;
wherein the cropped digital image depicts a portion of a background of the digital image surrounding the object; and
wherein the method comprises detecting one or more transitions between the background and the object within the cropped digital image.

15. The computer-implemented method as recited in claim 1 , comprising:
cropping the digital image based at least in part on the projected location of the one or more edges of the object; and
classifying the object depicted within the cropped digital image.

16. The computer-implemented method as recited in claim 1 , comprising:
generating a plurality of scaled images based on the digital image, each scaled image being characterized by a different resolution;
extracting one or more feature vectors from each scaled image; and
matching one or more of the scaled images to one of a plurality of reference images, each reference image depicting a known object type and being characterized by a known resolution.

17. The computer-implemented method as recited in claim 1 , comprising:
attempting to detect the object within the digital image using a plurality of predetermined object detection models each corresponding to a known object type; and
determining a classification of the object based on a result of attempting to detect the object within the digital image using the plurality of predetermined object detection models; and
wherein the classification of the object is determined to be the known object type corresponding to one of the object detection models for which the attempt to detect the object within the digital image was successful.

18. A computer program product for detecting an object depicted in a digital image, comprising a non-transitory computer readable medium having stored thereon computer readable program instructions configured to cause a processor, upon execution thereof, to:
generate, using the processor, a plurality of scaled images based on the digital image, each scaled image being characterized by a different resolution;
extract, using the processor, one or more feature vectors from each scaled image;
match, using the processor, one or more of the scaled images to one of a plurality of reference images based on the one or more feature vectors, each reference image depicting a known object type and being characterized by a known resolution;
detect, using the processor, a plurality of identifying features of the object within the scaled image matched to the on

Assignees

• Kofax Inc

Inventors

• Ma Jiyong
• Thompson Stephen Michael
• Amtrup Jan W

Classifications

• G06V30/414

  Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text · CPC title

• G06T11/60 Primary

  Creating or editing images; Combining images with text · CPC title

• G06V10/44

  Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components · CPC title

• G06V10/462

  Salient features, e.g. scale invariant feature transforms [SIFT] · CPC title

• G06T5/002

  Physics · mapped topic