System and method for search portions of objects in images and features thereof

Patent metadata

Field	Value
Publication number	US-9008435-B2
Application number	US-201213619939-A
Country	US
Kind code	B2
Filing date	Sep 14, 2012
Priority date	May 9, 2005
Publication date	Apr 14, 2015
Grant date	Apr 14, 2015

Abstract

Embodiments enable searching of portions of objects in images, including programmatically analyzing each image in a collection in order to determine image data that, for individual images in the collection, represents one or more visual characteristics of a portion of an object shown in that image. A user is enabled to specify one or more search criteria that includes image data, and a search result may be determined based on one or more images in the collection that show a corresponding object that has a portion that satisfies a threshold. The threshold is defined at least in part by the one or more search criteria.

First claim

The invention claimed is:

1. A method for enabling searching of images based on image features, the method comprising:
using one or more processors to perform steps comprising:
programmatically analyzing each image in a collection of images, including determining at least a local feature and a global feature for individual objects that appear in each image of the collection, the local feature of each object corresponding to a visual characteristic of a segment of that object, the global feature of each object corresponding to a visual characteristic of the object as a whole, wherein determining at least a local feature and a global feature for individual objects that appear in each image of the collection comprises, for each object in the image, determining at least a first global feature independently from determining at least a first local feature and by using a global feature extraction module to determine the first global feature from the image;
enabling a user to specify a search input; and
determining a search result comprising
(i) one or more first images that depict objects that individually have a global feature which satisfies a criteria identified from the input, and
(ii) one or more second images that depict objects that individually have a local feature that satisfies the criteria.

2. The method of claim 1 , wherein enabling a user to specify the search input includes enabling the user to select a parameter that corresponds to a color, a shape or a pattern.

3. The method of claim 2 , wherein determining a search result includes comparing image data, corresponding to the search input, to local and global features of objects depicted in at least some images of the collection, in determining N objects in images that comprise a portion of the collection, which have at least a segment that is deemed most similar in appearance to the image data.

4. The method of claim 2 , wherein enabling a user to specify the search input includes enabling the user
(i) to select an object depicted in a given image, and
(ii) to specify a parameter that selects or weights a visual characteristic that corresponds to either a local or global feature of the object in the given image.

5. The method of claim 2 , wherein enabling the user to specify the search input includes enabling the user to select an object depicted in an image and a parameter that corresponds to a shape or a pattern.

6. The method of claim 2 , wherein enabling the user to specify the search input includes enabling the user to select an object depicted in an image and a parameter that corresponds to criteria color value.

7. The method of claim 6 , wherein enabling the user to specify the parameter includes enabling the user to specify a parameter value by operating a graphical feature that is in form of a slider that can be moved in between a range of values for a given visual characteristic.

8. The method of claim 1 , wherein the global feature module includes at least one of a color feature extraction component, a shape feature component, or a text feature component, and
wherein determining at least a first global feature comprises determining at least one of a global color feature, a global shape feature, or a global text feature of the object in the image.

9. The method of claim 1 , wherein determining at least a first local feature comprises:
identifying one or more segments of the object in the image; and
executing a local feature extraction module to determine the first local feature using the identified segments.

10. The method of claim 1 , wherein determining a search result comprises:
determining the one or more first images that depict objects that individually have a global feature which satisfies a criteria identified from the input; and
determining, from the one or more first images, the one or more second images that depict objects that individually have a local feature that satisfies the criteria.

11. The method of claim 1 , wherein programmatically analyzing each image in a collection of images comprises:
segmenting content of the image into a foreground image and a background image; and
segmenting the individual objects from one another in the foreground image, the individual objecting being separated from the background image.

12. A system, comprising:
a data processing apparatus; and
a non-transitory computer readable storage medium in data communication with the data processing apparatus and storing instructions executable by the data processing apparatus and upon such execution cause the data processing to perform operations comprising:
programmatically analyzing each image in a collection of images, including determining at least a local feature and a global feature for individual objects that appear in each image of the collection, the local feature of each object corresponding to a visual characteristic of a segment of that object, the global feature of each object corresponding to a visual characteristic of the object as a whole, wherein determining at least a local feature and a global feature for individual objects that appear in each image of the collection comprises, for each object in the image, determining at least a first global feature independently from determining at least a first local feature and by using a global feature extraction module to determine the first global feature from the image;
enabling a user to specify a search input; and
determining a search result comprising
(i) one or more first images that depict objects that individually have a global feature which satisfies a criteria identified from the input, and
(ii) one or more second images that depict objects that individually have a a local feature that satisfies the criteria.

13. The system of claim 12 , wherein enabling a user to specify the search input includes enabling the user to select a parameter that corresponds to a color, a shape or a pattern.

14. The system of claim 12 , wherein determining a search result includes comparing image data, corresponding to the search input, to local and global features of objects depicted in at least some images of the collection, in determining N objects in images that comprise a portion of the collection, which have at least a segment that is deemed most similar in appearance to the image data.

15. The system of claim 12 , wherein enabling a user to specify the search input includes enabling the user
(i) to select an object depicted in a given image, and
(ii) to specify a parameter that selects or weights a visual characteristic that corresponds to either a local or global feature of the object in the given image.

16. The system of claim 12 , wherein enabling the user to specify the search input includes enabling the user to select an object depicted in an image and a parameter that corresponds to a shape or a pattern.

17. The system of claim 12 , wherein enabling the user to specify the search input includes enabling the user to select an object depicted in an image and a parameter that corresponds to criteria color value.

18. The system of claim 12 , wherein enabling the user to specify the parameter includes enabling the user to specify a parameter value by operating a graphical feature that is in form of a slider that can be moved in between a range of values for a given visual characteristic.

19. The system of claim 12 , wherein the global feature module includes at least one of a color feature extraction component, a shape feature component, or a text feature component,
wherein determining at least a

Assignees

• Gokturk Salih Burak
• Sumengen Baris
• Vu Diem
• Dalal Navneet
• Yang Danny
• Lin Xiaofan
• Khan Azhar
• Shah Munjal
• Anguelov Dragomir
• Torresani Lorenzo
• Vanhoucke Vincent
• Google Inc

Inventors

• Gokturk Salih Burak
• Sumengen Baris
• Vu Diem
• Dalal Navneet
• Yang Danny
• Lin Xiaofan
• Khan Azhar
• Shah Munjal
• Anguelov Dragomir
• Torresani Lorenzo
• Vanhoucke Vincent

Classifications

• G06Q30/0603 Primary

  Catalogue creation or management · CPC title

• G06F17/30259

  Physics · mapped topic

• G06K9/46 Primary

  Physics · mapped topic

• Y10S707/949

  Inventory · CPC title

• G06K9/4652

  Physics · mapped topic