Cascaded multi-tier visual search system

Patent metadata

Field	Value
Publication number	US-10242099-B1
Application number	US-201213448049-A
Country	US
Kind code	B1
Filing date	Apr 16, 2012
Priority date	Apr 16, 2012
Publication date	Mar 26, 2019
Grant date	Mar 26, 2019

Abstract

A system for multi-tier visual recognition comprises a processor and a memory. The processor is configured to calculate a rank for each of a plurality of object models. The plurality of object models are used for visual recognition. Each of the plurality of object models has a size. The processor is configured to divide the plurality of object models into a plurality of sets based at least in part on the rank for each of the plurality of object models, and provide a first set of the plurality of sets to a first tier and a second set of the plurality of sets to a second tier. The memory coupled to the processor and configured to provide the processor with instructions.

First claim

What is claimed is:

1. A system for search of image data, comprising:
a processor; and
a memory having stored thereon instructions that, when executed by the processor, cause the processor to:
receive the image data from a client device of a user,
obtain a plurality of object models used for visual recognition of objects in the image data, wherein each of the plurality of object models is associated with an image object capable of being identified within an image,
determine a rank for each of the plurality of object models,
identify a first set of the plurality of object models to be used by the client device and a second set of the plurality of object models to be used by a server based on the rank, and
provide the first set of the plurality of object models to the client device and the second set of the plurality of object models to the server so that the client device performs a first search for a first set of objects in the image data based on the first set of the plurality of object models and the server performs a second search for a second set of objects in the image data based on the second set of the plurality of object models.

2. The system as in claim 1 , wherein the first set of the plurality of object models are identified based, at least in part, upon storage capacity of the client device.

3. The system as in claim 2 , wherein the first set of the plurality of object models are identified based, at least in part, upon a size associated with each object model and the rank associated with each object model.

4. The system as in claim 1 , wherein the visual recognition comprises recognizing at least one of: barcode type, icons, logos, advertisements, media, brands, products, geography, and structures.

5. The system as in claim 4 , wherein:
at least one of the server and the client device are configured to provide a result for a first object identified in the image data using one of the plurality of object models; and
the result comprises one or more of: text tag, metadata, results page, results indicator, vocalized tag, visual animation, an action such as a redirection to a webpage or a video, a visual ranking of objects, or a vocalized ranking of objects.

6. A method implemented on a computing device having at least one processor, storage, and communication platform for search of image data, comprising:
receiving the image data from a client device of a user;
obtaining a plurality of object models used for visual recognition of objects in the image data, wherein each of the plurality of object models is associated with an image object capable of being identified within an image;
determining, by the processor, a rank for each of the plurality of object models;
identifying a first set of the plurality of object models to be used by the client device and a second set of the plurality of object models to be used by a server based on the rank; and
providing the first set of the plurality of object models to the client device and the second set of the plurality of object models to the server so that the client device performs a first search for a first set of objects in the image data based on the first set of the plurality of object models and the server performs a second search for a second set of objects in the image data based on the second set of the plurality of object models.

7. A computer program product for search of image data, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer executable instructions for:
receiving the image data from a client device of a user;
obtaining a plurality of object models used for visual recognition of objects in the image data, wherein each of the plurality of object models is associated with an image object capable of being identified within an image;
determining, by the processor, a rank for each of the plurality of object models;
identifying a first set of the plurality of object models to be used by the client device and a second set of the plurality of object models to be used by a server based on the rank; and
providing the first set of the plurality of object models to the client device and the second set of the plurality of object models to the server so that the client device performs a first search for a first set of objects in the image data based on the first set of the plurality of object models and the server performs a second search for a second set of objects in the image data based on the second set of the plurality of object models.

8. The method as in claim 6 , further comprising:
determining a change in the rank of at least one of the plurality of object models;
wherein the first set of the plurality of object models to be provided to the client device is identified based, at least in part, on the change in the rank of the at least one of the plurality of object models.

9. The method as in claim 8 , wherein the change in the rank of at least one of the plurality of object models is triggered by at least one of: a change of location of the client device, a change of user preference, a new search history pattern, feedback from the user, a change of friend or family information, and a local event.

10. The method of claim 6 , further comprising:
determining that there is an update to at least one of the first set and the second set; and
transmitting the update to the at least one of the first set and the second set to the client device.

11. The method as in claim 10 , wherein a timing of the update being transmitted to the client device is based on one or more thresholds.

12. The method as in claim 11 , wherein at least one of the one or more thresholds is user configurable or tailored to the user.

13. The method as in claim 6 , wherein the client device comprises a mobile device, a fixed device, a phone, a tablet, a camera, a laptop, a desktop computer, a webcam, an appliance, or any other imaging device.

14. The method as in claim 6 , wherein the relevance is determined based, at least in part, upon at least one of a location of the user, a user preference, a search history pattern, feedback from the user, friend or family information, or a local event.

15. The method as in claim 6 , wherein the ranking is performed based, at least in part, upon an ontology feature or a collaborative feature.

16. The method of claim 6 , further comprising:
determining, in response to the client device performing the first search for the first set of objects in the image data, that at least one recognition result has been recognized at the client device; and
providing the at least one recognition result to the client device.

17. The method of claim 16 , further comprising:
determining, in response in response to the server performing the second search for the second set of objects in the image data, that at least one recognition result has been recognized at the server; and
providing, from the server, the at least one recognition result to the client device.

Assignees

• Garrigues Pierre
• Culpepper Benjamin Jackson
• Nguyen Huy X
• Cadieu Charles
• Stadlen Andrew
• Pesavento Gerry
• Warland David
• Oath Inc

Inventors

• Garrigues Pierre
• Culpepper Benjamin Jackson
• Nguyen Huy X
• Cadieu Charles
• Stadlen Andrew
• Pesavento Gerry
• Warland David

Classifications

• G06F17/30067

  Physics · mapped topic

• G06F17/30781 Primary

  Physics · mapped topic

• G06F16/583 Primary

  using metadata automatically derived from the content · CPC title

• G06F16/70 Primary

  of video data · CPC title

• G06F16/10

  File systems; File servers · CPC title