Adaptable image search with computer vision assistance

Patent metadata

Field	Value
Publication number	US-9619488-B2
Application number	US-201414163999-A
Country	US
Kind code	B2
Filing date	Jan 24, 2014
Priority date	Jan 24, 2014
Publication date	Apr 11, 2017
Grant date	Apr 11, 2017

Abstract

A computing device having adaptable image search and methods for operating an image recognition program on the computing device are disclosed herein. An image recognition program may receive a query from a user and a target image within which a search based on the query is to be performed using one or more of a plurality of locally stored image recognition models, which are determined to be able to perform the search with sufficiently high confidence. The query may comprise text that is typed or converted from speech. The image recognition program performs the search within the target image for a target region of the target image using at least one selected image recognition model stored locally, and returns a search result to the user.

First claim

The invention claimed is:

1. A computing device having adaptable image search, the computing device comprising:
non-volatile memory configured to store a plurality of image recognition models;
an image recognition program executed by a processor of the computing device, the computing device being a user computing device, and the image recognition program configured to:
receive a query from a user, the query comprising text that is typed or converted from speech;
receive a target image within which a search based on the query is to be performed;
rank the image recognition models by confidence level for performing the search based on at least a comparison between the query and respective text descriptions of the image recognition models;
determine whether the confidence level of any of the image recognition models is above a confidence threshold; and
upon determining that at least one confidence level of the image recognition models is above the confidence threshold,
select at least one of the image recognition models whose confidence level is above the confidence threshold;
perform the search within the target image for a target region of the target image using at least one selected image recognition model locally on the processor; and
return a search result to the user.

2. The computing device of claim 1 , wherein the target image is a single image or one or a plurality of image frames that constitute a portion of video.

3. The computing device of claim 1 , wherein each image recognition model includes at least one of the following: an image recognition algorithm, an optical character recognition (OCR) algorithm, and a keyword matching algorithm.

4. The computing device of claim 1 , wherein the image recognition program uses location information about the user.

5. The computing device of claim 1 , wherein the target image is an image or video of the user's current surroundings and the query indicates a desired product or destination.

6. The computing device of claim 1 , wherein the computing device comprises a camera and wherein the target image is captured by the camera.

7. The computing device of claim 1 , wherein the computing device is a smartphone or tablet.

8. The computing device of claim 1 , wherein the computing device is a watch or other wearable device.

9. The computing device of claim 1 , wherein the image recognition program displays the search result on a head-up display.

10. The computing device of claim 1 , wherein the query includes a directive to search for target regions in the target image or in other images that are similar to the target region in the target image.

11. A method for operating an image recognition program on a computing device having adaptable image search, the method comprising:
executing the image recognition program on a processor of the computing device, the computing device being a user computing device;
receiving a query from a user, the query comprising text that is typed or converted from speech;
receiving a target image within which a search based on the query is to be performed;
ranking a plurality of image recognition models by confidence level for performing the search based on at least a comparison between the query and respective text descriptions of the image recognition models, wherein the image recognition models are stored in non-volatile memory of the computing device;
determining whether the confidence level of any of the image recognition models is above a confidence threshold; and
upon determining that at least one confidence level of the image recognition models is above the confidence threshold,
selecting at least one of the image recognition models whose confidence level is above the confidence threshold;
performing the search within the target image for a target region of the target image using at least one selected image recognition model locally on the processor; and
returning a search result to the user.

12. The method of claim 11 , wherein each image recognition model includes at least one of the following: an image recognition algorithm, an optical character recognition (OCR) algorithm, and a keyword matching algorithm.

13. The method of claim 11 , wherein the image recognition program uses location information about the user.

14. The method of claim 11 , wherein the target image is an image or video of the user's current surroundings and the query indicates a desired product or destination.

15. The method of claim 11 , wherein the computing device comprises a camera and wherein the target image is captured by the camera.

16. The method of claim 11 , wherein the computing device is a smartphone or tablet.

17. The method of claim 11 , wherein the image recognition program displays the search result on a head-up display.

18. The method of claim 11 , further comprising, upon determining that none of the confidence levels of the image recognition models is above the confidence threshold:
connecting to a web service including a plurality of image recognition models;
receiving a ranking of image recognition models of the web service by confidence level for performing the search based on the query within the target image;
determining whether any of the image recognition models of the web service is above the confidence threshold for performing the search; and
upon determining that at least one confidence level of the image recognition models of the web service is above the confidence threshold,
selecting at least one of the image recognition models of the web service whose confidence level is above the confidence threshold;
downloading at least one selected image recognition model to the computing device;
performing the search within the target image for a target region of the target image using at least one downloaded image recognition model; and
returning the search result to the user.

19. The method of claim 11 , further comprising, upon determining that none of the confidence levels of the image recognition models is above the confidence threshold:
performing a web image search in an image search engine, wherein the web image search is based on the query;
returning a predetermined number of example images;
selecting at least one example image from the predetermined number of example images;
creating an image recognition model based on at least one selected example image;
storing the created image recognition model in the non-volatile memory of the computing device;
performing the search within the target image for a target region of the target image using the created image recognition model; and
returning the search result to the user.

20. A computing device having adaptable image search, the computing device comprising:
non-volatile memory configured to store a plurality of image recognition models;
a camera, wherein a target image is captured by the camera and wherein the target image is a single image or one or a plurality of image frames that constitute a portion of video;
a head-up display; and
an image recognition program executed by a processor of the computing device, the computing device being a user computing device, and the image recognition program configured to:
receive a query from a user, the query comprising text that is typed or converted from speech;
receive the target image within which a search based on the query is to be performed;
rank the image recognition models by confidence level for performing the search based on at least a comparison between the query and respective te

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Ambardekar Amol Ashok
• Huybregts Christopher Leonard
• Wall Larry
• Houshangi Damoun
• Pathak Hrishikesh

Classifications

• G06V10/87

  using selection of the recognition techniques, e.g. of a classifier in a multiple classifier system · CPC title

• G06F18/285

  Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system · CPC title

• G06F16/583

  using metadata automatically derived from the content · CPC title

• G06F16/1734

  Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs · CPC title

• G06F16/24578

  using ranking · CPC title