Image search using text-based elements within the contents of images

Patent metadata

Field	Value
Publication number	US-9323784-B2
Application number	US-96466210-A
Country	US
Kind code	B2
Filing date	Dec 9, 2010
Priority date	Dec 9, 2009
Publication date	Apr 26, 2016
Grant date	Apr 26, 2016

Abstract

A mobile device searches for electronic content. The mobile device captures an image from a rendered document, and searches for an electronic version of the image using characteristics of the image and using text within the contents of the image. The mobile device receives a result for the search based upon the image characteristics and the text within the context of the image.

First claim

We claim:

1. A method for searching for electronic content, the method comprising:
receiving, at a search system, a query image captured from a rendered document;
extracting, from the query image, (i) one or more textual keywords, and (ii) one or more non-text-based, image features;
selecting, from among a set of images, one or more result images that have one or more extracted textual keywords that are associated with one or more of the textual keywords that were extracted from the query image;
after the one or more result images that have one or more extracted textual keywords have been selected, selecting, from the one or more result images, a subset of the one or more result images that have one or more extracted non-text-based, image features that are associated with one or more of the non-text-based, image features that were extracted from the query image; and
in response to receiving the query image, providing the subset of the one or more result images that have (i) one or more extracted textual keywords that are associated with one or more of the textual keywords that were extracted from the query image, and (ii) one or more extracted non-text-based, image features that are associated with one or more of the non-text-based, image features that were extracted from the query image.

2. The method of claim 1 , further comprising:
determining that the captured image has a skew; and
adjusting the captured image to correct the skew.

3. The method of claim 2 , wherein adjusting the captured image to correct the skew comprises:
determining a correction to the skew for the text; and
applying the correction to the skew to the captured image.

4. The method of claim 1 , wherein extracting one or more non-text-based, image features of the query image comprises determining that the image comprises highly gradual content.

5. The method of claim 4 , wherein the selection comprises selecting an algorithm configured for use with highly gradual content.

6. The method of claim 1 , wherein extracting one or more non-text based, image features of the query image comprises determining that the image comprises high contrast content.

7. The method of claim 6 , wherein the selection comprises selecting an algorithm configured for determining edges and relationships of edges in the high contrast content.

8. The method of claim 1 , wherein the one or more non-text-based, image features comprises color, texture, and edge information.

9. The method of claim 1 , wherein the selecting is further based on image characteristics of the query image, the characteristics of the query image comprising date and time information for the query image.

10. A system, comprising:
logic circuitry, and
tangible non-transitory memory containing computer-readable instructions that, when processed by the logic circuitry, cause the system to perform functions comprising:
receiving, at a search system, a query image captured from a rendered document;
extracting, from the query image, (i) one or more textual keywords, and (ii) one or more non-text-based, image features;

selecting, from among a set of images, one or more result images that have one or more extracted textual keywords that are associated with one or more of the textual keywords that were extracted from the query image;
after the one or more result images that have one or more extracted textual keywords have been selected, selecting, from the one or more result images, a subset of the one or more result images that have one or more extracted non-text-based, image features that are associated with one or more of the non-text-based, image features that were extracted from the query image; and
in response to receiving the query image, providing the subset of the one or more result images that have (i) one or more extracted textual keywords that are associated with one or more of the textual keywords that were extracted from the query image, and (ii) one or more extracted non-text-based, image features that are associated with one or more of the non-text-based, image features that were extracted from the query image.

11. The system of claim 10 , wherein the functions further comprise:
determining that the captured image has a skew; and
adjusting the captured image to correct the skew.

12. The system of claim 11 , wherein adjusting the captured image to correct the skew comprises:
determining a correction to the skew for the text; and
applying the correction to the skew to the captured image.

13. The system of claim 10 , wherein extracting one or more non-text-based, image features of the query image comprises highly gradual content.

14. The system of claim 13 , wherein the selection comprises selecting an algorithm configured for use with highly gradual content.

15. The system of claim 10 , wherein extracting one or more non-text based, image features of the query image comprises determining that the image comprises high contrast content.

16. The system of claim 15 , wherein the selection comprises selecting an algorithm configured for determining edges and relationships of edges in the high-contrast content.

17. A non-transitory computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:
receiving, at a search system, a query image captured from a rendered document;
extracting, from the query image, (i) one or more textual keywords, and (ii) one or more non-text-based, image features;
selecting, from among a set of images, one or more result images that have one or more extracted textual keywords that are associated with one or more of the textual keywords that were extracted from the query image;
after the one or more result images that have one or more extracted textual keywords have been selected, selecting, from the one or more result images, a subset of the one or more result images that have one or more extracted non-text-based, image features that are associated with one or more of the non-text-based, image features that were extracted from the query image; and
in response to receiving the query image, providing the subset of the one or more result images that have (i) one or more extracted textual keywords that are associated with one or more of the textual keywords that were extracted from the query image, and (ii) one or more extracted non-text-based, image features that are associated with one or more of the non-text-based, image features that were extracted from the query image.

Assignees

• King Martin T
• Grunbock Cheryl
• Mannby Claes-Fredrik
• Smith Michael J
• Google Inc

Inventors

• King Martin T
• Mannby Claes-Fredrik
• Smith Michael J

Classifications

• G06F16/5846 Primary

  using extracted text · CPC title

• G06F17/30253 Primary

  Physics · mapped topic