Systems and methods for associating an image with a business venue by using visually-relevant and business-aware semantics

Patent metadata

Field	Value
Publication number	US-10198635-B2
Application number	US-201615001130-A
Country	US
Kind code	B2
Filing date	Jan 19, 2016
Priority date	Jan 19, 2016
Publication date	Feb 5, 2019
Grant date	Feb 5, 2019

Abstract

Systems and methods disclosed herein associate images with business venues. An example method includes: receiving a first image and retrieving textual reviews and stored images that are associated with one or more candidate business venues. The method further includes: detecting, using trained visual detectors, a plurality of business-aware concepts in the first image and assessing likelihood that detected business-aware concepts are in the first image. The method additionally includes: (i) generating a first representation of the first image based on the likelihoods and one or more term vectors for high-scoring concepts and (ii) receiving second representations of each candidate based on the retrieved textual reviews and stored images. In accordance with determining that the first representation is most similar to a respective second representation of a first candidate, the method includes: (i) associating the first image with the first candidate and (ii) providing an indication of the association.

First claim

What is claimed is:

1. A method of associating an image with a business venue, the method comprising:
at a computing device having a display, one or more input devices, one or more processors, and memory:
receiving, via the one or more input devices, a first image that is associated with a particular geographic area;
retrieving, from a repository that includes user-submitted textual reviews and stored images associated with a plurality of business venues, a first set of user-submitted textual reviews and a second set of stored images that are associated with one or more candidate business venues of the plurality of business venues, wherein each business venue of the one or more candidate business venues is located within a predetermined distance of the particular geographic area;
detecting, based on an analysis of the first image using trained visual detectors, a plurality of business-aware concepts in the first image, wherein each detected business-aware concept is associated with a score that reflects likelihood that the detected business-aware concept is present in the first image;
generating a first representation that corresponds to the first image based on the associated scores and based on one or more word representations for high-scoring business-aware concepts of the plurality of business-aware concepts;
receiving second representations that correspond to each of the one or more candidate business venues, wherein the second representations are based on the first set of user-submitted textual reviews and the second set of stored images; and
in accordance with determining that the first representation is most similar to a respective second representation of a first candidate business venue, (i) associating the first image with the first candidate business venue and (ii) providing an indication that the first image is associated with the first candidate business venue.

2. The method of claim 1 , wherein receiving the second representations includes:
detecting, based on an analysis of each stored image of the second set of stored images using the trained visual detectors, the plurality of business-aware concepts in a respective stored image, wherein each detected business-aware concept is associated with a score that reflects likelihood that the detected business-aware concept is present in the respective image, and further wherein a respective second representation of a first candidate business venue is based on: (i) zero or more associated scores and (ii) one or more word representations for respective user-submitted textual reviews that are associated with the first candidate business venue.

3. The method of claim 1 , wherein each concept of the plurality of business-aware concepts satisfies the following conditions: (i) the respective concept is business aware;
(ii) the respective concept is visually consistent; and (iii) the respective concept is discriminative for business venue recognition.

4. The method of claim 1 , wherein the plurality of business-aware concepts are identified by mining texts and images associated with business venues to identify the business-aware concepts.

5. The method of claim 1 , wherein the trained visual detectors are trained by using images that are relevant to the plurality of business-aware concepts and constructing visual concept detectors that are configured to match visual concepts in a query image with one or more of the business-aware concepts.

6. The method of claim 5 , wherein the trained visual detectors are trained before receiving the first image.

7. The method of claim 1 , wherein, before associating the first image with the first candidate business venue, the repository included no images that were associated with the first candidate business venue.

8. The method of claim 1 , wherein the provided indication includes a visual indication with a textual description indicating that the first image was taken at the first candidate business venue.

9. The method of claim 1 , wherein determining that the first representation is most similar to a respective second representation of a first candidate business venue includes determining that the respective second representation is closest to the first representation in a multi-dimensional space of the plurality of business-aware concepts.

10. The method of claim 1 , wherein, before receiving the first image, the plurality of business-aware concepts are identified by analyzing images and user-submitted textual reviews associated with business venues to locate those concepts that satisfy the following conditions: (i) the respective concept is business aware; (ii) the respective concept is visually consistent; and (iii) the respective concept is discriminative for business venue recognition.

11. The method of claim 1 , wherein associating the first image with the first business venue includes sending the first image for storage in the repository.

12. The method of claim 1 , wherein the received first image corresponds to an image taken in an indoor environment.

13. The method of claim 1 , wherein the received first image corresponds to an image taken in an outdoor environment.

14. The method of claim 1 , wherein the particular geographic area is associated with rough coordinates at which the first image was taken.

15. The method of claim 1 , wherein the particular geographic area is determined based on Wi-Fi signals detected by the computing device when the first image was taken.

16. The method of claim 1 , wherein the first representation is a multimodal representation of the first image.

17. The method of claim 1 , wherein the high-scoring business-aware concepts are associated with an assigned score that is above a score threshold.

18. The method of claim 1 , wherein the repository of user-submitted textual reviews and stored images associated with the plurality of business venues is maintained at a server system that is remotely located from the computing device.

19. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions which, when executed by a computing device with a display, one or more input devices, one or more processors, and memory, cause the computing device to:
receive, via the one or more input devices, a first image that is associated with a particular geographic area;
retrieve, from a repository of user-submitted textual reviews and stored images associated with a plurality of business venues, a first set of user-submitted textual reviews and a second set of stored images that are associated with one or more candidate business venues of the plurality of business venues, wherein each business venue of the one or more candidate business venues is located within a predetermined distance of the particular geographic area;
detect, based on an analysis of the first image using trained visual detectors, a plurality of business-aware concepts in the first image, wherein each detected business-aware concept is associated with a score that reflects likelihood that the detected business-aware concept is present in the first image;
generate a first representation that corresponds to the first image based on the associated scores and based on one or more word representations for high-scoring business-aware concepts of the plurality of business-aware concepts;
receive second representations that correspond to each of the one or more candidate business venues, wherein the second representations are based on the first set of user-submitted textual reviews and the second s

Assignees

• Fuji Xerox Co Ltd

Inventors

• Chen Bor-Chun
• Chen Yin-Ying
• Chen Francine R
• Joshi Dhiraj

Classifications

• G06Q10/40

  Business processes related to social networking or social networking services · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06K9/00691

  Physics · mapped topic

• G06T2207/20081

  Training; Learning · CPC title

• G06K9/00671 Primary

  Physics · mapped topic