System and method for locating and performing fine grained classification from multi-view image data

Patent metadata

Field	Value
Publication number	US-10534960-B2
Application number	US-201715478101-A
Country	US
Kind code	B2
Filing date	Apr 3, 2017
Priority date	Apr 1, 2016
Publication date	Jan 14, 2020
Grant date	Jan 14, 2020

Abstract

Some embodiments of the invention provide a method for identifying geographic locations and for performing a fine-grained classification of elements detected in images captured from multiple different viewpoints or perspectives. In several embodiments, the method identifies the geographic locations by probabilistically combining predictions from the different viewpoints by warping their outputs to a common geographic coordinate frame. The method of certain embodiments performs the fine-grained classification based on image portions from several images associated with a particular geographic location, where the images are captured from different perspectives and/or zoom levels.

First claim

What is claimed is:

1. A method for locating and classifying elements from images, the method comprising:
receiving a plurality of geo-referenced images, the plurality of geo-referenced images comprising a first image of a first perspective and a second image of a second perspective; identifying portions of the first image corresponding to elements detected in the first image and portions of the second image corresponding to elements detected in the second image;
identifying a geographic location for each element detected in the first image using the portion of the first image corresponding to the respective element for which the geographic location is identified;
identifying a geographic location for each element detected in the second image using the portion of the second image corresponding to the respective element for which the geographic location is identified;
identifying first portions of the portions of the first image corresponding to second portions of the portions of the second image based on the identified geographic locations, the corresponding first and second portions corresponding to a particular element of the elements detected in the first and second images, where the particular element is visible in the corresponding first and second portions; and
performing a fine-grained classification of the particular element based on the identified corresponding first and second portions.

2. The method of claim 1 , wherein one of the first and second images comprises crowd-sourced images.

3. The method of claim 1 , wherein the first perspective is an aerial view and the second perspective is a street-level view.

4. The method of claim 1 , wherein the second image is captured at a first zoom level, wherein the plurality of geo-referenced images comprises a third image, wherein the third image is captured at a different, second zoom level, wherein the method further comprises:
identifying portions of the third image corresponding to elements detected in the third image;
identifying a geographic location for each element detected in the third image using the portion of the third image corresponding to the respective element for which the geographic location is identified; and
identifying third portions of the portions of the third image corresponding to the first and second portions based on the identified geographic locations, where the particular element is visible in the corresponding portions of the third image, wherein performing the fine-grained classification is further based on the third portions of the third image.

5. The method of claim 1 , wherein the second image is captured at a first resolution, wherein the plurality of geo-referenced images comprises a third image, wherein the third image is captured at a different, second resolution, wherein the method further comprises:
identifying portions of the third image corresponding to elements detected in the third image;
identifying a geographic location for each element detected in the third image using the portion of the third image corresponding to the respective element for which the geographic location is identified; and
identifying third portions of the portions of the third image corresponding to the first and second portions based on the identified geographic locations, where the particular element is visible in the corresponding portions of the third image, wherein performing the fine-grained classification is further based on the third portions of the third image.

6. The method of claim 3 , wherein the second image is captured from a first position at the street-level view, wherein the plurality of geo-referenced images comprises a third image, wherein the third image is captured at a different, second position at the street-level view, wherein the method further comprises:
identifying portions of the third image corresponding to elements detected in the third image;
identifying a geographic location for each element detected in the third image using the portion of the third image corresponding to the respective element for which the geographic location is identified; and
identifying third portions of the portions of the third image corresponding to the first and second portions based on the identified geographic locations, where the particular element is visible in the corresponding portions of the third image, wherein performing the fine-grained classification is further based on the third portions of the third image.

7. The method of claim 6 , wherein the method is performed by a driver assistance system of a vehicle, wherein the second and third images are gathered from the driver assistance system.

8. The method of claim 6 , wherein the second and third images are panorama images captured by a 360 degree camera mounted on a vehicle.

9. The method of claim 1 , wherein the plurality of geo-referenced images comprises images that capture at least a portion of the non-visible light spectrum.

10. The method of claim 1 , wherein identifying a geographic location for a given element detected in the first image comprises:
identifying a first set of bounding regions in the first image and a second set of bounding regions in the second image, wherein elements are detected in the first and second sets of bounding regions;
computing detection scores for bounding regions of the first and second sets of bounding regions;
identifying a geographic location for each bounding region of the first and second sets of bounding regions;
identifying a set of potential geographic locations for the given element based on correlations between the identified geographic locations for the first and second sets of bounding regions;
calculating multi-view detection scores for each potential geographic location of the set of potential geographic locations for the given element based on the computed detection scores for bounding regions associated with the potential geographic location; and
selecting the geographic location for the given element based on the calculated multi-view detection scores.

11. The method of claim 10 , wherein identifying the first and second sets of bounding regions comprises:
filtering bounding regions with a detection score below a particular threshold; and
filtering overlapping bounding regions using non-maximal suppression.

12. The method of claim 10 , wherein identifying the geographic location for each bounding region comprises warping the bounding region to corresponding geographic locations in a geographic coordinate system.

13. The method of claim 12 , wherein warping the bounding region comprises using a set of projection functions to compute a set of geographic coordinates for the bounding region.

14. The method of claim 10 , wherein calculating the multi-view detection scores for each potential geographic location comprises combining the computed detection scores for the bounding regions associated with the potential geographic location.

15. The method of claim 1 , wherein identifying the first and second portions of the first and second images comprises generating a bounding region within at least one of the first and second images for the particular element based on a set of inverse projection functions that identify a location and size for the bounding region based on a distance from a camera used to capture the at least one image to the associated geographic location.

16. The method of claim 1 , wherein the elements are of a particular category, wherein performing the fine-grained classification comprises classifying the particular element within a sub-category of the particular category.

Assignees

• California Inst Of Techn
• Eth Zuerich

Inventors

• Perona Pietro
• Branson Steven J
• Wegner Jan D
• Hall David C

Classifications

• G06V10/809

  of classification results, e.g. where the classifiers operate on the same input data · CPC title

• G06V20/13

  Satellite images · CPC title

• G06T7/74 Primary

  involving reference images or patches · CPC title

• G06F18/254

  of classification results, e.g. of results related to same input data · CPC title

• G06T2207/10032

  Satellite or aerial image; Remote sensing · CPC title