Visual object instance descriptor for place recognition

Patent metadata

Field	Value
Publication number	US-11295162-B2
Application number	US-201916671525-A
Country	US
Kind code	B2
Filing date	Nov 1, 2019
Priority date	Nov 1, 2019
Publication date	Apr 5, 2022
Grant date	Apr 5, 2022

Abstract

An approach to place recognition from an image makes use of the detection of objects at a set of known places as well as at an unknown place. Images of the detected objects in an image of the unknown place are processed to yield respective numerical descriptors, and these descriptors are used to compare the unknown place to the known places to recognize the unknown place. At least some embodiments make use of a trained parameterized image processor to transform an image of an object to an object descriptor, and the training of the processor is meant to preserve distinctions between different instances of a type of object, as well as distinctions between entirely different types of objects.

First claim

What is claimed is:

1. A method for place recognition, comprising:
receiving an image acquired for an unknown place;
computing a set of descriptors from the image for the unknown place, the computing including
locating objects from a first set of predefined classes in the image, including determining a corresponding region of the image for each located object, and
computing each descriptor to comprise a numerical vector representing a respective located object based at least on the associated region of the image;

accessing a database of reference records, each reference record being associated with a known place and including a set of descriptors previously computed for objects in a corresponding image of said known place;
matching the set of descriptors from the image for the unknown place with the reference records of the database, including
for each reference record of at least some of the reference records, determining a score representing a match between descriptors computed from the image of the unknown place and descriptors of the reference record, and
determining a best matching reference record of the database; and

recognizing the unknown place as being a place associated with the best matching record of the database.

2. The method of claim 1 , wherein computing the set of descriptors from the image for the unknown place includes:
processing the image using an object locator to determine
the set of objects in the image,
a region of the image associated with each object of the set of objects, and
processed features of the image associated with each region; and

computing the descriptors using a descriptor generator, including for each object of the set of objects,
inputting the processed features and
producing the corresponding descriptor as output.

3. The method of claim 2 , wherein the object locator comprises a first artificial neural network, and the descriptor generator comprises a second artificial neural network.

4. The method of claim 3 , wherein determining the processed features of the image for an object includes using intermediate values generated within the first artificial neural network for the determined region of the object in the image.

5. The method of claim 3 , further comprising configuring the first artificial neural network with values of first configuration parameters, said values having been determined from a first training corpus comprising images annotated with objects according to the first set of object classes.

6. The method of claim 5 , further comprising configuring the second artificial neural network with values of second configuration parameters, said values having been determined from a second training corpus of images annotated with objects according to a second set of object classes, and of instances of objects within said classes, the second training corpus including multiple images with a same instance of an object, and images with multiple different instances of an object class of the second set of object classes.

7. The method of claim 6 , wherein the second set of object classes is different than the first set of object classes, and the first training corpus is different than the second training corpus.

8. The method of claim 6 , wherein the values of the second configuration parameters are selected to distinguish different instances of an object class in the second set of object classes.

9. The method of claim 6 , further comprising determining the values of the first configuration parameters using an artificial neural network training procedure using the first training corpus.

10. The method of claim 6 , further comprising determining the values of the second configuration parameters using an artificial neural network training procedure using the second training corpus.

11. The method of claim 10 , wherein the training procedure used to determine the values of the second configuration parameters comprises optimizing discriminability of object instances within each object class of the second set of object classes.

12. The method of claim 1 , wherein matching the set of descriptors from the image for the unknown place with the records of the database, includes matching the set of descriptors for the unknown place with a first set of descriptors of a first record of the database, including:
determining a score between pairs of descriptors, each pair including one descriptor from the set of descriptors from the image for the unknown place and one descriptor from the first set of descriptors from the database;
determining a best match of some or all of the descriptors of set of descriptors from the image for the unknown place each with a respective descriptor of the first set of descriptors, the best match having a corresponding best score determined from the scores between pairs of descriptors.

13. The method of claim 12 , wherein matching the set of descriptors for the unknown place with a first set of descriptors of a first record of the database includes performing a graph matching procedure on a bipartite graphs with edge weights determined from the scores between pairs of descriptors.

14. The method of claim 1 , wherein locating the objects includes using a first artificial neural network configured with values determined from a first training corpus comprising images annotated with objects according to the first set of predefined classes.

15. The method of claim 1 , wherein matching the set of descriptors from the image for the unknown place with the records of the database includes matching the set of descriptors for the unknown place with a first set of descriptors of a first record of the database, including:
determining correspondences between respective descriptors from the image for the unknown place and descriptors from the first set of descriptors.

16. A computer-implement place recognition system, comprising:
an object locator configured to accept an image of an unknown place and configurable to locate objects from a set of predefined classes in the image, and to determine an associated region of the image for each located object;
descriptor generator configurable to, for each object of the set of objects located by the object locator, compute a descriptor comprising a numerical vector representing a respective located object based at least on the associated region of the image;
a database for storing a plurality of reference records, each record has an association with a known place, and a set of descriptors for objects associated with said known place; and
a matcher configured to, for each reference record of at least some of the reference records, determine a score representing a match between descriptors computed from the image of the unknown place and descriptors of the reference record, and
wherein the place recognition system is configured to recognize the unknown place according to the best matching record.

17. The place recognition system of claim 16 , wherein the object locator comprises a first artificial neural network, and the descriptor generator comprises a second artificial neural network, configurable according to values of first configuration parameters and second configuration parameters, respectively.

18. The place recognition system of claim 17 configured with values of the first and the second configuration parameters, the values of the first configuration parameters having been determined from a first training corpus comprising images annotated with objects according to the first set of object classes, and values of the second configuration par

Assignees

• Massachusetts Inst Technology
• Eth Zuerich

Inventors

• Rus Daniela
• Karaman Sertac
• Gilitschenski Igor
• Cramariuc Andrei
• Cadena Cesar
• Siegwart Roland

Classifications

• G06V20/35 Primary

  Categorising the entire scene, e.g. birthday party or wedding scene · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F18/2431

  Multiple classes · CPC title