Method and system for facilitating identification of an object-of-interest

Patent metadata

Field	Value
Publication number	US-10891509-B2
Application number	US-201816172557-A
Country	US
Kind code	B2
Filing date	Oct 26, 2018
Priority date	Oct 27, 2017
Publication date	Jan 12, 2021
Grant date	Jan 12, 2021

Abstract

There are described methods and systems for facilitating identification of an object-of-interest. A face similarity score and a body similarity score of a query image relative to a gallery image are determined. A fused similarity score of the query image relative to the gallery image is determined by applying a relationship between the face similarity score, the body similarity score, and the fused similarity score. The fused similarity score is indicative of whether or not the object-of-interest and the potential object-of-interest are the same object-of-interest. For example, a machine learning process is used to fuse the face similarity score and the body similarity into the fused similarity score. The process is repeated for multiple gallery images. The gallery images may then be ranked according to their respective fused similarity scores.

First claim

The invention claimed is:

1. A method of facilitating identification of a person-of-interest, comprising:
for each of multiple gallery images:
determining a face similarity score of a query image relative to the gallery image, wherein the query image comprises at least a portion of a person-of-interest, and wherein the gallery image comprises at least a portion of a potential person-of-interest;
determining a body similarity score of the query image relative to the gallery image; and
determining a fused similarity score of the query image relative to the gallery image by applying a relationship between the face similarity score, the body similarity score, and the fused similarity score, wherein the fused similarity score is indicative of whether or not the person-of-interest and the potential person-of-interest are the same person-of-interest;

ranking the gallery images according to each fused similarity score of the respective gallery images; and
transmitting the gallery images for display according to the ranking,
wherein at least one of:
(1) determining the face similarity score comprises:
determining whether either a face of the person-of-interest is absent from the query image or a face of the potential person-of-interest is absent from the gallery image; and
if so, determining the face similarity score to be a default face similarity score; and

(2) determining the body similarity score comprises:
determining whether either a body of the person-of-interest is absent from the query image or a body of the potential person-of-interest is absent from the gallery image; and
if so, determining the body similarity score to be a default body similarity score.

2. The method of claim 1 , further comprising determining the relationship.

3. The method of claim 2 , wherein determining the relationship comprises:
determining reference face similarity scores and reference body similarity scores for pairs of multiple reference images, wherein each reference image comprises at least a portion of a reference person-of-interest;
determining, based on the reference face similarity scores and the reference body similarity scores, reference fused similarity scores by constraining the reference fused similarity scores as a function of whether or not the pairs of the reference images comprise at least portions of the same reference persons-of-interest; and
determining the relationship based on the reference face similarity scores, the reference body similarity scores and the reference fused similarity scores.

4. The method of claim 3 , wherein one or more of the determining the reference face similarity scores and the reference body similarity scores, the determining the reference fused similarity scores, and the determining the relationship, are performed by a machine learning module configured to implement a machine learning process.

5. The method of claim 4 , wherein the machine learning module is configured to implement a neural network to which are input the multiple reference images.

6. The method of claim 5 , wherein the neural network is a multi-layer, soft-max neural network.

7. The method of claim 1 , wherein determining one or more of the face similarity score and the body similarity score comprises:
using a convolutional neural network to determine one or more similarity signatures between the query image and the gallery image; and
determining one or more separations between the one or more similarity signatures, wherein one or more of the face similarity score and the body similarity score are a function of the one or more separations.

8. The method of claim 1 , wherein the at least a portion of the person-of-interest comprises at least a portion of: a body of the person-of-interest; a torso of the person-of-interest; a face of the person-of-interest; and a body and a face of the person-of-interest.

9. The method of claim 1 , wherein the at least a portion of the potential person-of-interest comprises at least a portion of: a body of the potential person-of-interest; a torso of the potential person-of-interest; a face of the potential person-of-interest; and a body and a face of the potential person-of-interest.

10. The method of claim 1 , wherein determining the face similarity score comprises:
determining whether either a face of the person-of-interest is absent from the query image or a face of the potential person-of-interest is absent from the gallery image; and
if so, determining the face similarity score to be a default face similarity score.

11. The method of claim 10 , wherein the default face similarity score comprises a median of face similarity scores determined for the query image relative to other ones of the multiple gallery images.

12. The method of claim 1 , wherein determining the body similarity score comprises:
determining whether either a body of the person-of-interest is absent from the query image or a body of the potential person-of-interest is absent from the gallery image; and
if so, determining the body similarity score to be a default body similarity score.

13. The method of claim 12 , wherein the default face similarity score comprises a median of body similarity scores determined for the query image relative to other ones of the multiple gallery images.

14. The method of claim 1 , wherein, prior to determining the fused similarity score, the method further comprises adjusting at least one of the face similarity score and the body similarity score based on an image quality metric of one or more of the query image and the gallery image.

15. The method of claim 14 , wherein the image quality metric comprises an image resolution.

16. The method of claim 1 , wherein, prior to determining the fused similarity score, the method further comprises adjusting at least one of the face similarity score and the body similarity score based on time data associated with one or more of the query image and the gallery image.

17. The method of claim 16 , wherein the time data is indicative of when one or more of the query image and the gallery image were taken.

18. The method of claim 16 , wherein the method further comprises:
determining whether the time data meets a criterion; and
if not, determining one or more of the face similarity score and the body similarity score to be one or more of a default face similarity score and a default body similarity score.

19. The method of claim 1 , wherein the query image is part of a set of multiple query images, and wherein the method further comprises, prior to determining the fused similarity score, adjusting one or more of the face similarity score and the body similarity score based on:
a similarity of the query image relative to: a second query image of the multiple query images; or a second one of the multiple gallery images; and
a similarity of the second query image or the second gallery image relative to: a third query image of the multiple query images; or a third one of the multiple gallery images.

20. The method of claim 19 wherein the similarity comprises one or more of a face similarity score and a body similarity score.

21. A system for facilitating identification of a person-of-interest, comprising:
a display;
a processor; and
a memory communicatively coupled to the processor and having stored thereon computer program code executable by the processor and configured, when executed by the processor, to cause the processor to perform a method comprising:
for each of multiple gallery images:
deter

Assignees

• Avigilon Corp

Inventors

• Doumbouya Moussa
• He Lu
• Hu Yanyan
• Saptharishi Mahesh
• Zhang Hao
• Alcock Nicholas John
• Donaldson Roger David
• Azizabadifarahani Seyedmostafa
• Jessen Ken

Classifications

• G06V40/179

  metadata assisted face recognition · CPC title

• G06V10/811

  the classifiers operating on different input data, e.g. multi-modal recognition · CPC title

• G06V40/10

  Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title