Systems and methods of person recognition in video streams

Patent metadata

Field	Value
Publication number	US-10410086-B2
Application number	US-201715676848-A
Country	US
Kind code	B2
Filing date	Aug 14, 2017
Priority date	May 30, 2017
Publication date	Sep 10, 2019
Grant date	Sep 10, 2019

Abstract

The various implementations described herein include systems and methods for recognizing persons in video streams. In one aspect, a method includes: (1) obtaining a live video stream; (2) detecting person(s) in the stream; and (3) determining, from analysis of the live video stream, personally identifiable information of the detected person(s); (4) determining, based on the personally identifiable information, that the first person is not known to the computing system; (5) in accordance with the determination that the first person is not known: (a) storing the personally identifiable information; and (b) requesting a user to classify the first person; and (6) in accordance with (i) a determination that a predetermined amount of time has elapsed since the request was transmitted and a response was not received, or (ii) a determination that a response was received classifying the first person as a stranger, deleting the stored personally identifiable information.

First claim

What is claimed is:

1. A method comprising:
at a computing system having one or more processors and memory:
obtaining a live video stream;
detecting one or more persons in the live video stream;
determining, from analysis of the live video stream, personally identifiable information that identifies a first person of the detected one or more persons;
determining, based on at least some of the personally identifiable information, that the first person is not a known person to the computing system;
in accordance with the determination that the first person is not a known person:
storing at least some of the personally identifiable information; and
requesting a user to classify the first person; and

in accordance with (i) a determination that a predetermined amount of time has elapsed since the request was transmitted and a response was not received from the user, or (ii) a determination that a response was received from the user classifying the first person as a stranger, deleting the stored personally identifiable information.

2. The method of claim 1 , wherein determining the personally identifiable information comprises:
selecting one or images of the first person from the live video stream; and
characterizing a plurality of features of the first person based on the one or more images.

3. The method of claim 2 , further comprising:
identifying a pose of the first person in each of the one or more images; and
for each identified pose, in accordance with a determination that an image of the first person in the identified pose has not previously been stored, storing (i) the image of the first person in the identified pose, and (ii) corresponding pose information.

4. The method of claim 2 , wherein the plurality of features includes one or more features that can personally identify the first person.

5. The method of claim 2 , wherein the plurality of features includes one or more facial features of the first person.

6. The method of claim 1 , further comprising:
receiving, from the user, a designation that the first person is not a person; and
responsive to the designation, storing the designation in association with the personally identifiable information of the first person.

7. The method of claim 1 , further comprising, in accordance with the determination that the first person is not a known person, generating a timely notification of detection of an unknown person.

8. The method of claim 1 , further comprising:
detecting a second person in the live video stream;
characterizing a plurality of features of the second person based on one or more images of the second person in the live video stream; and
determining whether the second person is a same person as the first person by comparing the plurality of characterized features of the second person with the stored information regarding the plurality of characterized features of the first person.

9. A computing system comprising:
one or more processors; and
memory coupled to the one or more processors, the memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for:
obtaining a live video stream;
detecting one or more persons in the live video stream;
determining, from analysis of the live video stream, personally identifiable information that identifies a first person of the detected one or more persons;
determining, based on at least some of the personally identifiable information, whether the first person is a known person to the computing system;
in accordance with a determination that the first person is not a known person:
storing at least some of the personally identifiable information; and
requesting a user to classify the first person; and

in accordance with (i) a determination that a predetermined amount of time has elapsed since the request was transmitted and a response was not received from the user, or (ii) a determination that a response was received from the user classifying the first person as a stranger, deleting the stored personally identifiable information.

10. The system of claim 9 , wherein detecting the one or more persons comprises detecting the one or more persons within a first predefined threshold time.

11. The system of claim 9 , wherein the one or more programs further include instructions for comparing the personally identifiable information of the first person with stored personally identifiable information of one or more persons previously classified by the user; and
wherein determining that the first person is not a known person comprises determining that the personally identifiable information of the first person does not match, within a predetermined threshold, the stored personally identifiable information of the one or more persons previously classified by the user.

12. The system of claim 9 , wherein the one or more programs further include instructions for:
determining that the first person is a known person to the computing system; and
in accordance with the determination that the first person is a known person, forgoing requesting the user to classify the first person.

13. The system of claim 12 , wherein the one or more programs further include instructions for forgoing generating a timely notification in accordance with the determination that the first person is a known person.

14. The system of claim 12 , wherein the one or more programs further include instructions for generating a timely notification identifying the first person in accordance with the determination that the first person is a known person.

15. The system of claim 14 , wherein the one or more programs further include instructions for:
determining a location of the first person;
determining whether the first person has previously been authorized by the user to access the location; and
wherein generating the timely notification identifying the first person comprises generating the timely notification identifying the first person and the location in accordance with a determination that the first person has not previously been authorized to access the location.

16. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing system, cause the system to:
obtain a live video stream;
detect one or more persons in the live video stream;
determine, from analysis of the live video stream, personally identifiable information that identifies a first person of the detected one or more persons;
determine, based on at least some of the personally identifiable information, whether the first person is a known person to the computing system;
in accordance with a determination that the first person is not a known person:
store at least some of the personally identifiable information; and
request a user to classify the first person; and

in accordance with (i) a determination that a predetermined amount of time has elapsed since the request was transmitted and a response was not received from the user, or (ii) a determination that a response was received from the user classifying the first person as a stranger, delete the stored personally identifiable information.

17. The non-transitory computer-readable storage medium of claim 16 , wherein the one or more programs further comprise instructions, which when executed by the computing system, cause the system to:
detect the first person by performing facial detection on a first frame in the live video stream; and
track the first person t

Assignees

• Google Llc

Inventors

• Bapat Akshay R
• Heitz Iii George Alban
• Chaudhry Rizwan Ahmed
• Dalal Navneet
• Stewart James Edward
• Bush Jennifer
• Venters Joe Delone
• Gates Kara
• Butler Timothy
• Kifle Yohannes
• Bastov Anton

Classifications

• H04N7/18 Primary

  Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast · CPC title

• G06V40/179

  metadata assisted face recognition · CPC title

• G06V40/173

  face re-identification, e.g. recognising unknown faces across different face tracks · CPC title

• G06V20/53

  Recognition of crowd images, e.g. recognition of crowd congestion · CPC title

• G06F18/23

  Clustering techniques · CPC title