Machine-vision person tracking in service environment

Patent metadata

Field	Value
Publication number	US-12374152-B2
Application number	US-202117454786-A
Country	US
Kind code	B2
Filing date	Nov 12, 2021
Priority date	Nov 12, 2021
Publication date	Jul 29, 2025
Grant date	Jul 29, 2025

Abstract

A method to predict a traversal-time interval for traversal of a service queue comprises receiving video of a region including the service queue, recognizing in the video, via machine vision, a plurality of persons awaiting service within the region, estimating an average crossing-time interval between successive crossings, by the plurality of persons, of a fixed boundary along the service queue, wherein such estimating is based on features of the service queue and of the one or more persons awaiting service, and returning an estimate of the traversal-time interval based on a count of the persons awaiting service and on the average crossing-time interval as estimated.

First claim

The invention claimed is:

1. A computer-implemented method to predict a traversal-time interval for traversal of a service queue, the method comprising:
receiving video of a region including the service queue in a hardware interface of a computer system, the video comprising a time-sequential series of digital images each comprising a matrix of pixel elements, wherein each pixel element comprises at least one associated brightness or depth value;
recognizing in the video, via a machine-vision engine of the computer system, a plurality of persons awaiting service within the region, the machine-vision engine being configured to update a digital model of the region including plural data structures held in computer memory that evolve over time as new video is received, the machine-vision engine including a classification machine configured to divide the digital model into plural segments, identify those segments that correspond to persons above a pre-determined confidence level, and label each person segment;
estimating, via a prediction engine of the computer system, based on features of the service queue and of the plurality of persons awaiting service, an average crossing-time interval between successive crossings of two or more fixed boundaries within the service queue by the plurality of persons, each of the two or more fixed boundaries having first and second sides where a person waiting in the service queue is recognizable by the machine vision system, wherein the average crossing-time interval is an interval between successive crossings averaged over the two or more fixed boundaries, and wherein the prediction engine is configured to enact a binary classification scheme distinguishing a person awaiting service from a person not awaiting service; and
returning, via the prediction engine of the computer system, an estimate of the traversal-time interval based on a count of the plurality of persons awaiting service and on the average crossing-time interval as estimated.

2. The method of claim 1 wherein returning the estimate of the traversal-time interval includes multiplying the count by the average crossing-time interval as estimated.

3. The method of claim 1 further comprising receiving graphical user input defining the service queue in at least one frame of the video.

4. The method of claim 1 wherein recognizing the plurality of persons awaiting service includes using machine vision to recognize a superset of candidate persons within the region and filtering the superset of candidate persons by application of a binary classifier.

5. The method of claim 4 wherein filtering the superset of candidate persons includes filtering based on proximity of each of the candidate persons to the service queue.

6. The method of claim 4 wherein filtering the superset of candidate persons includes filtering based on orientation and/or posture of each of the candidate persons relative to a flow direction of the service queue.

7. The method of claim 4 wherein filtering the superset of candidate persons includes filtering based on velocity of each of the candidate persons.

8. The method of claim 4 wherein filtering the superset of candidate persons includes filtering based on direction of movement of each of the candidate persons relative to a predetermined local flow direction of the service queue.

9. The method of claim 1 wherein estimating the average crossing-time interval includes defining one or more fixed boundaries perpendicular to a tangent of the service queue.

10. The method of claim 9 further comprising assessing confidence for recognizing the plurality of persons awaiting service at different locations within the region, wherein defining the two or more fixed boundaries includes arranging the two or more fixed boundaries along the service queue based on the confidence as assessed.

11. The method of claim 1 wherein the video is received from a plurality of video cameras arranged above the region and having different fields-of-view, the method further comprising co-registering video from each of the plurality of video cameras.

12. The method of claim 1 wherein the service queue is a first service queue, wherein the video of the region also includes a second service queue, and wherein the method is also applied to predicting a traversal-time interval for traversal of the second service queue.

13. A computer system comprising:
a hardware interface configured to receive video of a region;
a machine-vision engine configured to update a model of the region in computer memory based on the video;
a detection engine configured to:
define a series of candidate boundaries within the region;
for each candidate boundary of the series, assess a confidence of recognizing a person on a first side of the candidate boundary in a first frame of the video and on a second, opposite side of the candidate boundary in a second, subsequent frame of the video, wherein the person is among a plurality of persons awaiting service in a service queue within the region;
identify the candidate boundary for which the confidence is highest; and

signal advance of the person across the region pursuant to recognizing, above a threshold confidence, that the person is on the first side of the identified candidate boundary in the first frame of the video and on the second side of the identified candidate boundary in the second frame of the video; and
a prediction engine configured to:
furnish a count of the plurality of persons awaiting service within the region based on recognizing the plurality of persons awaiting service;
estimate an average crossing-time interval between successive crossings, by the plurality of persons, of a fixed boundary along the service queue, wherein such estimating is based on features of the service queue and of the plurality of persons awaiting service; and
return a traversal-time interval based on the count of the plurality of persons awaiting service and on the average crossing-time interval as estimated.

14. A computer-implemented method to predict a traversal-time interval for traversal of a service queue, the method comprising:
receiving video of a region including the service queue in a hardware interface of a computer system, the video comprising a time-sequential series of digital images each comprising a matrix of pixel elements, wherein each pixel element comprises at least one associated brightness or depth value;
recognizing in the video, via a machine-vision engine of the computer system, a plurality of persons awaiting service within the region, the machine-vision engine being configured to update a digital model of the region including plural data structures held in computer memory that evolve over time as new video is received, the machine-vision engine including a classification machine configured to divide the digital model into plural segments, identify those segments that correspond to persons above a pre-determined confidence level, and label each person segment;
estimating, via a prediction engine of the computer system, based on features of the service queue and of the plurality of persons awaiting service, an average crossing-time interval between successive crossings of a fixed boundary within the service queue by the plurality of persons, the fixed boundary having first and second sides where a person waiting in the service queue is recognizable by the machine vision system, the prediction engine being configured to enact a binary classification scheme distinguishing a person awaiting service from a person not awaiting service; and
returning, via the prediction engine of the computer system, an estimate of

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Li Chenyang
• Deng Hongli
• Blanco Saldana Gabriel
• Filcik Joseph Milan
• Menezes Ryan Savio

Classifications

• G06V40/103 Primary

  Static body considered as a whole, e.g. static pedestrian or occupant recognition · CPC title

• G06V20/10 Primary

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title