System and method for position tracking using edge computing

Patent metadata

Field	Value
Publication number	US-11501455-B2
Application number	US-202117174544-A
Country	US
Kind code	B2
Filing date	Feb 12, 2021
Priority date	Oct 26, 2018
Publication date	Nov 15, 2022
Grant date	Nov 15, 2022

Abstract

A tracking system includes a camera subsystem that includes cameras that capture vide of a space. Each camera is coupled with a camera client that determines local coordinates of people in the captured video. The camera clients generate frames that include color frames and depth frames labeled with an identifier number of the camera and their corresponding timestamps. The camera clients generate tracks that include metadata describing historical people detections, tracking identifications, timestamps, and the identifier number of the camera. The camera clients send the frames and tracks to cluster servers that maintain the frames and tracks such that they are retrievable using their corresponding labels. A camera server queries the cluster servers to receive the frames and tracks using their corresponding labels. The camera server determines the physical positions of people in the space based on the determined local coordinates.

First claim

What is claimed is:

1. A system, comprising:
a first camera configured to generate a first plurality of frames showing a portion of a space, the space containing a person;
a first camera client operably coupled with the first camera, and configured to:
receive the first plurality of frames from the camera, the first plurality of frames comprising a first plurality of color frames and a first plurality of depth frames, wherein:
the first plurality of color frames corresponds to colors of objects in the space; and
the first plurality of depth frames corresponds to distances of objects in the space from the first camera;

generate a timestamp for each corresponding color and depth frame that is received by the first camera client;
send a first set of frames comprising the first plurality of color frames and the first plurality of depth frames labeled with one or more corresponding timestamps and an identifier number of the first camera to a first server from among a plurality of cluster servers;
for each depth frame from the first plurality of depth frames:
detect a contour associated with the person in the depth frame;
detect, based at least in part upon pixel coordinates of the contour, a bounding area around the person in the depth frame;
determine, based at least in part upon the bounding area, coordinates of the person in the depth frame;
associate, based at least in part upon the bounding area, a first tracking identification to the person, wherein the first tracking identification is linked to historical detections associated with the person, wherein the historical detections associated with the person comprise at least one of a contour, a bounding area, and a segmentation mask associated with the person; and
generate a track comprising the first tracking identification linked to the historical detections associated with the person, wherein the track is labeled with one or more of corresponding timestamps, the identifier number of the first camera, and the historical detections associated with the person;

send a first set of tracks comprising tracks generated from the first plurality of depth frames to a second server from among the plurality of cluster servers, wherein the second server is separate from the first server;

the first server operably coupled to the first camera and the first camera client, and configured to:
store the first set of frames;
receive a first request to receive a particular set of frames from among the first set of frames where the person is shown within the space from a first timestamp to a second timestamp;
in response to receiving the first request, query the first camera client to provide the particular set of frames; and
receive the particular set of frames from the first camera client; and

the second server operably coupled to the first camera and the first camera client, and configured to:
store the first set of tracks;
receive a second request to receive a particular set of tracks from among the first set of tracks that comprises tracking identifications associated with the person from a third timestamp to a fourth timestamp;
in response to receiving the second request, query the first camera client to provide the particular set of tracks; and
receive the particular set of tracks from the first camera client.

2. The system of claim 1 , wherein the first camera client is further configured to:
compress the first plurality of color frames; and
compress the first plurality of depth frames.

3. The system of claim 1 , further comprising:
a second camera configured to generate a second plurality of frames showing a portion of the space, the space containing the person;
a second camera client operably coupled with the second camera, and configured to:
receive the second plurality of frames from the second camera, the second plurality of frames comprising a second plurality of color frames and a second plurality of depth frames, wherein:
the second plurality of color frames corresponds to colors of objects in the space; and
the second plurality of depth frames corresponds to distances of objects in the space from the second camera;

generate a timestamp for each corresponding color and depth frame that is received by the second camera client;
send a second set of frames comprising the second plurality of color frames and the second plurality of depth frames labeled with one or more corresponding timestamps and an identifier number of the second camera to the first server;
for each depth frame from the second plurality of depth frames:
detect a contour associated with the person in the depth frame;
detect, based at least in part upon pixel coordinates of the contour, a bounding area around the person in the depth frame;
determine, based at least in part upon the bounding area, coordinates of the person in the depth frame;
associate, based at least in part upon the bounding area, a second tracking identification to the person, wherein the second tracking identification is linked to historical detections associated with the person; and
generate a track comprising the second tracking identification linked with the historical detections associated with the person, wherein the track is labeled with one or more of corresponding timestamps, the identifier number of the second camera, and the historical detections associated with the person;

send a second set of tracks comprising tracks generated from the second plurality of depth frames to the second server; and

wherein:
the first server is operably coupled with the second camera client and the second camera and further configured to store the second set of frames; and
the second server is operably coupled with the second camera client and the second camera and further configured to store the second set of tracks.

4. The system of claim 3 , further comprising:
a camera server operably coupled with the plurality of cluster servers, and configured to:
receive the first and second set of frames from the first server in response to sending a request to the first server to send frames;
receive the first and second set of tracks from the second server in response to sending a request to the second server to send tracks;
determine that a first timestamp of first depth and color frames from the first set of frames and tracks falls within a first time window;
in response to determining that the first timestamp falls within the first time window, assign first coordinates defining a first bounding area in a first track from the first set of tracks to the first time window;
determine that a second timestamp of second depth and color frames from the first set of frames and tracks falls within the first time window;
in response to determining that the second timestamp falls within the first time window, assign second coordinates defining a second bounding area in a second track from the first set of tracks to the first time window;
determine that a third timestamp of third depth and color frames from the second set of frames and tracks falls within the first time window;
in response to determining that the third timestamp falls within the first time window, assign third coordinates defining a third bounding area in a third track from the second set of tracks to the first time window;
determine that a fourth timestamp of fourth depth and color frames from the second set of frames and tracks falls within a second time window that follows the first time window;
in response to determining that the fourth timestamp falls within the second time window, assign fourth coordinates defining a fourth bounding area in a fourth track from the second set of tracks to the second time window;
determine that coordinates assigned to the first time window should be processed;
in response to determining that coordina

Assignees

• 7 Eleven Inc

Inventors

• Crain Jon Andrew
• Krishnamurthy Sailesh Bharathwaaj
• Dalal Kyle
• Mirza Shahmeer Ali

Classifications

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• G01S17/46

  Indirect determination of position data · CPC title

• G01S17/66 Primary

  Tracking systems using electromagnetic waves other than radio waves · CPC title

• G06T7/60

  Analysis of geometric attributes · CPC title

• G06T2207/30232

  Surveillance · CPC title