Scalable position tracking system for tracking position in large spaces

Patent metadata

Field	Value
Publication number	US-10885642-B1
Application number	US-201916663633-A
Country	US
Kind code	B1
Filing date	Oct 25, 2019
Priority date	Oct 25, 2019
Publication date	Jan 5, 2021
Grant date	Jan 5, 2021

Abstract

A scalable tracking system includes a camera subsystem, a weight subsystem, and a central server. The camera subsystem includes cameras that capture video of a space, camera clients that determine local coordinates of people in the captured videos, and a camera server that determines the physical positions of people in the space based on the determined local coordinates. The weight subsystem determines when items were removed from shelves. The central server determines which person in the space removed the items based on the physical positions of the people in the space and the determination of when items were removed.

First claim

What is claimed is:

1. A system comprising:
an array of cameras positioned above a space, each camera of the array of cameras configured to capture a video of a portion of the space, the space containing a person;
a first camera client configured to:
receive a first plurality of frames of a first video from a first camera of the array of cameras, each frame of the first plurality of frames showing the person within the space;
for a first frame of the first plurality of frames:
determine a first bounding area around the person shown in the first frame; and
generate a first timestamp of when the first frame was received by the first camera client;

for a second frame of the first plurality of frames:
determine a second bounding area around the person shown in the second frame; and
generate a second timestamp of when the second frame was received by the first camera client;

a second camera client separate from the first camera client, the second camera client configured to:
receive a second plurality of frames of a second video from a second camera of the array of cameras, each frame of the second plurality of frames showing the person within the space;
for a third frame of the second plurality of frames:
determine a third bounding area around the person shown in the third frame; and
generate a third timestamp of when the third frame was received by the second camera client;

for a fourth frame of the second plurality of frames:
determine a fourth bounding area around the person shown in the fourth frame; and
generate a fourth timestamp of when the fourth frame was received by the second camera client;

a camera server separate from the first and second camera clients, the camera server configured to:
determine that the first timestamp falls within a first time window;
in response to determining that the first timestamp falls within the first time window, assign coordinates defining the first bounding area to the first time window;
determine that the second timestamp falls within the first time window;
in response to determining that the second timestamp falls within the first time window, assign coordinates defining the second bounding area to the first time window;
determine that the third timestamp falls within the first time window;
in response to determining that the third timestamp falls within the first time window, assign coordinates defining the third bounding area to the first time window;
determine that the fourth timestamp falls within a second time window that follows the first time window;
in response to determining that the fourth timestamp falls within the second time window, assign coordinates defining the fourth bounding area to the second time window;
determine that coordinates assigned to the first time window should be processed;
in response to determining that coordinates assigned to the first time window should be processed:
calculate, based at least on the coordinates defining the first bounding area and the coordinates defining the second bounding area, a combined coordinate for the person during the first time window for the first video from the first camera; and
calculate, based at least on the coordinates defining the third bounding area, a combined coordinate for the person during the first time window for the second video from the second camera; and

determine, based at least on the combined coordinate for the person during the first time window for the first video from the first camera and the combined coordinate for the person during the first time window for the second video from the second camera, a position of the person within the space during the first time window;

a plurality of weight sensors positioned within the space, each weight sensor of the plurality of weight sensors configured to produce a signal indicative of a weight experienced by that weight sensor;
a weight server separate from the first and second camera clients and the camera server, the weight server configured to determine, based at least on a signal produced by a first weight sensor of the plurality of weight sensors, that an item positioned above the first weight sensor was removed; and
a central server separate from the first and second camera clients, the camera server, and the weight server, the central server configured to determine, based at least on the position of the person within the space during the first time window, that the person removed the item, wherein, based at least on the determination that the first person removed the item, the person is charged for the item when the person exits the space.

2. The system of claim 1 , further comprising:
an array of light detection and ranging (LiDAR) sensors positioned above the space; and
a LiDAR server separate from the first and second camera clients, the camera server, the weight server, and the central server, the LiDAR server configured to determine a position of the person within the space during the first time window based at least on a coordinate received from a LiDAR sensor of the array of LiDAR sensors.

3. The system of claim 1 , wherein the determination that the first person removed the item is based at least on (1) a distance between the position of the person in the space and a position of the first weight sensor in the space and (2) a time of when the signal produced by the first weight sensor was received by the weight server falling within the first time window.

4. The system of claim 1 , wherein:
the first camera client is further configured to receive a height of the person shown in the first frame and a height of the person shown in the second frame;
the second camera client is further configured to receive a height of the person shown in the third frame and a height of the person shown in the fourth frame; and
the camera server is further configured to:
calculate, based at least on the height of the person shown in the first frame and the height of the person shown in the second frame, a combined height for the person during the first time window for the first video from the first camera; and
calculate, based at least on the height of the person shown in the third frame, a combined coordinate for the person during the first time window for the second video from the second camera.

5. The system of claim 1 , wherein:
the first camera client implements a first clock used to generate the first and second timestamps;
the second camera client implements a second clock used to generate the third and fourth timestamps; and
the camera server implements a third clock, the first, second, and third clocks are synchronized using a clock synchronization protocol.

6. The system of claim 5 , wherein the weight server implements a fourth clock that is synchronized with the first, second, and third clocks using the clock synchronization protocol.

7. The system of claim 1 , wherein the combined coordinate for the person during the first time window for the first video from the first camera comprises an average of the coordinates defining the first bounding area and the coordinates defining the second bounding area.

8. The system of claim 1 , wherein the array of cameras is arranged in a grid such that a camera that is communicatively coupled to the first camera client is not directly adjacent in the same row or the same column of the grid to another camera that is communicatively coupled to the first camera client.

9. The system of claim 1 , further comprising:
a rack within the space, the rack comprising a shelf and a base comprising a drawer, the base positioned vertically lower than the shelf, the shelf divided into a first region and a second region, the first weight sensor of the plurality of weight sensors

Assignees

• 7 Eleven Inc

Inventors

• Krishnamurthy Sailesh Bharathwaaj
• Vakacharla Sarath
• NGUYEN Trong Nghia
• Mirza Shahmeer Ali
• Chinnam Madan Mohan
• Boulio Caleb Austin
• Paul Deepanjan
• Mirza Fahad

Classifications

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• G06Q20/32

  using wireless devices · CPC title

• G06Q20/206

  comprising security or operator identification provisions, e.g. password entry · CPC title

• G06Q20/203

  Inventory monitoring · CPC title

• G07G1/14

  Systems including one or more distant stations co-operating with a central processing unit · CPC title