Systems for estimating three-dimensional trajectories of physical objects

Patent metadata

Field	Value
Publication number	US-11635507-B2
Application number	US-202117190711-A
Country	US
Kind code	B2
Filing date	Mar 3, 2021
Priority date	Mar 3, 2021
Publication date	Apr 25, 2023
Grant date	Apr 25, 2023

Abstract

In implementations of systems for estimating three-dimensional trajectories of physical objects, a computing device implements a three-dimensional trajectory system to receive radar data describing millimeter wavelength radio waves directed within a physical environment using beamforming and reflected from physical objects in the physical environment. The three-dimensional trajectory system generates a cloud of three-dimensional points based on the radar, each of the three-dimensional points corresponds to a reflected millimeter wavelength radio wave within a sliding temporal window. The three-dimensional points are grouped into at least one group based on Euclidean distances between the three-dimensional points within the cloud. The three-dimensional trajectory system generates an indication of a three-dimensional trajectory of a physical object corresponding to the at least one group using a Kalman filter to track a position and a velocity a centroid of the at least one group in three-dimensions.

First claim

What is claimed is:

1. A method comprising:
transmitting, by a processing device, millimeter wavelength radio waves directed within a physical environment using beamforming that reflect from physical objects in the physical environment;
receiving, by the processing device, radar data describing the millimeter wavelength radio waves;
generating, by the processing device, a cloud of three-dimensional points based on the radar data, each of the three-dimensional points corresponding to a reflected millimeter wavelength radio wave within a sliding temporal window and removing three-dimensional points corresponding to reflected millimeter wavelength radio waves that have arrival times greater than a threshold arrival time;
determining, by the processing device, a position and a velocity for each of the three-dimensional points in the cloud of three-dimensional points;
removing, by the processing device, additional three-dimensional points having velocities below a threshold velocity from the three-dimensional points within the cloud;
grouping, by the processing device, the three-dimensional points into at least one group based on Euclidean distances between the three-dimensional points within the cloud; and
generating, by the processing device for display in a user interface of a display device, an indication of a three-dimensional trajectory of a physical object corresponding to the at least one group, displayed relative to a current position of the physical object, using a Kalman filter to track a position and a velocity of a centroid of the at least one group in three-dimensions.

2. The method as described in claim 1 , wherein the Kalman filter is a six state Kalman filter.

3. The method as described in claim 1 , wherein the physical object is a person.

4. The method as described in claim 3 , further comprising generating an indication for display in the user interface that the person is within a threshold distance of an additional person within the physical environment.

5. The method as described in claim 3 , wherein the indication of the three-dimensional trajectory is generated without identifying the person or capturing data usable to identify the person.

6. The method as described in claim 1 , wherein the indication of the three-dimensional trajectory is generated without using a digital camera.

7. The method as described in claim 1 , wherein the sliding temporal window corresponds to period of time that is less than 300 milliseconds.

8. The method as described in claim 1 , wherein the indication of the three-dimensional trajectory is displayed in the user interface as a virtual object relative to the physical object in digital video frames depicting the physical environment.

9. The method as described in claim 1 , wherein the grouping includes performing density-based spatial clustering of applications with noise.

10. A system comprising:
a memory component; and
a processing device coupled to the memory component, the processing device to perform operations comprising:
transmitting millimeter wavelength radio waves directed within a physical environment using beamforming that reflect from physical objects in the physical environment
receiving radar data describing the millimeter wavelength radio waves; and
generating a cloud of three-dimensional points based on the radar data, each of the three-dimensional points corresponding to a reflected millimeter wavelength radio wave within a sliding temporal window and removing three-dimensional points corresponding to reflected millimeter wavelength radio waves that have arrival times greater than a threshold arrival time;
determining a position and a velocity for each of the three-dimensional points in the cloud of three-dimensional points;
removing additional three-dimensional points having velocities below a threshold velocity from the three-dimensional points within the cloud;
grouping the three-dimensional points into at least one group based on Euclidean distances between the three-dimensional points within the cloud; and
generating, for display in a user interface of a display device, an indication of a three-dimensional trajectory of a physical object corresponding to the at least one group, displayed relative to a current position of the physical object, using a Kalman filter to track a position and a velocity of a centroid of the at least one group in three-dimensions.

11. The system as described in claim 10 , wherein the physical object is a person.

12. The system as described in claim 11 , wherein the indication of the three-dimensional trajectory is generated without identifying the person or capturing data usable to identify the person.

13. The system as described in claim 11 , wherein operations further comprise generating an indication for display in the user interface that the person is within a threshold distance of an additional person within the physical environment.

14. The system as described in claim 10 , wherein the indication of the three-dimensional trajectory is displayed in the user interface as a virtual object relative to the physical object in digital video frames depicting the physical environment.

15. The system as described in claim 10 , wherein the operations further comprise accessing path data describing path objects corresponding to detections of the physical objects in the physical environment and associate the physical object with a particular path object based on a distance between the centroid and a center of the particular path object.

16. A non-transitory computer-readable storage medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:
transmitting millimeter wavelength radio waves directed within a physical environment using beamforming that reflect from physical objects in the physical environment
receiving radar data describing the millimeter wavelength radio waves;
generating a cloud of three-dimensional points based on the radar data, each of the three-dimensional points corresponding to a reflected millimeter wavelength radio wave within a sliding temporal window and removing three-dimensional points corresponding to reflected millimeter wavelength radio waves that have arrival times greater than a threshold arrival time;
determining a position and a velocity for each of the three-dimensional points in the cloud of three-dimensional points;
removing additional three-dimensional points having velocities below a threshold velocity from the three-dimensional points within the cloud;
grouping the three-dimensional points into at least one group based on Euclidean distances between the three-dimensional points within the cloud, the Euclidean distances computed in an x-plane, a y-plane, and a z-plane; and
generating, for display in a user interface of a display device, an indication of a three-dimensional trajectory of a physical object corresponding to the at least one group, displayed relative to a current position of the physical object, using a Kalman filter to track a velocity and a position of a centroid of the at least one group in the x-plane, the y-plane, and the z-plane.

17. The non-transitory computer-readable storage medium as described in claim 16 , wherein the operations further comprise associating the physical object with a particular path object of a plurality of path objects based on a distance between the centroid and a center of the particular path object.

18. The non-transitory computer-readable storage medium as described in claim 16 , wh

Assignees

• Adobe Inc

Inventors

• Healey Jennifer Anne
• Wang Haoliang
• Zhang Ding

Classifications

• G01S7/20

  Stereoscopic displays; Three-dimensional [3D] displays; Pseudo-3D displays · CPC title

• G01S7/352

  Receivers · CPC title

• G01S13/726 Primary

  Multiple target tracking · CPC title

• G01S13/42

  Simultaneous measurement of distance and other co-ordinates (indirect measurement G01S13/46) · CPC title

• G01S7/356

  involving particularities of FFT processing · CPC title