System and method for calibrating moving cameras capturing broadcast video

Patent metadata

Field	Value
Publication number	US-12299900-B2
Application number	US-202418425629-A
Country	US
Kind code	B2
Filing date	Jan 29, 2024
Priority date	Feb 28, 2019
Publication date	May 13, 2025
Grant date	May 13, 2025

Abstract

A system and method of calibrating moving cameras capturing a sporting event is disclosed herein. A computing system retrieves a broadcast video feed for a sporting event. The broadcast video feed includes a plurality of video frames. The computing system labels, via a neural network, components of a playing surface captured in each video frame. The computing system matches a subset of labeled video frames to a set of templates with various camera perspectives. The computing system fits a playing surface model to the set of labeled video frames that were matched to the set of templates. The computing system identifies camera motion in each video frame using an optical flow model. The computing system generates a homography matrix for each video frame based on the fitted playing surface model and camera motion. The computing system calibrates each camera based on the homography matrix generated for each video frame.

First claim

What is claimed:

1. A computer-implemented method for tracking players, the computer-implemented method comprising:
receiving, by one or more processors, a plurality of trackable frames for a sporting match, wherein the plurality of trackable frames include body pose information and camera calibration data;
generating, by the one or more processors, one or more sets of tracklets based on the plurality of trackable frames;
predicting, by the one or more processors, a motion of an agent in each of the one or more sets of tracklets based on a motion field of a playing surface of the sporting match; and
outputting, by the one or more processors, a graphical representation of the predicted motion of the agent.

2. The computer-implemented method of claim 1 , the computer-implemented method further comprising:
connecting, by the one or more processors, at least one gap between each of the one or more sets of tracklets, wherein connecting the at least one gap includes augmenting one or more affinity measures to include a motion field estimation.

3. The computer-implemented method of claim 2 , wherein the motion field estimation corresponds to a change of an agent direction that occurs over the plurality of trackable frames.

4. The computer-implemented method of claim 1 , wherein the one or more sets of tracklets are derived from the body pose information.

5. The computer-implemented method of claim 1 , wherein predicting the motion of the agent in each of the one or more sets of tracklets includes generating one or more tracks for each agent on the playing surface.

6. The computer-implemented method of claim 5 , wherein outputting the graphical representation of the predicted motion of the agent includes generating one or more graphical representations corresponding to the one or more tracks for each agent on the playing surface.

7. The computer-implemented method of claim 1 , wherein predicting the motion of the agent includes utilizing a neural network to predict one or more player trajectories based on a ground truth player trajectory.

8. A non-transitory computer readable medium comprising one or more sequences of instructions, which, when executed by one or more processors, causes a computing system to perform operations, comprising:
receiving, by the computing system, a plurality of trackable frames for a sporting match, wherein the plurality of trackable frames include body pose information and camera calibration data;
generating, by the computing system, one or more sets of tracklets based on the plurality of trackable frames;
predicting, by the computing system, a motion of an agent in each of the one or more sets of tracklets based on a motion field of a playing surface of the sporting match; and
outputting, by the computing system, a graphical representation of the predicted motion of the agent.

9. The non-transitory computer readable medium of claim 8 , the operations further comprising:
connecting, by the computing system, at least one gap between each of the one or more sets of tracklets, wherein connecting the at least one gap includes augmenting one or more affinity measures to include a motion field estimation.

10. The non-transitory computer readable medium of claim 9 , wherein the motion field estimation corresponds to a change of an agent direction that occurs over the plurality of trackable frames.

11. The non-transitory computer readable medium of claim 8 , wherein the one or more sets of tracklets are derived from the body pose information.

12. The non-transitory computer readable medium of claim 8 , wherein predicting the motion of the agent in each of the one or more sets of tracklets includes generating one or more tracks for each agent on the playing surface.

13. The non-transitory computer readable medium of claim 12 , wherein outputting the graphical representation of the predicted motion of the agent includes generating one or more graphical representations corresponding to the one or more tracks for each agent on the playing surface.

14. A system comprising:
a processor; and
a memory comprising one or more sequences of instructions, which, when executed by the processor, causes the system to perform operations comprising:
receiving a plurality of trackable frames for a sporting match, wherein the plurality of trackable frames include body pose information and camera calibration data;
generating one or more sets of tracklets based on the plurality of trackable frames;
predicting a motion of an agent in each of the one or more sets of tracklets based on a motion field of a playing surface of the sporting match; and
outputting a graphical representation of the predicted motion of the agent.

15. The system of claim 14 , wherein predicting the motion of the agent in each of the one or more sets of tracklets includes generating one or more tracks for each agent on the playing surface.

16. The system of claim 15 , wherein outputting the graphical representation of the predicted motion of the agent includes generating one or more graphical representations corresponding to the one or more tracks for each agent on the playing surface.

17. The system of claim 14 , wherein predicting the motion of the agent includes utilizing a neural network to predict one or more player trajectories based on a ground truth player trajectory.

Assignees

• Stats Llc

Inventors

• Sha Long
• Ganguly Sujoy
• Wei Xinyu
• Lucey Patrick Joseph
• Cherukumudi Aditya

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title

• G06V10/82

  using neural networks · CPC title