System and method for calibrating moving cameras capturing broadcast video

Patent metadata

Field	Value
Publication number	US-11935247-B2
Application number	US-202318175278-A
Country	US
Kind code	B2
Filing date	Feb 27, 2023
Priority date	Feb 28, 2019
Publication date	Mar 19, 2024
Grant date	Mar 19, 2024

Abstract

A system and method of calibrating moving cameras capturing a sporting event is disclosed herein. A computing system retrieves a broadcast video feed for a sporting event. The broadcast video feed includes a plurality of video frames. The computing system labels, via a neural network, components of a playing surface captured in each video frame. The computing system matches a subset of labeled video frames to a set of templates with various camera perspectives. The computing system fits a playing surface model to the set of labeled video frames that were matched to the set of templates. The computing system identifies camera motion in each video frame using an optical flow model. The computing system generates a homography matrix for each video frame based on the fitted playing surface model and camera motion. The computing system calibrates each camera based on the homography matrix generated for each video frame.

First claim

What is claimed:

1. A method of calibrating a moving camera capturing a sporting event, comprising:
identifying, by a computing system, a live broadcast video feed for a sporting event, the live broadcast video feed comprising a plurality of video frames captured by a camera located in a sporting venue;
receiving, by the computing system, pose data for one or more players;
removing, by the computing system, the one or more players from the plurality of video frames based on the pose data to generate a plurality of empty video frames;
analyzing, by the computing system, the plurality of empty video frames to identify a flow field between successive video frames of the plurality of empty video frames;
generating, by the computing system, a homography matrix for each empty video frame based on the flow field between the successive video frames; and
calibrating, by the computing system over a network, the camera based on the homography matrix generated for each empty video frame of the plurality of empty video frames by applying each homography matrix to a corresponding video frame of the plurality of video frames.

2. The method of claim 1 , wherein removing, by the computing system, the players from the plurality of video frames to generate the plurality of empty video frames comprises:
identifying one or more pixels corresponding to at least one of the one or more players in a video frame; and
removing the one or more pixels from the video frame.

3. The method of claim 1 , wherein the homography matrix indicates a transform that is used to project the players' locations from the plurality of video frames to real world coordinates on a playing surface.

4. The method of claim 1 , wherein analyzing, by the computing system, the plurality of empty video frames to identify the flow field between the successive video frames of the plurality of empty video frames comprises:
identifying an object in a first video frame; and
tracking motion of the object in at least one second video frame succeeding the first video frame.

5. The method of claim 1 , wherein identifying, by the computing system, the live broadcast video feed for the sporting event comprises:
identifying a video frame of the plurality of video frames captured by a second camera remotely located in the sporting venue.

6. The method of claim 5 , wherein calibrating, by the computing system over the network, the camera based on the homography matrix generated for each empty video frame of the plurality of empty video frames comprises:
calibrating the second camera based on the homography matrix generated for an empty video frame corresponding to the video frame.

7. A non-transitory computer readable medium comprising one or more sequences of instructions, which, when executed by one or more processors, causes a computing system to perform operations, comprising:
identifying, by the computing system, a live broadcast video feed for a sporting event, the live broadcast video feed comprising a plurality of video frames captured by a camera located in a sporting venue;
receiving, by the computing system, pose data for one or more players;
removing, by the computing system, the one or more players from the plurality of video frames based on the pose data to generate a plurality of empty video frames;
analyzing, by the computing system, the plurality of empty video frames to identify a flow field between successive video frames of the plurality of empty video frames;
generating, by the computing system, a homography matrix for each empty video frame based on the flow field between the successive video frames; and
calibrating, by the computing system over a network, the camera based on the homography matrix generated for each empty video frame of the plurality of empty video frames by applying each homography matrix to a corresponding video frame of the plurality of video frames.

8. The non-transitory computer readable medium of claim 7 , wherein removing, by the computing system, the players from the plurality of video frames to generate the plurality of empty video frames comprises:
identifying one or more pixels corresponding to at least one of the one or more players in a video frame; and
removing the one or more pixels from the video frame.

9. The non-transitory computer readable medium of claim 7 , wherein the homography matrix indicates a transform that is used to project the players' locations from the plurality of video frames to real world coordinates on a playing surface.

10. The non-transitory computer readable medium of claim 7 , wherein analyzing, by the computing system, the plurality of empty video frames to identify the flow field between the successive video frames of the plurality of empty video frames comprises:
identifying an object in a first video frame; and
tracking motion of the object in at least one second video frame succeeding the first video frame.

11. The non-transitory computer readable medium of claim 7 , wherein identifying, by the computing system, the live broadcast video feed for the sporting event comprises:
identifying a video frame of the plurality of video frames captured by a second camera remotely located in the sporting venue.

12. The non-transitory computer readable medium of claim 11 , wherein calibrating, by the computing system over the network, the camera based on the homography matrix generated for each empty video frame of the plurality of empty video frames comprises:
calibrating the second camera based on the homography matrix generated for an empty video frame corresponding to the video frame.

13. A system comprising:
a processor; and
a memory comprising one or more sequences of instructions, which, when executed by the processor, causes the system to perform operations comprising:
identifying a live broadcast video feed for a sporting event, the live broadcast video feed comprising a plurality of video frames captured by a camera located in a sporting venue;
receiving pose data for one or more players;
removing the one or more players from the plurality of video frames based on the pose data to generate a plurality of empty video frames;
analyzing the plurality of empty video frames to identify a flow field between successive video frames of the plurality of empty video frames;
generating a homography matrix for each empty video frame based on the flow field between the successive video frames; and
calibrating, over a network, the camera based on the homography matrix generated for each empty video frame of the plurality of empty video frames by applying each homography matrix to a corresponding video frame of the plurality of video frames.

14. The system of claim 13 , wherein removing the players from the plurality of video frames to generate the plurality of empty video frames comprises:
identifying one or more pixels corresponding to at least one of the one or more players in a video frame; and
removing the one or more pixels from the video frame.

15. The system of claim 13 , wherein the homography matrix indicates a transform that is used to project the players' locations from the plurality of video frames to real world coordinates on a playing surface.

16. The system of claim 13 , wherein analyzing the plurality of empty video frames to identify the flow field between the successive video frames of the plurality of empty video frames comprises:
identifying an object in a first video frame; and
tracking motion of the object in at least one second video frame succeeding the first video frame.

17. The system of claim 13 , w

Assignees

• Stats Llc

Inventors

• Sha Long
• Ganguly Sujoy
• Wei Xinyu
• Lucey Patrick Joseph
• Cherukumudi Aditya

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06V10/761

  Proximity, similarity or dissimilarity measures · CPC title

• H04N21/44008

  involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream (arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59) · CPC title