Automated or assisted umpiring of baseball game using computer vision

Patent metadata

Field	Value
Publication number	US-10632359-B2
Application number	US-201816124861-A
Country	US
Kind code	B2
Filing date	Sep 7, 2018
Priority date	May 17, 2016
Publication date	Apr 28, 2020
Grant date	Apr 28, 2020

Abstract

Methods and systems for use in automating or assisting umpiring of a baseball or softball game are described herein. A location of a strike zone is determined based on video images of a batter standing next to home plate captured by a camera. Locations of a ball traveling towards the batter, and locations of the bat being held by the batter, are autonomously tracked using computer vision based on video images captured by at least two cameras having different positions. Additionally, there are autonomous determinations of whether a location of the ball intersects with the strike zone, and whether the batter made a genuine attempt to swing the bat at the ball, and based one at least one of these determinations, there is an autonomous determination of whether a “strike” or a “ball” occurred. Additionally, an indication of whether a “strike” or a “ball” occurred is autonomously output.

First claim

The invention claimed is:

1. A method for automating or assisting umpiring of a baseball or softball game comprising:
receiving images of a batter, images of a bat held by the batter, and images of a ball traveling towards the batter from at least two cameras;
determining a location of a strike zone based on the images of the batter;
adding a strike zone graphic to the location of the strike zone on the images of the batter;
autonomously tracking locations of the ball traveling towards the batter based on the images of the ball traveling towards the batter using transformations associated with the at least two cameras to determine the locations of the ball traveling towards the batter in three-dimensional (3D) space;
adding an animated trail representing the locations of the ball on the images of the batter;
autonomously tracking locations of the bat based on the images of the bat using transformations associated with the at least two cameras to determine locations of the bat in 3D space;
autonomously determining whether a location of the bat in 3D space at a point in time is the same as a location of the ball in 3D space at the point in time;
autonomously determining whether at least one location of the ball traveling towards the batter intersects with the strike zone based on the determined location of the strike zone and the tracked locations of the ball traveling towards the batter;
autonomously determining whether the batter made a genuine attempt to swing the bat at the ball traveling towards the batter;
autonomously determining that the bat contacted the ball traveling towards the batter if there was an autonomous determination that the location of the bat in 3D space at the point in time was the same as the location of the ball traveling towards the batter in 3D space at the point in time;
autonomously determining that the bat did not contact the ball traveling towards the batter if there was an autonomous determination that the location of the bat in 3D space at the point in time was not the same as the location of the ball traveling towards the batter in 3D space at the point in time; and
autonomously determining whether a “strike” or a “ball” occurred;
displaying the strike zone graphic and the animated trail on an output display device; and
autonomously indicating whether a “strike” or a “ball” occurred on the output display device.

2. The method of claim 1 , further comprising:
receiving audio associated with a mitt of a catcher;
autonomously determining whether the bat contacted the ball traveling towards the batter based on the audio associated with the mitt of the catcher;
indicating whether the ball traveling towards the batter contacted the bat; and
using the indication to determine whether a foul tip occurred or the ball was caught in the mitt of the catcher.

3. The method of claim 1 , wherein the autonomously determining whether a “strike” or a “ball” occurred includes determining that a “strike” occurred if there was an autonomous determination that the bat did not contact the ball traveling towards the batter, and there was an autonomous determination that the batter made a genuine attempt to swing the bat at the ball traveling towards the batter; and
determining that a “ball” occurred if there was an autonomous determination that at least one location of the ball traveling towards the batter did not intersect with the strike zone and there was an autonomous determination that the batter did not make a genuine attempt to swing the bat at the ball traveling towards the batter.

4. The method of claim 1 , wherein the strike zone graphic comprises a 3D box representing a 3D strike zone, and wherein the 3D strike zone is displayed from different perspectives.

5. The method of claim 1 , further comprising:
determining a vertical plane in 3D space that is aligned with a front of a home plate;
autonomously determining whether the location of the bat in 3D space intersects with the vertical plane in 3D space that is aligned with the front of the home plate; and
wherein the autonomously determining whether the batter made a genuine attempt to swing the bat at the ball traveling towards the batter is based on the autonomous determination of whether the location of the bat in 3D space intersected with the vertical plane in 3D space that is aligned with the front of the home plate.

6. The method of claim 1 , further comprising:
autonomously determining a rotation angle of the bat;
comparing the rotation angle of the bat to a threshold angle; and
wherein the autonomously determining whether the batter made a genuine attempt to swing the bat at the ball traveling towards the batter is based on whether the rotation angle of the bat exceeded the threshold angle.

7. The method of claim 1 , further comprising autonomously:
determining a trajectory of the ball traveling towards the batter in 3D space;
determining whether at least one location of the bat in 3D space during a time frame is within a specified vicinity of at least one location of the ball traveling towards the batter in 3D space during the time frame;
determining whether the trajectory of the ball traveling towards the batter in 3D space changed beyond a specified threshold during the time frame;
determining whether the bat contacted the ball traveling towards the batter; and
indicating whether the ball traveling towards the batter contacted the bat.

8. The method of claim 1 , further comprising:
receiving audio associated with the batter;
autonomously determining whether the bat contacted the ball traveling towards the batter based on the audio associated with the batter; and
indicating whether the ball traveling towards the batter contacted the bat.

9. The method of claim 1 , further comprising:
receiving images from the at least two cameras of the ball traveling away from the batter if there was an autonomous determination that the bat contacted the ball traveling towards the batter;
autonomously determining whether the ball traveling away from the batter was hit fair or foul if there was an autonomous determination that the bat contacted the ball traveling towards the batter; and
indicating whether the ball traveling away from the batter was hit fair or foul.

10. A system for automating or assisting umpiring of a baseball or softball game comprising:
at least two cameras that capture images of a batter, images of a ball traveling towards the batter, and images of a bat being held by the batter;
at least one processor in communication with the at least two cameras and the output device configured to:
determine a location of a strike zone based on the images of the batter,
add a strike zone graphic at the location of the strike zone on the images of the batter,
autonomously track locations of the ball traveling towards the batter based on the images of the ball traveling towards the batter,
add an animated trail representing the locations of the ball on the images of the batter, autonomously track locations of the bat as the ball travels towards the batter based on the images of the bat,
use transformations associated with the at least two cameras to determine locations of the ball traveling towards the batter and locations of the bat in three-dimensional (3D) space,
autonomously determine whether at least one location of the ball traveling towards the batter intersects with the strike zone based on the determined location of the strike zone and the tracked locations of the ball traveling towards the batter,
autonomously determine whether the batter made a genuine attempt to swing the bat at the ball traveling towards the batter,
autonomously determine whether a location of the bat in 3D

Assignees

• Sportsmedia Tech Corp

Inventors

• Spivak Josh D
• Goldbeck Graham W
• Zander Ryan A
• Chen Ted C
• Amyx Robert C
• Painter James G

Classifications

• G06T2207/30241

  Trajectory · CPC title

• G06T7/246 Primary

  using feature-based methods, e.g. the tracking of corners or segments · CPC title

• G06T7/254

  involving subtraction of images · CPC title

• A63B71/0605 Primary

  Decision makers and devices using detection means facilitating arbitration · CPC title

• A63B2220/806

  Video cameras · CPC title