Method for estimating a 3D trajectory of a projectile from 2D camera images

Patent metadata

Field	Value
Publication number	US-10380409-B2
Application number	US-201715815696-A
Country	US
Kind code	B2
Filing date	Nov 16, 2017
Priority date	Nov 16, 2017
Publication date	Aug 13, 2019
Grant date	Aug 13, 2019

Abstract

A method that estimates the 3D trajectory of a projectile, such as a golf ball or baseball, by analyzing a sequence of images from a single-lens, 2D camera. Image analysis may be used to locate the projectile in the camera images, using motion detection and filters for the projectile's expected size and shape. A physics model may be used to calculate the projectile's trajectory as a function of its initial position and velocity (for example, just after impact with a golf club or bat). A camera projection transform may map this trajectory into predicted pixel locations, which may be compared to the observed projectile locations in the camera images. The projectile's trajectory may be estimated by finding initial conditions that minimize differences between observed and predicted pixel locations, using a nonlinear least squares solver for example. The method may be extended to multiple cameras.

First claim

What is claimed is:

1. A method for estimating a 3D trajectory of a projectile from 2D camera images, comprising:
obtaining a sequence of 2D camera images from a camera, wherein one or more 2D camera images of said sequence of 2D camera images comprise an image of a projectile;
identifying a pixel location of said projectile in each 2D camera image of said one or more 2D camera images;
calculating a camera projection transform that maps points in a world reference frame into pixel positions in said sequence of 2D camera images;
calculating a model that maps projectile initial conditions into a modeled pixel position for said projectile in each 2D camera image of said one or more 2D camera images, wherein said projectile initial conditions comprise
an initial position vector in said world reference frame for said projectile; and,
an initial velocity vector in said world reference frame for said projectile; and,

calculating estimated projectile initial conditions as projectile initial conditions that minimize differences between said modeled pixel position for said projectile and said pixel location of said projectile across said one or more 2D camera images.

2. The method of claim 1 , wherein said projectile comprises one or more of a golf ball, a baseball, a softball, a soccer ball, a football, a hockey puck, a tennis ball, a table tennis ball, a squash ball, a racket ball, a shuttlecock, a handball, a lacrosse ball, a field hockey ball, a volleyball, a kickball, a horseshoe, and a lawn dart.

3. The method of claim 1 , further comprising:
calculating an estimated projectile trajectory in said world reference frame based on said estimated projectile initial conditions and on a projectile physics model.

4. The method of claim 3 , wherein said projectile physics model comprises a model of forces on said projectile, wherein said forces comprise
gravity, and
drag.

5. The method of claim 4 , wherein
said forces further comprise a Magnus effect force; and,
said projectile physics model further comprises an estimate or calculation of an initial spin of said projectile.

6. The method of claim 3 , further comprising:
calculating an estimated carry distance for said projectile as a horizontal distance between said initial position vector of said estimated projectile initial conditions and a point on said estimated projectile trajectory where said projectile contacts a horizontal plane.

7. The method of claim 1 , wherein said calculating said camera projection transform comprises:
defining a camera reference frame fixed to said camera;
calculating a transformation between said world reference frame and said camera reference frame; and,
calculating a transformation between said camera reference frame and said pixel positions based on one or more camera parameters.

8. The method of claim 7 , further comprising:
measuring a size in pixels of a camera image of an object of a known size at a known distance to determine one or more of said one or more camera parameters.

9. The method of claim 7 , wherein said calculating a transformation between said world reference frame and said camera reference frame comprises:
measuring an orientation of said camera in said world reference frame using one or more sensors coupled to said camera; and,
calculating said transformation between said world reference frame and said camera reference frame from said orientation of said camera in said world reference frame.

10. The method of claim 9 , wherein
said one or more sensors coupled to said camera comprise a three-axis accelerometer;
said measuring said orientation of said camera in said world reference frame comprises obtaining a measured gravity vector using said three-axis accelerometer; and,
said transformation between said world reference frame and said camera reference frame comprises a rotation that rotates a vertical gravity vector in said world reference frame to said measured gravity vector.

11. The method of claim 1 , wherein said identifying a pixel location of said projectile in each 2D camera image comprises:
applying a three-frame difference algorithm to identify one or more moving objects in said one or more 2D camera images, yielding one or more difference frames;
applying a moving average filter to said one or more difference frames, wherein a size of said moving average filter matches a largest expected pixel size of said projectile, yielding one or more filtered images; and,
identifying isolated peaks in said one or more filtered images as potential pixel locations of said projectile, wherein each isolated peak of said isolated peaks has a value that exceeds a detection threshold and is separated from other potential pixel locations of said potential pixel locations by a distance that exceeds a distance threshold.

12. The method of claim 11 , wherein
said projectile is a ball; and,
said identifying a pixel location of said projectile in each 2D camera image further comprises:
filtering said potential pixel locations of said projectile to retain only pixel locations of objects that are substantially round.

13. The method of claim 12 , wherein said filtering said potential pixel locations of said projectile to retain only pixel locations of objects that are substantially round comprises:
for each potential pixel location of said potential pixel locations of said projectile,
identifying candidate pixels within a region surrounding said potential pixel location that have a pixel value exceeding a value threshold;
determining a centroid of said candidate pixels;
calculating an average distance between said candidate pixels and said centroid;
calculating a disc having a center at said centroid and a radius of 1.5 times said average distance;
determining the fraction of pixels within said disc that are candidate pixels; and,
retaining said potential pixel location when said fraction exceeds a density threshold.

14. The method of claim 1 , further comprising:
fixing one coordinate of said initial position vector in said world reference frame based on a measured or assumed distance between said camera and said initial position vector; and,
obtaining or calculating an initial time for motion of said projectile, wherein
said initial position vector corresponds to a position of said projectile at said initial time, and
said initial velocity vector corresponds to a velocity of said projectile at said initial time.

15. The method of claim 14 , wherein said calculating estimated projectile initial conditions comprises:
obtaining a nonlinear least-squares solution that minimizes the sum of squared distances between said modeled pixel position for said projectile and said pixel location of said projectile across said one or more 2D camera images, wherein parameters of said model comprise two unknown coordinates for said initial position vector and three unknown coordinates for said initial velocity vector.

16. The method of claim 15 , wherein said calculating said model comprises:
calculating or obtaining a time for each 2D camera image of said one or more 2D camera images;
mapping said time for each 2D camera image into a projectile location in said world reference frame based on
said projectile initial conditions, and
a physics model for an initial period of said 3D trajectory of said projectile;

mapping said projectile location in said world reference frame into said modeled pixel position based on said camera projection transform.

17. The method of claim 16 , wherein said physics model for said initial period of said 3D tr

Assignees

• Blast Motion Inc

Inventors

• Thornbrue James
• Cherveny Patrick
• Haas Juergen

Classifications

• G06T7/73

  using feature-based methods · CPC title

• G06T7/251

  involving models · CPC title

• G06T2207/10021

  Stereoscopic video; Stereoscopic image sequence · CPC title

• G06T2207/30224

  Ball; Puck · CPC title

• G06T2207/30241

  Trajectory · CPC title