Extrinsic calibration of multi-camera system

Patent metadata

Field	Value
Publication number	US-2021295559-A1
Application number	US-202016846521-A
Country	US
Kind code	A1
Filing date	Apr 13, 2020
Priority date	Mar 19, 2020
Publication date	Sep 23, 2021
Grant date	—

Abstract

A method for extrinsic calibration of cameras in a monitoring system, performed by a calibration device, includes obtaining preliminary calibration data of the cameras, obtaining a preliminary pose for an object observed by the cameras, and determining calibration data that results in an optimization of an objective function using the preliminary calibration data and the preliminary pose as starting values for the optimization. The method further includes defining the objective function with calibration data and pose as unknown variables and including a first function yielding aggregated reprojection error, and a second function yielding pose quality based on relative orientation and relative distance between 3D points of the pose. The second function enables improved accuracy of the calibration data and may be configured to yield pose quality for individual poses or motion quality for a time sequence of poses.

First claim

1 . A method of calibrating cameras that are arranged to generate at least partly overlapping views of a scene, said method comprising:
obtaining preliminary calibration data of the cameras, said preliminary calibration data representing an arrangement of the cameras in relation to a common coordinate system; obtaining a preliminary pose for an object in the scene, the preliminary pose comprising point locations in the common coordinate system and being calculated based on feature points detected in the views; defining an objective function comprising calibration data and pose as unknown variables, the objective function comprising a first function that determines an aggregation of differences between the feature points and the pose as projected onto the at least partly overlapping views by use of the calibration data, and a second function that determines a quality of the pose as a function of relative orientation and relative distance between point locations included in the pose; optimizing the objective function as a function of the calibration data and the pose, with starting values for the calibration data and the pose given by the preliminary calibration data and the preliminary pose; and providing the calibration data that results in an optimization of the objective function.

2 . The method of claim 1 , wherein the second function is configured to determine the quality as a function of a difference between the pose and one or more valid object poses.

3 . The method of claim 2 , wherein the second function comprises a trained model which is configured to determine the quality in relation to an ensemble of valid object poses.

4 . The method of claim 1 , wherein the second function comprises a trained discriminator of a Generative Adverserial Network, GAN.

5 . The method of claim 1 , wherein the second function is configured to determine includes the pose, and a valid object motion.

6 . The method of claim 5 , wherein the second function is configured to operate a prediction function for valid object motion on at least a subset of the poses in the time sequence to estimate a valid motion pose for the pose, and determine the quality as a function of a difference between the pose and the valid motion pose.

7 . The method of claim 6 , wherein the valid motion pose is estimated to be concurrent with the pose in the time sequence of poses.

8 . The method of claim 1 , wherein said generating the starting values comprises: projecting the preliminary pose onto a kinematic chain space, KCS, to form a KCS matrix, and generating the starting values as a function of the KCS matrix.

9 . The method of claim 1 , wherein said optimizing the objective function comprises: evaluating, in a plurality of iterations, the objective function for current values of the pose and the calibration data in relation to an optimization criterion, wherein at least one of the iterations comprises: identifying a set of feature points with a major influence on the objective function, and modifying the set of feature points to reduce its influence on the objective function.

10 . The method of claim 1 , wherein the first function is configured to compute the differences as a function of distances between the feature points and projected locations for the pose in the respective view, wherein the projected locations are projections of the point locations onto the respective view by use of the calibration data, and wherein the first function is configured to operate a Huber loss function on the distances to compute the differences.

11 . The method of claim 1 , wherein the first function is configured to compute the differences as a function of distances between the feature points and projected locations for the pose in the respective view, wherein the projected locations are projections of the point locations onto the respective view by use of the calibration data, and wherein the first function is configured to apply weights to the differences to form weighted differences, and aggregate the weighted differences.

12 . The method of claim 11 , wherein the first function is configured to determine, for a respective camera, point distances between the respective camera and the point locations included in the pose, and calculate the weights as a function of the point distances.

13 . The method of claim 12 , wherein the weighted differences are calculated to favor larger point distances over shorter point distances.

14 . The method of claim 11 , wherein the first function is configured to calculate the weights as a function of confidence scores assigned to the feature points, the confidence scores representing detection certainty of a respective feature point.

15 . The method of claim 14 , wherein the weighted differences are calculated to favor higher detection certainty over lower detection certainty.

16 . The method of claim 1 , wherein the objective function further comprises a smooth motion function that is configured to aggregate fourth-order differential values for trajectories of the point locations, wherein each of the trajectories comprises a subset of point locations that are associated over time.

17 . The method of claim 1 , wherein said obtaining preliminary calibration data comprises: calculating a plurality of candidate calibration data for a plurality of different subsets of the feature points in the at least partly overlapping views, and determining the preliminary calibration data as a function of the plurality of candidate calibration data.

18 . The method of claim 1 , wherein said obtaining preliminary calibration data comprises:
selecting a group of views among the at least partly overlapping views; determining a collection of corresponding feature points in the group of views; and performing a number of repetitions of an estimation procedure for first and second groups of corresponding feature points in the collection, wherein the first and second groups differ within and between the repetitions, and wherein the estimation procedure comprises: determining candidate calibration data as a function of the first group; calculating a candidate group of point locations as a function of the second group and the candidate calibration data; and determining a match score representing a correspondence between the second group and the candidate group; the method further comprising: determining the preliminary calibration data based on the match score and the candidate calibration data determined by the repetitions of the estimation procedure.

19 . A non-transitory computer-readable medium comprising computer instructions which, when executed by a processing system, cause the processing system to perform the method of claim 1 .

20 . A device configured to calibrate cameras that are arranged to generate at least partly overlapping views of a scene, said device comprising logic configured to control the device to: obtain preliminary calibration data of the cameras, said preliminary calibration data representing an arrangement of the cameras in relation to a common coordinate system; obtain a preliminary pose for an object in the scene, the preliminary pose comprising point locations in the common coordinate system and being calculated based on feature points detected in at least partly overlapping views; define an objective function comprising calibration data and pose as unknown variables, the objective function comprising a first function representing an aggregation of differences between the feature points and the pose as proje

Assignees

• Sony Corp

Inventors

• Huang Sangxia
• Moliner Olivier

Classifications

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/30208

  Marker matrix · CPC title

• G06T7/80 Primary

  Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title