Method of calibrating a direction of a pan, tilt, zoom, camera with respect to a fixed camera, and a system in which such a calibration is carried out

Patent metadata

Field	Value
Publication number	US-10430972-B2
Application number	US-201816110535-A
Country	US
Kind code	B2
Filing date	Aug 23, 2018
Priority date	Aug 29, 2017
Publication date	Oct 1, 2019
Grant date	Oct 1, 2019

Abstract

A method of calibrating a pan, tilt, zoom (PTZ) camera with a fixed camera utilizing an overview image of a scene captured by the fixed camera, and an image of the scene captured by the PTZ camera when directed in a first direction. By matching features in the overview image and the PTZ camera image, a first calibration is carried out by correlating the first direction to matching features in the overview image. A mapping between the PTZ camera image and the overview image is defined based on the matching features. The mapping is used to map an object from the PTZ camera image to the overview image. Based on an appearance of the mapped object, a quality of the mapping is calculated. If the quality is not good enough, the PTZ camera is redirected to a second direction, and a further calibration is carried out by again.

First claim

What is claimed is:

1. A method of calibrating a direction of a pan, tilt, zoom, PTZ, camera with respect to a first, fixed, camera, comprising:
receiving an overview image of a scene captured by a first, fixed, camera,
directing a PTZ camera in a first direction,
when the PTZ camera is in the first direction, performing the steps of:
a) receiving an image of the scene captured by the PTZ camera, wherein a field of view of the image captured by the PTZ camera partly overlaps a field of view the overview image,
b) identifying a first set of features in the image of the scene captured by the PTZ camera,
c) localizing at least a portion of the first set of features in the overview image so as to associate the at least a portion of first set of features in the image captured by the PTZ camera with a second set of features in the overview image,
d) logging positional data of the second set of features in the overview image,
e) defining a mapping between the image captured by the PTZ camera and the overview image based on the at least a portion of first set of features and the second set of features, and calculating a quality of the mapping,
performing a first calibration of the PTZ camera by correlating the first direction of the PTZ camera with the positional data of the second set of features being logged when the PTZ camera is directed in the first direction,
in case the quality of the mapping is below a first threshold:
redirecting the PTZ camera to a second direction,
performing steps a)-d) when the PTZ camera is in the second direction, and
performing a second calibration of the PTZ camera by correlating the second direction of the PTZ camera with positional data of the second set of features being logged when the PTZ camera is directed in the second direction,
wherein step e) further comprises mapping an object in the image captured by the PTZ camera to the overview image by using the defined mapping, wherein calculating the quality of the mapping includes calculating a similarity between an appearance of the object in the image captured by the PTZ camera and an appearance of the object after mapping to the overview image.

2. The method of claim 1 , further comprising performing step e) when the PTZ camera is in the second direction, wherein the step of performing a second calibration of the PTZ camera is made on a condition that the quality of the mapping calculated in step e) when the PTZ camera is in the second direction is greater than or equal to the first threshold.

3. The method of claim 1 , further comprising:
keep redirecting the PTZ camera to further directions, and repeating steps a)-e) until the quality of the mapping calculated in step e) is greater than or equal to the first threshold.

4. The method of claim 1 , further comprising:
identifying an area in the overview image where a density of features in the overview image exceeds a second threshold, and
selecting the second direction on basis of the first calibration of the PTZ camera such that an image captured by the PTZ camera when directed in the second direction covers the identified area in the overview image.

5. The method of claim 1 , further comprising:
selecting the second direction on basis of the first calibration of the PTZ camera such that an image captured by the PTZ camera when directed in the second direction covers a center of the overview image.

6. The method of claim 1 , wherein, prior to calculating the similarity, the appearance of the object in the image captured by the PTZ camera is adjusted on basis of properties of a lens system of the first camera.

7. The method of claim 1 , wherein the first threshold depends on the positional data of the second set of features, such that the first threshold decreases with a distance from a center of the overview image.

8. The method of claim 1 , wherein the appearance of an object is at least one of a size of the object and a geometrical shape of the object.

9. The method of claim 1 , wherein the object in the image captured by the PTZ camera corresponds to a periphery of the image captured by the PTZ camera.

10. The method of claim 1 , wherein a feature includes at least one of an edge or a corner in a captured image of the scene.

11. The method of claim 1 , further comprising
capturing a further overview image of the scene using a second, fixed, camera having a known position and direction in relation to the first camera, and
calibrating the PTZ camera with respect to the further overview image of the scene captured by the second camera on basis of the first or second calibration of the first camera and the known position and direction of the second camera in relation to the first camera.

12. A system, comprising:
a first, fixed, camera arranged to capture an overview image of a scene,
a pan, tilt, zoom, PTZ, camera which is separate from the first, fixed camera, and
a controller operatively connected to the first camera and the PTZ camera, the controller being configured to direct PTZ camera in a first direction, and, when the PTZ camera is in the first direction, to perform the steps of:
a) controlling the PTZ camera to capture an image of the scene, wherein a field of view of the image captured by the PTZ camera partly overlaps a field of view the overview image,
b) identifying a first set of features in the image of the scene captured by the PTZ camera,
c) localizing at least a subset of the first set of features in the overview image of the scene so as to associate the at least a subset of the first set of features in the image captured by the PTZ camera with a second set of features in the overview image,
d) logging positional data of the second set of features in the overview image,
e) defining a mapping between the image captured by the PTZ camera and the overview image based on the at least a subset of the first set of features and the second set of features, and calculating a quality of the mapping,
the controller further being configured to:
perform a first calibration of the PTZ camera by correlating the first direction of the PTZ camera with the positional data of the second set of features being logged when the PTZ camera is directed in the first direction,
in case the quality of the mapping is below a first threshold:
redirect the PTZ camera to a second direction,
perform steps a)-d) when the PTZ camera is in the second direction, and
perform a second calibration of the PTZ camera by correlating the second direction of the PTZ camera with positional data of the second set of features being logged when the PTZ camera is directed in the second direction,
wherein step e) further comprises mapping an object in the image captured by the PTZ camera to the overview image by using the defined mapping, wherein calculating the quality of the mapping includes calculating a similarity between an appearance of the object in the image captured by the PTZ camera and an appearance of the object after mapping to the overview image.

13. The system of claim 12 , further comprising:
at least one further fixed camera, wherein the first camera and the at least one further fixed camera are directed in different directions so as to capture overview images covering different portions of the scene,
wherein the PTZ camera is mounted in relation to the first camera and the at least one further fixed camera such that the PTZ camera is directable to capture images which overlap an overview image captured by the first camera, and to capture images which overlap an overview image captured by the at least one further fixed camera.

14. A non-transitory computer-readable medium h

Assignees

• Axis Ab

Inventors

• Dahlström Robin
• Gylin Mattias

Classifications

• G08B13/1963 Primary

  Arrangements allowing camera rotation to change view, e.g. pivoting camera, pan-tilt and zoom [PTZ] · CPC title

• G06T7/80 Primary

  Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration · CPC title

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• H04N23/695

  Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title