Normalized metadata generation device, object occlusion detection device and method

Patent metadata

Field	Value
Publication number	US-10885667-B2
Application number	US-201716337790-A
Country	US
Kind code	B2
Filing date	Mar 7, 2017
Priority date	Sep 28, 2016
Publication date	Jan 5, 2021
Grant date	Jan 5, 2021

Abstract

Disclosed a normalized metadata generation device, and object occlusion detection device and method. A normalized metadata generation method includes generating a multi-ellipsoid based three-dimensional human model using perspective features of a plurality of two-dimensional images obtained by the multiple cameras, performing scene calibration based on the three-dimensional human model to normalize object information of the object included in the two-dimensional images, and generating normalized metadata of the object from the two-dimensional images on which the scene calibration is performed.

First claim

The invention claimed is:

1. A normalized metadata generation method, the method is performed by a metadata generation device of a multi-camera-based video surveillance system including different kinds of cameras, the method comprising:
generating a multi-ellipsoid based three-dimensional (3D) human model using perspective features of a plurality of two-dimensional (2D) images obtained by the multiple cameras;
performing scene calibration based on the three-dimensional human model to normalize object information of an object included in the two-dimensional images; and
generating normalized metadata of the object from the two-dimensional images on which the scene calibration is performed,
wherein the generating of the normalized metadata of the object includes:
compensating colors of the two-dimensional (2D) image;
extracting representative color information;
extracting non-color metadata; and
integrating the extracted metadata into one data model.

2. The normalized metadata generation method of claim 1 , wherein the generating of the three-dimensional human model generates a human model having a height from a position of a foot, using three ellipsoids including a head, a body, and a leg in 3D world coordinates.

3. The normalized metadata generation method of claim 2 , wherein the ellipsoid is back-projected onto a two-dimensional space to match an actual object to perform shape matching.

4. The normalized metadata generation method of claim 3 , wherein a moving object region is detected by background modeling using a Gaussian mixture model (GMM) and a detected shape is normalized, to perform the shape matching.

5. The normalized metadata generation method of claim 4 , wherein the normalized shape is calculated as a set of boundary points and each of the boundary points is generated at a position where a radial line from a center of gravity meets an outermost boundary of the object.

6. The normalized metadata generation method of claim 5 , wherein the performing of the scene calibration includes
extracting valid data for line segments from the foot to the head;
estimating homology from the foot to the head using the extracted valid data; and
detecting a vanishing line and a vanishing point from the homology.

7. The normalized metadata generation method of claim 6 , wherein the valid data is selected according to a first condition that the line segment from the foot to the head is within a restricted region with respect to a y-axis and a second condition that the line segment from the foot to the head is a major axis of an ellipse to be approximated to a human object.

8. The normalized metadata generation method of claim 7 , wherein an angle, a major axis and a minor axis of the object are calculated through matching operation between the object and the ellipsoid to acquire the valid data.

9. The normalized metadata generation method of claim 6 , wherein invalid data is removed from the extracted valid data using robust random sample consensus (RANSAC) to prevent error due to the homology from the foot to the head.

10. The normalized metadata generation method of claim 6 , wherein the vanishing line and the vanishing points are determined by three human positions which are not on the same line.

11. The normalized metadata generation method of claim 10 , wherein the vanishing points are points under the ground plane, at which line segments from the foot to the head respectively representing positions of humans standing at various positions on the ground plane extend and meet each other.

12. The normalized metadata generation method of claim 10 , wherein the vanishing line is a line segment connecting a first point and a second point,
wherein the first point is a point at which a straight line connecting head points of a first human position and a second human position and a straight line connecting foot points of the first human position and the second human meet each other, and
wherein the second point is a point at which a straight line connecting head points of the first human position and a third human position and a straight line connecting foot points of the first human position and the third human meet each other.

13. The normalized metadata generation method of claim 6 , wherein the homology from the foot to the head is determined by calculating a projection matrix of a camera using the vanishing line, the vanishing points, and an object height.

14. The normalized metadata generation method of claim 13 , wherein the projection matrix projects an object on the two-dimensional image onto the three dimensional world coordinates which are not affected by camera parameters.

15. The normalized metadata generation method of claim 6 , wherein internal parameters and external parameters of the camera are estimated using the detected vanishing line and the vanishing points, and
wherein the internal parameters include a focal length, a principal point and an aspect ratio, and the external parameters include a panning angle, a tilting angle, a rolling angle, a camera height with respect to the z-axis, transformation in x-axis and y-axis directions.

16. The normalized metadata generation method of claim 1 , wherein the compensating of the colors includes
estimating a color of a light source by calculating a modified Minkowsky norm based color considering local correlation using the following equation; and

(

∫

(

f
σ

⁡

(
x
)

)

p

⁢
dx

∫
dx

)

1
/
p

=
ke

where f(x) represents an image defined as an image x=[x y] T , f σ =f*G σ , filtered by the Gaussian filter G σ and the Minkowsky norm p,
compensating the estimated color of the light source using the following equation[H]:
f corr c =f c /ω c 3 , for c∈{R,G,B}

where f corr c represents a color-corrected c-channel image, f c represents a c-channel image and ω c represents a scaling parameter.

17. The normalized metadata generation method of claim 1 , wherein the extracting of the representative color information includes
extracting the representative color information of the object by performing K-means clustering on the object region detected from the two-dimensional image of which the colors are compensated.

18. The normalized metadata generation method of claim 1 , wherein the non-color metadata includes size information including the height and width of the object, moving spe

Assignees

• Univ Chung Ang Ind Acad Coop Found

Inventors

• Paik Joon Ki
• Jung Jae Hoon
• Yoon In Hye
• Lee Seung Won

Classifications

• G06T7/80 Primary

  Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration · CPC title

• G06F16/5854 Primary

  using shape and object relationship · CPC title

• G06V10/763

  Non-hierarchical techniques, e.g. based on statistics of modelling distributions · CPC title

• G06F18/23213

  with fixed number of clusters, e.g. K-means clustering · CPC title

• G06V10/56

  relating to colour · CPC title