Method and system for detecting and analyzing objects

Patent metadata

Field	Value
Publication number	US-2022230443-A1
Application number	US-202117551621-A
Country	US
Kind code	A1
Filing date	Dec 15, 2021
Priority date	Jan 19, 2021
Publication date	Jul 21, 2022
Grant date	—

Abstract

A method for detecting objects and labeling the objects with distances in an image includes steps of: obtaining a thermal image from a thermal camera, an RGB image from an RGB camera, and radar information from an mmWave radar; adjusting the thermal image based on the RGB image to generate an adjusted thermal image, and generating a fused image based on the RGB image and the adjusted thermal image; generating a second fused image based on the fused image and the radar information; detecting objects in the images, and generating, based on the fused image, another fused image including bounding boxes marking the objects; and determining motion parameters of the objects.

First claim

What is claimed is:

1 . A method for detecting and analyzing objects that is to be performed by a system including a thermal camera, an RGB camera, a millimeter wave (mmWave) radar and an image processing device, the image processing device including an image fusion module, a coordinate transformation module, an object detection module and an image integration module, the method comprising steps of:
A) controlling the thermal camera, the RGB camera and the mmWave radar to operate synchronously, so that the thermal camera captures a thermal image of a scene at a time point and sends the thermal image to the image processing device, the RGB camera captures an RGB image of the scene at the same time point and sends the RGB image to the image processing device, and the mmWave radar generates radar information with respect to the scene at the same time point by scanning the scene and sends the radar information to the image processing device, wherein the radar information includes coordinates of a plurality of radar points related to multiple objects in the scene that have been detected by the mmWave radar; B) by the image fusion module of the image processing device, adjusting the thermal image based on the RGB image in order to generate an adjusted thermal image, and generating a first fused image by combining the RGB image and the adjusted thermal image; C) by the coordinate transformation module of the image processing device, generating a coordinate chart that includes a plurality of data points based on the coordinates of the radar points, wherein each of the data points corresponds to at least one of the radar points, and generating a second fused image based on the first fused image by projecting the data points of the coordinate chart onto the first fused image according to a radar-to-camera projection matrix, the second fused image including graphical points corresponding respectively to the data points; D) by the object detection module of the image processing device, detecting objects in the scene that are present in the RGB image and the adjusted thermal image by analyzing the RGB image and the adjusted thermal image, locating the objects thus detected in the first fused image, and generating a third fused image based on the first fused image by drawing, for each of the objects in the first fused image, a bounding box around the object to mark the object; and E) by the image integration module of the image processing device and for each of the objects in the scene that is marked in the third fused image, determining a set of motion parameters of the object that includes a distance from the mmWave radar to the object, wherein the distance is determined by:
comparing the second fused image and the third fused image to find at least one graphical point among the graphical points in the second fused image that is within a region defined by the bounding box that marks the object,
determining which one of the radar points that correspond to at least one data point corresponding to the at least one graphical point thus found is a nearest radar point to the mmWave radar based on coordinates of the radar points, and
calculating the distance from the mmWave radar to the object based on the coordinates of the nearest radar point.

2 . The method of claim 1 , wherein:
step D) is to, with respect to each of the objects in the scene that is present in the RGB image and the adjusted thermal image, detect the object by classifying the object into a category; and the method further comprises a step of:
F) by the image processing device, generating object information that indicates, for each of the objects in the scene that is marked in the third fused image, the category of the object and the set of motion parameters of the object.

3 . The method of claim 2 , wherein:
step E) includes determining, for each of the objects in the scene that is marked in the third fused image, the set of motion parameters that further includes a location of the object and a speed of the object, wherein the location and the speed are determined based on the coordinates of the radar points.

4 . The method of claim 1 , wherein step E) further includes:
generating a result image based on the third fused image, wherein the result image includes, with respect to each of the objects in the scene that is marked in the third fused image, a bounding box that marks the object and a label near the bounding box that indicates the distance from the mmWave radar to the object.

5 . The method of claim 1 , wherein:
the RGB camera uses a lens that is not a wide-angle lens; and step B) includes adjusting the RGB image by utilizing a matrix of intrinsic parameters for the RGB camera, and generating the first fused image by combining the RGB image thus adjusted and the adjusted thermal image.

6 . The method of claim 1 , wherein:
the RGB camera uses a wide-angle lens; and step B) includes adjusting the RGB image by utilizing a calibration matrix for the RGB camera that reduces distortion effect in the RGB image, and generating the first fused image by combining the RGB image thus adjusted with the adjusted thermal image.

7 . The method of claim 1 , further comprising steps of:
G) controlling the RGB camera to capture an RGB image that includes an image of a calibration plate and that serves as a calibration image, and controlling the mmWave radar to scan a place at which the calibration plate is located in order to generate calibration radar information, the calibration radar information including coordinates of multiple radar points related to the calibration plate; and H) generating another coordinate chart that includes multiple data points corresponding respectively to the multiple radar points related to the calibration plate based on the coordinates of the multiple radar points, and determining the radar-to-camera projection matrix by utilizing perspective transformation and based on the calibration image and on the another coordinate chart, the radar-to-camera projection matrix being a homography matrix that is for use in projecting the multiple data points onto the image of the calibration plate.

8 . The method of claim 7 , wherein:
the RGB camera uses a lens that is not a wide-angle lens; and step H) includes adjusting the calibration image based on a matrix of intrinsic parameters for the RGB camera; and step H) is to determine the radar-to-camera projection matrix based on the another coordinate chart and the calibration image that has been adjusted and that includes the image of the calibration plate which has been adjusted, the radar-to-camera projection matrix being for use in projecting the multiple data points in the another coordinate chart onto the image of the calibration plate that has been adjusted.

9 . The method of claim 7 , wherein:
the RGB camera uses a wide-angle lens; and step H) includes adjusting the calibration image by utilizing a calibration matrix for the RGB camera that reduces distortion effect in the calibration images; and step H) is to determine the radar-to-camera projection matrix based on the another coordinate chart and the calibration image that has been adjusted and that includes the image of the calibration plate which has been adjusted, the radar-to-camera projection matrix being for use in projecting the multiple data points in the another coordinate chart onto the image of the calibration plate that has been adjusted.

10 . A system for detecting and analyzing objects, the system comprising:
an image processing device including an image fusion module, a coordinate transformation module, an object detection module and an image integration module; a thermal camera that is in

Assignees

• Micromax Int Corp
• Univ Nat Taiwan Science & Technology

Inventors

• Hua Kai-Lung
• Chen Yung-Yao
• Jhong Sin-Ye
• Chen Yo-Cheng
• Lin Ba-Lin
• Lin Tzyy-Yzong
• Wen Cheng-Shu
• Wang Yen-Po
• Chen Chun-Jung
• Yang Tung-Hsin
• Lu Wen-Hsiang
• Huang Chyi-Jia

Classifications

• H04N23/55

  Optical parts specially adapted for electronic image sensors; Mounting thereof · CPC title

• G01S13/867

  Combination of radar systems with cameras · CPC title

• G01S13/931

  of land vehicles · CPC title

• G01S7/2955

  Means for determining the position of the radar coordinate system for evaluating the position data of the target in another coordinate system (G01S7/24 takes precedence; sighting devices adapted for indirect laying of fire F41G3/16; inertial navigation G01C21/16) · CPC title

• G06V10/80

  Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level (multimodal speaker identification or verification G10L17/10) · CPC title