System and method for simultaneously multiple sensor calibration and transformation matrix computation

Patent metadata

Field	Value
Publication number	US-11635313-B2
Application number	US-202016847972-A
Country	US
Kind code	B2
Filing date	Apr 14, 2020
Priority date	Apr 14, 2020
Publication date	Apr 25, 2023
Grant date	Apr 25, 2023

Abstract

The present teaching relates to apparatus, method, medium, and implementations for simultaneously calibrating multiple sensors of different types. Multiple sensors of different types are first activated to initiate simultaneous calibration thereof based on a 3D construct including a plurality of fiducial marks. Sensors of different types including visual and depth based sensors operate in their respective coordinate systems. Each of the sensors is calibrated by acquiring sensor information of the 3D construct, detecting a feature point on each of the plurality of fiducial markers based on the sensor information, estimating a set of 3D coordinates, with respect to its coordinate system, corresponding to the detected feature points, based on which calibration parameters are generated. Sets of 3D coordinates derived in different coordinate systems are then used to compute at least one transformation matrix for corresponding at least one pair of the plurality of sensors.

First claim

I claim:

1. A computer-implemented method, comprising:
calibrating simultaneously a plurality of sensors including a first sensor, a second sensor, and a third sensor, the plurality of sensors having a plurality of sensor types including at least a visual-based sensor type and a depth-based sensor type, by:
for each of the first sensor, the second sensor, and the third sensor,
acquiring sensor information representative of a 3D construct comprising a plurality of fiducial markers,
detecting a feature point from a plurality of feature points on each fiducial marker of the plurality of fiducial markers based on the sensor information, and
estimating a set of 3D coordinates, each 3D coordinate from the set of 3D coordinates corresponding to a feature point from the plurality of feature points, wherein the set of 3D coordinates is with respect to a coordinate system associated with that sensor;

computing, based on the set of 3D coordinates for each of the first sensor, the second sensor, and the third sensor in the respective coordinate systems, a respective transformation matrix between each possible pair of the first sensor, the second sensor, and the third sensor; and
sending a signal to a vehicle to facilitate driving of the vehicle based on first data received from one of the first sensor, the second sensor, and the third sensor and second data determined based on the first data and at least one of the transformation matrices.

2. The method of claim 1 , wherein:
each fiducial marker from the plurality of fiducial markers further includes a plurality of planar facets spatially connected to form that fiducial marker,
each planar facet from the plurality of planar facets of that fiducial marker includes a different visual property; and
at least some of the plurality of planar facets form a protruded vertex on that fiducial marker.

3. The method of claim 1 , wherein:
the plurality of fiducial markers are distributed in space within a field of view of the first sensor, a field of view of the second sensor, and a field of view of the third sensor;
the plurality of fiducial markers are grouped together to form the 3D construct; and
the plurality of feature points on the plurality of fiducial markers are facing the same direction.

4. The method of claim 2 , wherein, for each fiducial marker from the plurality of fiducial markers, the detecting the feature point from the plurality of feature points comprises:
detecting different visual properties associated with the plurality of planar facets based on visual information acquired by the first sensor;
identifying a location where the at least some of the plurality of planar facets with different visual properties meet; and
estimating the feature point of the fiducial marker based on the identified location.

5. The method of claim 2 , wherein, for each fiducial marker from the plurality of fiducial markers, the detecting the feature point from the plurality of feature points comprises:
analyzing depth information associated with the plurality of planar facets of the fiducial marker acquired by the second sensor;
identifying a location based on the depth information; and
estimating, based on the location, the feature point of the fiducial marker.

6. A non-transitory computer-readable storage medium including instructions that, when executed by at least one processor of a computing system, cause the computing system to perform operations comprising:
calibrating simultaneously a plurality of sensors including a first sensor, a second sensor, and a third sensor, the plurality of sensors having a plurality of sensor types including at least a visual-based sensor type and a depth-based sensor type, by:
for each of the first sensor, the second sensor, and the third sensor,
acquiring sensor information representative of a 3D construct comprising a plurality of fiducial markers,
detecting a feature point from a plurality of feature points on each fiducial marker of the plurality of fiducial markers based on the sensor information, and
estimating a set of 3D coordinates, each 3D coordinate from the set of 3D coordinates corresponding to a feature point from the plurality of feature points, wherein the set of 3D coordinates is with respect to a coordinate system associated with that sensor;

computing, based on the set of 3D coordinates for each of the first sensor, the second sensor, and the third sensor in the respective coordinate systems a respective transformation matrix between each possible pair of the first sensor, the second sensor, and the third sensor; and
sending a signal to a vehicle to facilitate driving of the vehicle based on first data received from one of the first sensor, the second sensor, and the third sensor and second data determined based on the first data and at least one of the transformation matrices.

7. The non-transitory computer-readable storage medium of claim 6 , wherein:
each fiducial marker from the plurality of fiducial markers further includes a plurality of planar facets spatially connected to form that fiducial marker,
each planar facet from the plurality of planar facets of that fiducial marker includes a different visual property; and
at least some of the plurality of planar facets form a protruded vertex on that fiducial marker.

8. The non-transitory computer-readable storage medium of claim 6 , wherein:
the plurality of fiducial markers are distributed in space within a field of view of the first sensor, a field of view of the second sensor, and a field of view of the third sensor;
the plurality of fiducial markers are grouped together to form the 3D construct; and
the plurality of feature points on the plurality of fiducial markers are facing the same direction.

9. The non-transitory computer-readable storage medium of claim 7 , wherein:
for each fiducial marker from the plurality of fiducial markers, the detecting the feature point from the plurality of feature points comprises:
detecting different visual properties associated with the plurality of planar facets based on visual information acquired by the first sensor;
identifying a location where the at least some of the plurality of planar facets with different visual properties meet; and
estimating the feature point of the fiducial marker based on the identified location.

10. The non-transitory computer-readable storage medium of claim 7 , wherein, for each fiducial marker from the plurality of fiducial markers, the detecting the feature point from the plurality of feature points comprises:
analyzing depth information associated with the plurality of planar facets of the fiducial marker acquired by the second sensor;
identifying a location based on the depth information; and
estimating, based on the location, the feature point of the fiducial marker.

11. A system, comprising:
at least one processor;
a memory operably coupled to the at least one processor and storing instructions to cause the system to perform operations comprising:
calibrating simultaneously a plurality of sensors including a first sensor, a second sensor, and a third sensor, the plurality of sensors having a plurality of sensor types including at least a visual-based sensor type and a depth-based sensor type, by:
for each of the first sensor, the second sensor, and the third sensor,
acquiring sensor information representative of a 3D construct comprising a plurality of fiducial markers,
detecting a feature point from a plurality of feature points on each fiducial marker of the plurality of fiducial markers based on the sensor information, and
estimating a set of 3D coordinates, each 3D coordinate from the set of 3D coordinates correspond

Assignees

• Plusai Inc

Inventors

• Chang Huan Yang

Classifications

• G06T2207/30204

  Marker · CPC title

• G01S7/497

  Means for monitoring or calibrating · CPC title

• G01S17/42

  Simultaneous measurement of distance and other co-ordinates (indirect measurement G01S17/46) · CPC title

• G06T2207/10012

  Stereo images · CPC title

• G01S13/867

  Combination of radar systems with cameras · CPC title