Systems and methods for synchronizing an image sensor

Patent metadata

Field	Value
Publication number	US-11989905-B2
Application number	US-202117241487-A
Country	US
Kind code	B2
Filing date	Apr 27, 2021
Priority date	Apr 27, 2021
Publication date	May 21, 2024
Grant date	May 21, 2024

Abstract

Systems and methods for synchronization are provided. In some aspects, a method for synchronizing an image sensor is provided. The method includes receiving image data captured using an image sensor that is moving along a pathway, and assembling an image sensor trajectory using the image data. The method also includes receiving position data acquired along the pathway using a position sensor, wherein timestamps for the image data and position data are asynchronous, and assembling a position sensor trajectory using the position data. The method further includes generating a spatial transformation that aligns the image sensor trajectory and position sensor trajectory, and synchronizing the image sensor based on the spatial transformation.

First claim

The invention claimed is:

1. A method for synchronizing an image sensor, the method comprising:
receiving image data captured using an image sensor that is moving along a pathway;
assembling an image sensor trajectory by applying a structure-from-motion algorithm to the image data to generate a plurality of positions forming the image sensor trajectory, wherein the plurality of positions indicates a plurality of estimated positions in which images were captured by the image sensor as the image sensor move along the pathway, and wherein the image sensor trajectory indicates a plurality of timestamps in which the images were captured by the image sensor;
receiving position data acquired along the pathway using a position sensor, wherein timestamps for the image data and position data are asynchronous;
assembling a position sensor trajectory using the position data, wherein the position sensor trajectory indicates a plurality of positions in which position measurements were captured by the position sensor as the position sensor move along the pathway and a plurality of timestamps in which the position measurements were captured by the position sensor, and wherein the position sensor trajectory is a Global Navigation Satellite System (GNSS) data acquired along the pathway;
generating a spatial transformation that aligns the image sensor trajectory and position sensor trajectory by applying a coherent point drift algorithm to the image sensor trajectory and position sensor trajectory, wherein the coherent point drift algorithm matches a shape, position, and scale of the image sensor trajectory to the position sensor trajectory; and
synchronizing the image sensor based on the spatial transformation.

2. The method of claim 1 , wherein the method further comprises acquiring the image data and the position data while the image sensor and position sensor are co-located.

3. The method of claim 1 , wherein the method further computing a timestamp offset between the image sensor and position sensor.

4. The method of claim 3 , wherein the method further comprises applying the timestamp offset to timestamps of image data acquired by the image sensor.

5. A system comprising:
a camera configured to capture image data, wherein the camera comprises a camera clock configured to generate timestamps corresponding to the image data;
a processor in communication with the camera;
at least one memory comprising instructions executable by the processor, the instructions causing the processor to:
receive image data captured using the camera while the camera is moving along a pathway;
assemble a camera trajectory by applying a structure-from-motion algorithm to the image data to generate a plurality of positions forming the camera trajectory, wherein the plurality of positions indicates a plurality of estimated positions in which images were captured by the camera as the camera move along the pathway, and wherein the camera trajectory indicates a plurality of timestamps in which the images were captured by the camera;
receive position data acquired along the pathway using a position sensor, wherein timestamps for the image data and position data are asynchronous;
assemble a position sensor trajectory using the position data, wherein the position sensor trajectory indicates a plurality of positions in which position measurements were captured by the position sensor as the position sensor move along the pathway and a plurality of timestamps in which the position measurements were captured by the position sensor, and wherein the position sensor trajectory is a Global Navigation Satellite System (GNSS) data acquired along the pathway;
generate a spatial transformation that aligns the camera trajectory and position sensor trajectory by applying a coherent point drift algorithm to the camera trajectory and position sensor trajectory, wherein the coherent point drift algorithm matches a shape, position, and scale of the camera trajectory to the position sensor trajectory; and
synchronize the camera based on the spatial transformation.

6. The system of claim 5 , wherein the system further comprises the position sensor that is co-located with the camera.

7. The system of claim 5 , wherein the instructions further cause the processor to compute a timestamp offset between the camera and position sensor.

8. The system of claim 7 , wherein the instructions further cause the processor to apply the timestamp offset to timestamps of image data acquired by the camera.

9. The system of claim 8 , wherein the instructions further cause the processor to synchronize the camera clock using the timestamp offset.

10. A non-transitory computer-readable storage medium for synchronizing a camera sensor, carrying one or more sequences of one or more instructions which, when executed by one or more processors, cause a system to perform steps to:
receive image data captured using an image sensor that is moving along a pathway;
generate an image sensor trajectory by applying a structure-from-motion algorithm to the image data to generate a plurality of positions forming the image sensor trajectory, wherein the plurality of positions indicates a plurality of estimated positions in which images were captured by the image sensor as the image sensor move along the pathway, and wherein the image sensor trajectory indicates a plurality of timestamps in which the images were captured by the image sensor;
receive position data acquired along the pathway using a position sensor, wherein timestamps for the image data and position data are asynchronous;
assemble a position sensor trajectory using the position data, wherein the position sensor trajectory indicates a plurality of positions in which position measurements were captured by the position sensor as the position sensor move along the pathway and a plurality of timestamps in which the position measurements were captured by the position sensor, and wherein the position sensor trajectory is a Global Navigation Satellite System (GNSS) data acquired along the pathway;
generate a spatial transformation that aligns the image sensor trajectory and position sensor trajectory by applying a coherent point drift algorithm to the image sensor trajectory and position sensor trajectory, wherein the coherent point drift algorithm matches a shape, position, and scale of the image sensor trajectory to the position sensor trajectory; and
synchronize the image sensor using the spatial transformation.

Assignees

• Here Global Bv

Inventors

• Huffman Landis
• Finken Joshua Michael
• Aroskar Amey
• Rasal Sumedh
• Boddhu Sanjay Kumar

Classifications

• G06T7/70 Primary

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G01S19/14

  specially adapted for specific applications · CPC title

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G06V30/1918

  Fusion techniques, i.e. combining data from various sources, e.g. sensor fusion · CPC title

• G06V30/2552

  Combination of methods, e.g. classifiers, working on different input data, e.g. sensor fusion · CPC title