Real-time monocular structure from motion

Patent metadata

Field	Value
Publication number	US-10706582-B2
Application number	US-201815965480-A
Country	US
Kind code	B2
Filing date	Apr 27, 2018
Priority date	Sep 17, 2012
Publication date	Jul 7, 2020
Grant date	Jul 7, 2020

Abstract

Systems and methods are described for multithreaded navigation assistance by acquired with a single camera on-board a vehicle, using 2D-3D correspondences for continuous pose estimation, and combining the pose estimation with 2D-2D epipolar search to replenish 3D points.

First claim

What is claimed is:

1. A system for autonomous vehicular navigation using a multithreaded monocular structure from motion (SFM) architecture, comprising:
a single camera positioned on-board a vehicle; and
a multithreaded processor coupled to the single camera and a non-transitory computer-readable storage medium, the multithreaded processor being included in the multithreaded SFM architecture, and being configured for:
acquiring images using the single camera positioned on-board the vehicle;
estimating camera motion using monocular SFM by performing continuous camera pose estimation of 2D-3D correspondences;
detecting a local planarity of a road and correcting for scale drift using the monocular SFM based on the camera pose estimation;
determining visual odometry correspondences and replenishing 3D points of the images by combining the pose estimation with a 2D-2D multi-threaded per-frame epipolar search, the per-frame epipolar search continuously generating, for each frame, redundantly validated 3D points persistent across comparatively long tracks, the determining visual odometry further comprising:
parallelizing the epipolar search across a plurality of threads;
validating the 3D points among intermediate points, performing a local bundle adjustment, and retaining the validated 3D points for insertion of new 3D points at a keyframe in a main thread to frontload the validated 3D points for the pose estimation;
performing, in real-time, the pose estimation at each of a plurality of frames using only the validated 3D points to reduce processing requirements and increase processing speed of the pose estimation;
executing a real-time global bundle adjustment in a thread-safe architecture in parallel with the real-time pose estimation;
determining an optimized planar homography mapping between a road in two frames based on one or more determined ground estimation cues, the cues including triangulated sparse 3D points, inter-frame dense stereo matching, and a displacement computed based on a distance from a vanishing point; and
estimating per-frame relative importances of the cues by computing observation covariances for each of the cues and performing ground plane estimation by combining the triangulated sparse 3D points and cues using a Kalman filter; and

autonomously controlling driving functions of the vehicle based on the determined visual odometry and the local planarity of the road.

2. The system of claim 1 , wherein the processor is further configured for providing fast 3D-2D correspondences using pose-guided matching.

3. The system of claim 1 , wherein the processor is further configured for performing epipolar constrained search to produce per-frame 2D-2D correspondences.

4. The system of claim 1 , wherein the processor is further configured for performing vanishing point detection to hypothesize a feature match search window along one or more radial lines from the VP for pruning mismatches due to repeated features.

5. The system of claim 1 , wherein the processor is further configured for validating each 3D point against all frames in real-time, refining cameras and 3D points by the performing the local bundle adjustment, and collecting and refining 3D points from an epipolar thread.

6. The system of claim 1 , wherein the processor is further configured for collecting and refinding to allow bundle adjustment using long tracks.

7. The system of claim 1 , wherein the processor is further configured for performing real-time scale correcting by combining scale estimates from 3D points and planar homography mappings.

8. The system of claim 1 , wherein the validating the 3D points provides increased accuracy and timing of the pose estimation based on the epipolar constrained search, triangulation and backprojection.

9. The system of claim 1 , wherein the data-driven covariance learning comprises adapting the observation covariance on a per-frame basis by combining the cues triangulated sparse 3D points and homography-guided dense inter-frame stereo cues using a Kalman filter, where relative importance of cues is estimated on a per-frame basis learned through observation covariances.

10. A method for vehicular navigation using a multithreaded monocular structure from motion (SFM) architecture, comprising:
autonomously navigating a vehicle using a multithreaded processor coupled to a non-transitory computer-readable storage medium, the multithreaded processor being included in the multithreaded SFM architecture, and being configured for:
acquiring images using a single camera positioned on-board a vehicle;
estimating camera motion using monocular SFM by performing continuous camera pose estimation of 2D-3D correspondences;
detecting a local planarity of a road and correcting for scale drift using the monocular SFM based on the camera pose estimation;
determining visual odometry correspondences and replenishing 3D points of the images by combining the pose estimation with a 2D-2D multi-threaded per-frame epipolar search, the per-frame epipolar search continuously generating, for each frame, redundantly validated 3D points persistent across comparatively long tracks, the determining visual odometry further comprising:
parallelizing the epipolar search across a plurality of threads;
validating the 3D points among intermediate points, performing a local bundle adjustment, and retaining the validated 3D points for insertion of new 3D points at a keyframe in a main thread to frontload the validated 3D points for the pose estimation;
performing, in real-time, the pose estimation at each of a plurality of frames using only the validated 3D points to reduce processing requirements and increase processing speed of the pose estimation;
executing a real-time global bundle adjustment in a thread-safe architecture in parallel with the real-time pose estimation;
determining an optimized planar homography mapping between a road in two frames based on one or more determined ground estimation cues, the cues including triangulated sparse 3D points, inter-frame dense stereo matching, and a displacement computed based on a distance from a vanishing point; and
estimating per-frame relative importances of the cues by computing observation covariances for each of the cues and performing ground plane estimation by combining the triangulated sparse 3D points and cues using a Kalman filter; and

autonomously controlling driving functions of the vehicle based on the determined visual odometry and the local planarity of the road.

11. The method of claim 10 , further comprising providing fast 3D-2D correspondences using pose-guided matching.

12. The method of claim 10 , further comprising performing epipolar constrained search to produce per-frame 2D-2D correspondences.

13. The method of claim 10 , further comprising performing vanishing point detection to hypothesize a feature match search window along one or more radial lines from the VP for pruning mismatches due to repeated features.

14. The method of claim 10 , further comprising validating each 3D point against all frames in real-time, refining cameras and 3D points by the performing the local bundle adjustment, and collecting and refining 3D points from an epipolar thread.

15. The method of claim 10 , further comprising collecting and refinding to allow bundle adjustment using long tracks.

16. The method of claim 10 , further comprising performing real-time scale correcting by combining scale estimates from 3D points and planar homography mappings.

17. The method of claim 10 , wherein the validating the 3D points provides inc

Assignees

• Nec Lab America Inc
• Nec Corp

Inventors

• Chandraker Manmohan
• Song Shiyu

Classifications

• G06T7/73 Primary

  using feature-based methods · CPC title

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G06V20/10

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title

• G06V20/588

  Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road · CPC title

• G06T2207/20164

  Salient point detection; Corner detection · CPC title