Systems and methods for simultaneous localization and mapping using asynchronous multi-view cameras

Patent metadata

Field	Value
Publication number	US-12124269-B2
Application number	US-202117515923-A
Country	US
Kind code	B2
Filing date	Nov 1, 2021
Priority date	Oct 30, 2020
Publication date	Oct 22, 2024
Grant date	Oct 22, 2024

Abstract

Systems and methods for the simultaneous localization and mapping of autonomous vehicle systems are provided. A method includes receiving a plurality of input image frames from the plurality of asynchronous image devices triggered at different times to capture the plurality of input image frames. The method includes identifying reference image frame(s) corresponding to a respective input image frame by matching the field of view of the respective input image frame to the fields of view of the reference image frame(s). The method includes determining association(s) between the respective input image frame and three-dimensional map point(s) based on a comparison of the respective input image frame to the one or more reference image frames. The method includes generating an estimated pose for the autonomous vehicle the one or more three-dimensional map points. The method includes updating a continuous-time motion model of the autonomous vehicle based on the estimated pose.

First claim

What is claimed is:

1. A computer-implemented method, the method comprising:
(a) receiving a plurality of input image frames from a plurality of asynchronous image devices of an autonomous vehicle, wherein at least a subset of the plurality of input image frames have a plurality of different capture times;
(b) identifying one or more reference image frames that at least partially overlap a field of view of a respective input image frame of the plurality of input image frames, wherein the one or more reference image frames are associated with one or more reference times preceding the plurality of different capture times;
(c) determining one or more associations between the respective input image frame and one or more three-dimensional map points of a plurality of three-dimensional map points representative of an environment of the autonomous vehicle based on a comparison of the respective input image frame to the one or more reference image frames;
(d) generating an estimated pose for the autonomous vehicle relative to the environment at a respective time of the plurality of different capture times based on the one or more associations between the respective input image frame and the one or more three-dimensional map points; and
(e) updating a trajectory of the autonomous vehicle through the environment based on the estimated pose for the autonomous vehicle, wherein the trajectory of the autonomous vehicle is represented as a continuous-time motion model;
wherein (i) the plurality of three-dimensional map points and (ii) the continuous-time motion model are previously generated based on the one or more reference image frames.

2. The computer-implemented method of claim 1 , wherein (c) comprises:
processing the respective input image frame with a point extractor model to generate one or more two-dimensional points of interest for the respective input image frame;
identifying one or more matching two-dimensional points of interest for at least one of the one or more reference image frames that correspond to the one or more two-dimensional points of interest for the respective input image frame; and
determining the one or more associations between the respective input image frame and the one or more three-dimensional map points based on the one or more matching two-dimensional points of interest for the at least one of the one or more reference image frames.

3. The computer-implemented method of claim 1 , wherein the plurality of input image frames correspond to an input multi-frame of a plurality of multi-frames received by the autonomous vehicle during an operational time period, wherein the plurality of multi-frames comprise one or more key multi-frames, and wherein (i) the plurality of three-dimensional map points and (ii) the continuous-time motion model are previously generated based on the one or more key multi-frames.

4. The computer-implemented method of claim 3 , wherein the one or more key multi-frames are associated with one or more key times preceding the plurality of different capture times, and wherein the one or more reference image frames correspond to a recent key multi-frame associated with a recent key time of the one or more key times that is closest in time to the plurality of different capture times.

5. The computer-implemented method of claim 3 , wherein (e) further comprises:
selecting the input multi-frame as a new key multi-frame based on: (i) the estimated pose for the autonomous vehicle relative to the environment at the respective input time or (ii) a quantity of the one or more three-dimensional map points; and
in response to selecting the input multi-frame as the new key multi-frame, updating the trajectory of the autonomous vehicle through the environment based on the input multi-frame.

6. The computer-implemented method of claim 5 , wherein (e) further comprises:
in response to selecting the input multi-frame as the new key multi-frame, refining a position of at least one of the plurality of three-dimensional map points based on the input multi-frame.

7. The computer-implemented method of claim 5 , wherein the continuous-time motion model comprises a cumulative cubic basis spline parameterized by a set of control poses, and wherein (e) comprises:
updating the set of control poses based on the input multi-frame and one or more previous estimated poses corresponding to the one or more key multi-frames.

8. The computer-implemented method of claim 7 , wherein updating the set of control poses based on the input multi-frame and the one or more previous estimated poses corresponding to the one or more key multi-frames comprises:
refining at least one of the one or more previous poses corresponding to the one or more key multi-frames based on the input multi-frame.

9. The computer-implemented method of claim 7 , wherein (e) further comprises:
in response to selecting the input multi-frame as the new key multi-frame, generating one or more new three-dimensional map points based on the input multi-frame and the updated set of control poses.

10. The computer-implemented method of claim 9 , wherein the one or more new three-dimensional map points are generated based on a comparison between (i) at least two input images frames of the plurality of input image frames or (ii) at least one input image frame and at least one key image frame of at least one of the one or more key multi-frames.

11. The computer-implemented method of claim 1 , wherein (d) comprises:
generating the estimated pose for the autonomous vehicle relative to the environment at the respective time of the plurality of different capture times based on a continuous-time linear motion model and a previously estimated pose corresponding to the one or more reference image frames.

12. The computer-implemented method of claim 1 , further comprising:
(f) determining that the environment of the autonomous vehicle corresponds to a revisited environment, the revisited environment indicative of a respective environment that has been previously visited by the autonomous vehicle; and
(g) in response to determining that the environment of the autonomous vehicle corresponds to the revisited environment:
(i) adjusting the trajectory of the autonomous vehicle based on one or more previously estimated vehicle poses associated with the revisited environment, and
(ii) updating the plurality of map points representative of the environment of the autonomous vehicle based on the adjusted trajectory of the autonomous vehicle.

13. The computer-implemented method of claim 12 , wherein (f) comprises:
determining a travel distance for the autonomous vehicle;
determining a similarity score for the plurality of input image frames and the one or more reference image frames;
determining a geometry score for the plurality of input image frames based on a relative pose associated with the plurality of input image frames and the one or more previously estimated poses associated with the revisited environment; and
determining that the environment of the autonomous vehicle corresponds to the revisited environment based on the travel distance, the similarity score, and the geometry score.

14. The computer-implemented method of claim 1 , wherein the plurality of asynchronous image devices of the autonomous vehicle are associated with a plurality of different fields of view.

15. An autonomous vehicle control system comprising:
one or more processors; and
one or more computer-readable medium storing instructions that when executed by the one or more processors cause the autonomous vehicle control system to perform operations, the operations com

Assignees

• Aurora Operations Inc

Inventors

• Yang Anqi Joyce
• Cui Can
• Bârsan Ioan Andrei
• Wang Shenlong
• Urtasun Raquel

Classifications

• G05D1/646

  Following a predefined trajectory, e.g. a line marked on the floor or a flight path · CPC title

• G05D1/2435

  Extracting 3D information · CPC title

• G05D1/247

  using signals provided by artificial sources external to the vehicle, e.g. navigation beacons · CPC title

• B60R1/27

  providing all-round vision, e.g. using omnidirectional cameras · CPC title

• B60R2300/304

  using merged images, e.g. merging camera image with stored images · CPC title