System for operating a movable device

Patent metadata

Field	Value
Publication number	US-12475596-B2
Application number	US-202318344027-A
Country	US
Kind code	B2
Filing date	Jun 29, 2023
Priority date	Jun 29, 2023
Publication date	Nov 18, 2025
Grant date	Nov 18, 2025

Abstract

A computer that includes a processor and a memory, the memory including instructions executable by the processor to generate a current keyframe point cloud based on pairs of stereo images while a stereo camera travels through a scene to determine a similar viewpoint query matrix and an opposing viewpoint query matrix based the current keyframe point cloud. A distance matrix and an opposing view distance matrix can be generated by comparing the similar viewpoint query matrix and the opposing viewpoint query matrix to reference matrices. A relative pose between a stereo camera and a reference can be determined to determine a location in the scene during travel of the stereo camera through the scene by performing sequence matching in the distance matrix and the opposing view distance matrix to determine a minimum sequence.

First claim

The invention claimed is:

1 . A system, comprising:
a computer that includes a processor and a memory, the memory including instructions executable by the processor to: generate a current keyframe point cloud based on pairs of stereo images while a stereo camera travels through a scene; determine a similar viewpoint query matrix and an opposing viewpoint query matrix based on the current keyframe point cloud wherein the opposing viewpoint query matrix is determined by flipping the similar viewpoint query matrix laterally and longitudinally by exchanging opposing rows and columns of the similar viewpoint query matrix; generate a distance matrix and an opposing view distance matrix by comparing the similar viewpoint query matrix and the opposing viewpoint query matrix to reference matrices; and determine a relative pose between a stereo camera and a reference to determine a location in the scene during travel of the stereo camera through the scene by performing sequence matching in the distance matrix and the opposing view distance matrix to determine a minimum sequence.

2 . The system of claim 1 , the instructions including further instructions to generate the keyframe point cloud based on the pairs of stereo images by projecting pixel data from the pairs of stereo images into the keyframe point cloud based on a pose estimate for the stereo camera and camera intrinsic parameters from one or more of a left or right camera of the stereo camera.

3 . The system of claim 1 , the instructions including further instructions to determine the similar viewpoint query matrix and the opposing viewpoint query matrix based on data points included in the current keyframe point cloud and distances above a ground plane of the data points.

4 . The system of claim 1 , wherein the similar viewpoint query matrix is arranged in columns that are parallel to a direction of motion of the stereo camera and rows that are perpendicular to the direction of motion of the stereo camera.

5 . The system of claim 1 , the instructions including further instructions to compare the similar viewpoint query matrix and the opposing viewpoint query matrix to the reference matrices to generate similar view distance matrix columns and opposing view distance matrix columns by shifting the similar viewpoint query matrix and the opposing viewpoint query matrix laterally and longitudinally with respect to the reference matrices and comparing overlapping bin values.

6 . The system of claim 5 , the instructions including further instructions to perform sequence matching on the similar view distance matrix columns and the opposing view distance matrix columns by searching for diagonals that include minimum values that indicate matches.

7 . The system of claim 1 , the instructions including further instructions to generate reference keyframes by traveling in the scene in a first direction while acquiring pairs of stereo images and generating the reference keyframes at equal displacements of the stereo camera.

8 . The system of claim 1 , the instructions including further instructions to determine a real world location in the scene determined during the travel of the stereo camera through the scene based on a georeferenced reference database.

9 . The system of claim 1 , the instructions including further instructions to operate a vehicle based on the location in the scene by determining a path polynomial.

10 . The system of claim 5 , wherein the bin values are a maximum distance above a ground plane.

11 . A method, comprising:
generating a current keyframe point cloud based on pairs of stereo images while a stereo camera travels through a scene; determining a similar viewpoint query matrix and an opposing viewpoint query matrix based on the current keyframe point cloud; generating a distance matrix and an opposing view distance matrix by comparing the similar viewpoint query matrix and the opposing viewpoint query matrix to reference matrices wherein the opposing viewpoint query matrix is determined by flipping the similar viewpoint query matrix laterally and longitudinally by exchanging opposing rows and columns of the similar viewpoint query matrix; and determining a relative pose between a stereo camera and a reference to determine a location in the scene during travel of the stereo camera through the scene by performing sequence matching in the distance matrix and the opposing view distance matrix to determine a minimum sequence.

12 . The method of claim 11 , further comprising generating the keyframe point cloud based on the pairs of stereo images by projecting pixel data from the pairs of stereo images into the keyframe point cloud based on a pose estimate for the stereo camera and camera intrinsic parameters from one or more of a left or right camera of the stereo camera.

13 . The method of claim 11 , further comprising determining the similar viewpoint query matrix and the opposing viewpoint query matrix based on data points included in the current keyframe point cloud and distances above a ground plane of the data points.

14 . The method of claim 11 , wherein the similar viewpoint query matrix is arranged in columns that are parallel to a direction of motion of the stereo camera and rows that are perpendicular to the direction of motion of the stereo camera.

15 . The method of claim 11 , further comprising comparing the similar viewpoint query matrix and the opposing viewpoint query matrix to the reference matrices to generate similar view distance matrix columns and opposing view distance matrix columns by shifting the similar viewpoint query matrix and the opposing viewpoint query matrix laterally and longitudinally with respect to the reference matrices and comparing overlapping bin values.

16 . The method of claim 15 , further comprising performing sequence matching on the similar view distance matrix columns and the opposing view distance matrix columns by searching for diagonals that include minimum values that indicate matches.

17 . The method of claim 11 , further comprising generating reference keyframes by traveling in the scene in a first direction while acquiring pairs of stereo images and generating the reference keyframes at equal displacements of the stereo camera.

18 . The method of claim 11 , further comprising determining a real world location in the scene determined during the travel of the stereo camera through the scene based on a georeferenced reference database.

19 . The method of claim 11 , further comprising operating a vehicle based on the location in the scene by determining a path polynomial.

20 . The method of claim 15 , wherein the bin values are a maximum distance above a ground plane.

Assignees

• Ford Global Tech Llc

Inventors

• Carmichael Spencer
• Vasudevan Ramanarayan
• Skinner Katherine
• Agrawal Rahul Kumar
• Carlson Alexandra
• Pandey Gaurav
• Parchami Mostafa

Classifications

• G06T2207/10012

  Stereo images · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T2207/30244

  Camera pose · CPC title

• G06T2207/30252

  Vehicle exterior; Vicinity of vehicle · CPC title

• G06T7/74 Primary

  involving reference images or patches · CPC title