Stationary-vehicle structure from motion

Patent metadata

Field	Value
Publication number	US-10354149-B2
Application number	US-201816121012-A
Country	US
Kind code	B2
Filing date	Sep 4, 2018
Priority date	Dec 29, 2015
Publication date	Jul 16, 2019
Grant date	Jul 16, 2019

Abstract

A vehicular structure from motion (SfM) system can store a number of image frames acquired from a vehicle-mounted camera in a frame stack according to a frame stack update logic. The SfM system can detect feature points, generate flow tracks, and compute depth values based on the image frames, the depth values to aid control of the vehicle. The frame stack update logic can select a frame to discard from the stack when a new frame is added to the stack, and can be changed from a first in, first out (FIFO) logic to last in, first out (LIFO) logic upon a determination that the vehicle is stationary. An optical flow tracks logic can also be modified based on the determination. The determination can be made based on a dual threshold comparison to insure robust SfM system performance.

First claim

What is claimed is:

1. An automotive system on a vehicle, comprising:
a camera configured to generate a sequence of image frames;
a structure from motion (SfM) system coupled to the camera, the SfM system comprising:
at least one processor; and
at least one non-transitory computer readable storage medium storing a program for execution by the at least one processor, the program including instructions to cause the at least one processor to:
receive the sequence of image frames from the camera;
store, in the at least one non-transitory computer readable storage medium,

a portion of the sequence of image frames in a frame stack, by selecting, according to a frame stack logic, a frame to discard from the frame stack, in response to adding a new frame to the frame stack;
compute depth values based on the frame stack;
modify the frame stack update logic from first in, first out (FIFO) logic to last in, first out (LIFO) logic, in response to determining that the vehicle is stationary; and
send, to a vehicle controller, the depth values; and

the vehicle controller coupled to the SfM system, the vehicle controller configured to control the vehicle based on the depth values.

2. The automotive system of claim 1 , wherein the instructions further comprise instructions to cause the at least one processor to modify the frame stack update logic from LIFO logic to FIFO logic, in response to determining that the vehicle is moving.

3. The automotive system of claim 2 , wherein instructions to determine that the vehicle is moving comprise instructions to cause the at least one processor to:
estimate a pose of the camera and compute translation vectors for image frames in the sequence of image frames relating the pose of the camera to a reference position;
compare a first value with a first threshold in a first comparison, the first value indicating a magnitude of a difference between the translation vectors corresponding to image frames of the sequence of image frames that are consecutive in time;
compare a second value with a second threshold in a second comparison, the second value indicating a magnitude of a difference between a translation vector corresponding to a most recently acquired frame and a translation vector corresponding to the last frame acquired while the vehicle was moving; and
determine that the vehicle is moving in response to determining that:
the first value is equal to or greater than the first threshold; and
the second value is equal to or greater than the second threshold.

4. The automotive system of claim 1 , wherein the instructions further comprise instructions to cause the at least one processor to:
implement optical flow tracks logic to prune optical flow tracks generated from corresponding feature points in different frames; and
modify the optical flow tracks logic from pruning based on the last-computed set of tracks to pruning based on the last set of tracks computed from a frame acquired while the vehicle was moving, in response to determining that the vehicle is stationary.

5. The automotive system of claim 4 , wherein the instructions further comprise instructions to cause the at least one processor to modify the optical flow tracks logic from pruning based on the last set of tracks computed from a frame acquired while the vehicle was moving to pruning based on the last-computed set of tracks, in response to determining that the vehicle is moving.

6. The automotive system of claim 1 , wherein the instructions further comprise instructions to cause the at least one processor to:
estimate a pose of the camera and compute translation vectors relating the pose of the camera to a reference position;
compare a first value with a first threshold in a first comparison, the first value indicating a magnitude of a difference between the translation vectors corresponding to acquired image frames that are consecutive in time;
compare a second value with a second threshold in a second comparison, the second value indicating a magnitude of a difference between a translation vector corresponding to the most recently acquired frame and a translation vector corresponding to the last frame acquired while the vehicle was moving; and
determine that the vehicle is stationary based on both the first and second comparisons.

7. The automotive system of claim 6 , wherein the instructions to determine that the vehicle is stationary comprise instructions to cause the at least one processor to determine that:
the first value is less than the first threshold; or
the second value is less than the second threshold.

8. The automotive system of claim 6 , wherein the first threshold is between 0.02 meters and 0.05 meters.

9. The automotive system of claim 6 , wherein the second threshold is product of the first threshold and a sliding window size equal to a number of frames in the frame stack.

10. The automotive system of claim 1 , wherein the at least one processor comprise:
a vision processor to detect feature points and generate flow tracks; and
a digital signal processor (DSP) to compute a fundamental matrix, estimate a pose of the camera, and perform 3D triangulation to compute 3D sparse points.

11. The automotive system of claim 1 , further comprising an inertial measurement unit (IMU) coupled to the SfM system, the IMU configured to provide information to the SfM system indicating whether the vehicle is stationary.

12. At least one non-transitory computer readable storage medium storing a program for execution by at least one processor, the program including instructions to cause the at least one processor to:
receive a sequence of image frames from a camera on a vehicle;
store, in the at least one non-transitory computer readable storage medium, a portion of the sequence of image frames in a frame stack, by selecting, according to a frame stack logic, a frame to discard from the frame stack, in response to adding a new frame to the frame stack;
compute depth values based on the frame stack;
modify the frame stack update logic from first in, first out (FIFO) logic to last in, first out (LIFO) logic, in response to determining that the vehicle is stationary; and
send, to a vehicle controller, the depth values for controlling the vehicle.

13. The at least one non-transitory computer readable storage medium of claim 12 , wherein the instructions further comprise instructions to cause the at least one processor to modify the frame stack update logic from LIFO logic to FIFO logic, in response to determining that the vehicle is moving.

14. The at least one non-transitory computer readable storage medium of claim 13 , wherein instructions to determine that the vehicle is moving comprise instructions to cause the at least one processor to:
estimate a pose of the camera and compute translation vectors for image frames in the sequence of image frames relating the pose of the camera to a reference position;
compare a first value with a first threshold in a first comparison, the first value indicating a magnitude of a difference between the translation vectors corresponding to image frames of the sequence of image frames that are consecutive in time;
compare a second value with a second threshold in a second comparison, the second value indicating a magnitude of a difference between a translation vector corresponding to a most recently acquired frame and a translation vector corresponding to the last frame acquired while the vehicle was moving; and
determine that the vehicle is moving in response to determining that:
the first value is equal to or greater than the first threshold; and
the second value is e

Assignees

• Texas Instruments Inc

Inventors

• Viswanath Prashanth Ramanathpu
• Nagori Soyeb
• Mathew Manu

Classifications

• G06T7/579 Primary

  from motion · CPC title

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G06V10/44

  Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components · CPC title

• G06T2207/30244

  Camera pose · CPC title

• G06T7/269

  using gradient-based methods · CPC title