Stationary-vehicle structure from motion

Patent metadata

Field	Value
Publication number	US-10108864-B2
Application number	US-201615235516-A
Country	US
Kind code	B2
Filing date	Aug 12, 2016
Priority date	Dec 29, 2015
Publication date	Oct 23, 2018
Grant date	Oct 23, 2018

Abstract

A vehicular structure from motion (SfM) system can store a number of image frames acquired from a vehicle-mounted camera in a frame stack according to a frame stack update logic. The SfM system can detect feature points, generate flow tracks, and compute depth values based on the image frames, the depth values to aid control of the vehicle. The frame stack update logic can select a frame to discard from the stack when a new frame is added to the stack, and can be changed from a first in, first out (FIFO) logic to last in, first out (LIFO) logic upon a determination that the vehicle is stationary. An optical flow tracks logic can also be modified based on the determination. The determination can be made based on a dual threshold comparison to insure robust SfM system performance.

First claim

What is claimed is:

1. A structure from motion (SfM) system for controlling a vehicle, the system comprising:
at least one processor; and
at least one non-transitory computer readable storage medium storing a program for execution by the at least one processor, the program including instructions to:
receive a sequence of image frames from a camera on the vehicle;
store, in the at least one non-transitory computer readable storage medium, a portion of the sequence of image frames in a frame stack, by selecting, according to a frame stack logic, a frame to discard from the frame stack, in response to adding a new frame to the frame stack;
compute depth values based on the frame stack;
modify the frame stack update logic from first in, first out (FIFO) logic to last in, first out (LIFO) logic, in response to determining that the vehicle is stationary; and
send, to a vehicle controller, the depth values for controlling the vehicle.

2. The system of claim 1 , wherein the instructions further comprise instructions to modify the frame stack update logic from LIFO logic to FIFO logic, in response to determining that the vehicle is moving.

3. The system of claim 2 , wherein instructions to determine that the vehicle is moving comprise instructions to:
estimate a pose of the camera and compute translation vectors for image frames in the sequence of image frames relating the pose of the camera to a reference position;
compare a first value with a first threshold in a first comparison, the first value indicating a magnitude of a difference between the translation vectors corresponding to image frames of the sequence of image frames that are consecutive in time;
compare a second value with a second threshold in a second comparison, the second value indicating a magnitude of a difference between a translation vector corresponding to a most recently acquired frame and a translation vector corresponding to the last frame acquired while the vehicle was moving; and
determine that the vehicle is moving in response to determining that that:
the first value is equal to or greater than the first threshold; and
the second value is equal to or greater than the second threshold.

4. The system of claim 1 , wherein the instructions further comprise instructions to:
implement optical flow tracks logic to prune optical flow tracks generated from corresponding feature points in different frames; and
modify the optical flow tracks logic from pruning based on the last-computed set of tracks to pruning based on the last set of tracks computed from a frame acquired while the vehicle was moving, in response to determining that the vehicle is stationary.

5. The system of claim 4 , wherein the instructions further comprise instructions to modify the optical flow tracks logic from pruning based on the last set of tracks computed from a frame acquired while the vehicle was moving to pruning based on the last-computed set of tracks, in response to determining that the vehicle is moving.

6. The system of claim 1 , wherein the instructions further comprise instructions to:
estimate a pose of the camera and compute translation vectors relating the pose of the camera to a reference position;
compare a first value with a first threshold in a first comparison, the first value indicating a magnitude of a difference between the translation vectors corresponding to acquired image frames that are consecutive in time;
compare a second value with a second threshold in a second comparison, the second value indicating a magnitude of a difference between a translation vector corresponding to the most recently acquired frame and a translation vector corresponding to the last frame acquired while the vehicle was moving; and
determine that the vehicle is stationary based on both the first and second comparisons.

7. The system of claim 6 , wherein the instructions to determine that the vehicle is stationary comprise instructions to determine that:
the first value is less than the first threshold; or
the second value is less than the second threshold.

8. The system of claim 6 , wherein the first threshold is between 0.02 meters and 0.05 meters.

9. The system of claim 6 , wherein the second threshold is product of the first threshold and a sliding window size equal to a number of frames in the frame stack.

10. The system of claim 1 , wherein the at least one processor comprise:
a vision processor to detect feature points and generate flow tracks; and
a digital signal processor (DSP) to compute a fundamental matrix, estimate a pose of the camera, and perform 3D triangulation to compute 3D sparse points.

11. A method for controlling a vehicle using structure from motion (SfM), the method comprising:
receiving, by at least one processor from a camera on the vehicle, a sequence of image frames;
storing, by the at least one processor, in a memory, a portion of the sequence of image frames in a frame stack by selecting, according to a frame stack logic, a frame to discard from the frame stack, in response to adding a new frame to the frame stack;
modifying, by the at least one processor, the frame stack update logic from first in, first out (FIFO) logic to last in, first out (LIFO) logic, in response to determining that the vehicle is stationary;
computing, by the at least one processor, depth values based on the frame stack; and
sending, by the at least one processor to a vehicle controller, the depth values, for controlling the vehicle.

12. The method of claim 11 , further comprising:
modifying the frame stack update logic from LIFO logic to FIFO logic, in response to determining that the vehicle is moving.

13. The method of claim 11 , further comprising:
computing a set of tracks based on feature points in a current image frame in the sequence of image frames and a previous image frame in the sequence of image frames;
pruning optical flow tracks in a current set of optical flow tracks based on a previous set of optical flow tracks tracks in accordance with a pruning scheme; and
modifying the pruning scheme from pruning based on the last-computed set of tracks to pruning based on the last set of tracks computed from a frame acquired while the vehicle was moving, in response to determining that the vehicle is stationary.

14. The method of claim 13 , further comprising:
modify the pruning scheme from pruning based on the last set of tracks computed from a frame acquired while the vehicle was moving to pruning based on the last-computed set of tracks, in response to determining that the vehicle is moving.

15. The method of claim 11 , further comprising:
estimating a pose of the camera and computing translation vectors relating the pose of the camera to a reference position;
in a first comparison, comparing a first value with a first threshold, the first value indicating a magnitude of a difference between the translation vectors corresponding to acquired image frames that are consecutive in time;
in a second comparison, comparing a second value with a second threshold, the second value indicating a magnitude of a difference between the translation vector corresponding to the most recently acquired frame and the translation vector corresponding to the last frame acquired while the vehicle was moving; and
determining whether the vehicle is stationary based on both the first and second comparisons.

16. The method of claim 15 , further comprising determining that the vehicle is stationary in response to determining that:
the first value is less than the first threshold; or
the second value is less than the se

Assignees

• Texas Instruments Inc

Inventors

• Viswanath Prashanth Ramanathpu
• Nagori Soyeb
• Mathew Manu

Classifications

• G06T7/579 Primary

  from motion · CPC title

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G06V10/44

  Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components · CPC title

• G06T2207/30244

  Camera pose · CPC title

• G06T2207/30252

  Vehicle exterior; Vicinity of vehicle · CPC title