Stationary-vehicle structure from motion

Patent metadata

Field	Value
Publication number	US-2017186169-A1
Application number	US-201615235516-A
Country	US
Kind code	A1
Filing date	Aug 12, 2016
Priority date	Dec 29, 2015
Publication date	Jun 29, 2017
Grant date	—

Abstract

A vehicular structure from motion (SfM) system can store a number of image frames acquired from a vehicle-mounted camera in a frame stack according to a frame stack update logic. The SfM system can detect feature points, generate flow tracks, and compute depth values based on the image frames, the depth values to aid control of the vehicle. The frame stack update logic can select a frame to discard from the stack when a new frame is added to the stack, and can be changed from a first in, first out (FIFO) logic to last in, first out (LIFO) logic upon a determination that the vehicle is stationary. An optical flow tracks logic can also be modified based on the determination. The determination can be made based on a dual threshold comparison to insure robust SfM system performance.

First claim

What is claimed is:

1 . A vehicular structure from motion (SfM) system comprising:
an input to receive a sequence of image frames acquired from a camera on a vehicle; a memory to store a finite number of the frames in a frame stack according to a frame stack update logic, the frame stack update logic to select a frame to discard from the stack when a new frame is added to the stack; and one or more processors to implement the frame stack update logic and to compute depth values based on the image frames, the depth values to aid control of the vehicle; wherein the frame stack update logic is changed from a first in, first out (FIFO) logic to last in, first out (LIFO) logic upon a determination that the vehicle is stationary.

2 . The system of claim 1 , wherein the frame stack update logic is changed from the LIFO logic to the FIFO logic upon a determination that the vehicle is moving.

3 . The system of claim 2 , wherein the determination that the vehicle is moving is made by the one or more processors or by another component, the one or more processors or the other component to:
estimate the pose of the camera and compute a translation vector relating the camera pose to a reference position; compare a first value with a first threshold in a first comparison, the first value being the magnitude of the difference between the translation vectors corresponding to acquired image frames that are consecutive in time; compare a second value with a second threshold in a second comparison, the second value being the magnitude of the difference between the translation vector corresponding to the most recently acquired frame and the translation vector corresponding to the last frame acquired while the vehicle was moving; and determine that the vehicle is moving upon a determination that both of the following conditions is met:
the first value is equal to or greater than the first threshold; and
the second value is equal to or greater than the second threshold.

4 . The system of claim 1 , wherein the one or more processors implement an optical flow tracks logic to prune optical flow tracks generated from corresponding feature points in different frames, and wherein, upon a determination that the vehicle is stationary, the optical flow tracks logic is changed from pruning based on the last-computed set of tracks to pruning based on the last set of tracks computed from a frame acquired while the vehicle was moving.

5 . The system of claim 4 , wherein, upon a determination that the vehicle is moving, the optical flow tracks logic is changed from pruning based on the last set of tracks computed from a frame acquired while the vehicle was moving to pruning based on the last-computed set of tracks.

6 . The system of claim 1 , wherein the determination that the vehicle is stationary is made by the one or more processors or by another component, the one or more processors or the other component to:
estimate the pose of the camera and compute a translation vector relating the camera pose to a reference position; compare a first value with a first threshold in a first comparison, the first value being the magnitude of the difference between the translation vectors corresponding to acquired image frames that are consecutive in time; compare a second value with a second threshold in a second comparison, the second value being the magnitude of the difference between the translation vector corresponding to the most recently acquired frame and the translation vector corresponding to the last frame acquired while the vehicle was moving; and determine that the vehicle is stationary based on both the first and second comparisons.

7 . The system of claim 6 , wherein the determination that the vehicle is stationary is made upon a determination that at least one of the following conditions is met:
the first value is less than the first threshold; or the second value is less than the second threshold.

8 . The system of claim 6 , wherein the first threshold is between 0.02 meters and 0.05 meters.

9 . The system of claim 6 , wherein the second threshold is product of the first threshold and a sliding window size equal to the number of frames in the frame stack.

10 . The system of claim 1 , wherein the one or more processors comprise:
a vision processor to detect feature points and generate flow tracks; and a digital signal processor (DSP) to compute a fundamental matrix, estimate the pose of the camera, and perform 3D triangulation to compute 3D sparse points.

11 . A method comprising:
acquiring, from a camera on a vehicle, a sequence of image frames in a frame stack having an update scheme; determining whether or not the vehicle is stationary; based on determining that the vehicle is stationary, modifying the frame stack update scheme from a first in, first out (FIFO) scheme to a last in, first out (LIFO) scheme; computing a 3D point cloud based on the image frames in the frame stack; and controlling the vehicle based on the 3D point cloud.

12 . The method of claim 11 , further comprising:
determining that the vehicle is moving; and based on the determining that the vehicle is moving, modifying the frame stack update scheme from a LIFO scheme to a FIFO scheme.

13 . The method of claim 11 , further comprising:
for each frame in the sequence after an initial frame, computing a set of tracks based on feature points in the frame and an earlier frame; for each set of tracks, pruning the set based on the previously computed set of tracks in accordance with a pruning scheme; based on the determining that the vehicle is stationary, modifying the pruning scheme from pruning based on the last-computed set of tracks to pruning based on the last set of tracks computed from a frame acquired while the vehicle was moving.

14 . The method of claim 12 , further comprising:
based on the determining that the vehicle is moving, modify the pruning scheme from pruning based on the last set of tracks computed from a frame acquired while the vehicle was moving to pruning based on the last-computed set of tracks.

15 . The method of claim 11 , wherein the determining whether or not the vehicle is stationary comprises:
estimating the pose of the camera and computing a translation vector relating the camera pose to a reference position; in a first comparison, comparing a first value with a first threshold, the first value being the magnitude of the difference between the translation vectors corresponding to acquired image frames that are consecutive in time; in a second comparison, comparing a second value with a second threshold, the second value being the magnitude of the difference between the translation vector corresponding to the most recently acquired frame and the translation vector corresponding to the last frame acquired while the vehicle was moving; and determining whether or not the vehicle is stationary based on both the first and second comparisons.

16 . The method of claim 15 , wherein the determination that the vehicle is stationary is made upon a determination that at least one of the following conditions is met:
the first value is less than the first threshold; or the second value is less than the second threshold.

17 . A method comprising:
acquiring a sequence of image frames from a camera on a vehicle; estimating the camera pose and computing a translation vector relating the camera pose to a reference position; in a first comparison, comparing a first value with a first threshold, the first value being the magnitude of the differen

Assignees

• Texas Instruments Inc

Inventors

• Viswanath Prashanth Ramanathpu
• Nagori Soyeb
• Mathew Manu

Classifications

• G06T7/579 Primary

  from motion · CPC title

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G06V10/44

  Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components · CPC title

• G06T2207/30244

  Camera pose · CPC title

• G06T7/269

  using gradient-based methods · CPC title