Method for monitoring blind spot of vehicle and blind spot monitor using the same

Patent metadata

Field	Value
Publication number	US-9947228-B1
Application number	US-201715725394-A
Country	US
Kind code	B1
Filing date	Oct 5, 2017
Priority date	Oct 5, 2017
Publication date	Apr 17, 2018
Grant date	Apr 17, 2018

Abstract

A method of monitoring a blind spot of a monitoring vehicle by using a blind spot monitor is provided. The method includes steps of: the blind spot monitor (a) acquiring a feature map from rear video images, on condition that video images with reference vehicles in the blind spot are acquired, reference boxes for the reference vehicles are created, and the reference boxes are set as proposal boxes; (b) acquiring feature vectors for the proposal boxes on the feature map by pooling, inputting the feature vectors into a fully connected layer, acquiring classification and regression information; and (c) selecting proposal boxes by referring to the classification information, acquiring bounding boxes for the proposal boxes by using the regression information, determining the pose of the monitored vehicle corresponding to each of the bounding boxes, and determining whether a haphazard vehicle is located in the blind spot of the monitoring vehicle.

First claim

What is claimed is:

1. A method for monitoring a blind spot of a vehicle, comprising steps of:
(a) a blind spot monitor acquiring or supporting another device to acquire at least one feature map from a rear video image if the rear video image is acquired from a monitoring vehicle in a driving state;
(b) the blind spot monitor performing or supporting another device to perform (i) a process of acquiring m proposal boxes corresponding to one or more objects located in the rear video image, (ii) a process of acquiring feature vectors each of which corresponds to each of the m proposal boxes by applying pooling operation to each area, on the feature map, corresponding to the m proposal boxes, and (iii) a process of inputting each of the feature vectors corresponding to each of the m proposal boxes into a first FC layer to acquire (iii-1) each of first classification scores for each of first kinds of class, which correspond to each of the m proposal boxes, to confirm whether the objects are monitored vehicles, and (iii-2) each of second classification scores for each of second kinds of class, which correspond to each of the m proposal boxes, to confirm poses of the objects; and
(c) the blind spot monitor performing or supporting another device to perform (i) a process of selecting n proposal boxes, which have probabilities over a certain threshold to be regarded as including the monitored vehicle, among the m proposal boxes by referring to the first classification scores, (ii) a process of acquiring n bounding boxes, each of which corresponds to each of the n proposal boxes, by referring to regression information corresponding to each of the n proposal boxes, (iii) a process of determining the pose of the monitored vehicle corresponding to each of at least some of the n bounding boxes by referring to the second classification scores, and (iv) a process of determining whether a haphazard vehicle is located in a blind spot of the monitoring vehicle by referring to at least some of the n bounding boxes and the pose of the monitored vehicle, and
wherein, at the step of (c), the blind spot monitor performs or supports another device to perform (i) a process of calculating first overlap ratios which are pairwise-overlapping ratios among the n bounding boxes, (ii) a process of determining specific bounding boxes confirmed to have the first overlap ratios equal to or greater than a first threshold among then-bounding boxes as corresponding to a single identical monitored vehicle, (iii) a process of calculating second overlap ratios which are pairwise-overlapping ratios among the specific bounding boxes and their respective corresponding proposal boxes, and (iv) a process of determining a certain bounding box with a maximum ratio among the second overlap ratios as including the single identical monitored vehicle.

2. The method of claim 1 , wherein, at the step of (b), the blind spot monitor performs or supports another device to perform a process of inputting each of the feature vectors corresponding to each of the m proposal boxes into a second FC layer to further acquire (iii-3) regression information for each of the first kinds of class, which corresponds to each of the m proposal boxes, by applying regression operation to the feature vectors through the second FC layer.

3. The method of claim 1 , further comprising a step of:
(d) the blind spot monitor supporting a control unit to prevent the monitoring vehicle from changing lanes in a direction to the blind spot where the haphazard vehicle is located by transmitting information on the haphazard vehicle to the control unit if the haphazard vehicle is determined as located in the blind spot.

4. The method of claim 1 , wherein, at the step of (c), the blind spot monitor performs or supports another device to perform (i) a process of selecting k bounding boxes determined as located in the blind spot of the monitoring vehicle among the n bounding boxes, and then (ii) a process of determining that the haphazard vehicle is located in the blind spot if the pose of the monitored vehicle in at least one of the k bounding boxes corresponds to a direction in which the monitoring vehicle is traveling.

5. The method of claim 4 , wherein the blind spot monitor determines or supports another device to determine that the monitored vehicle is traveling in a same direction with the monitoring vehicle if a view of the monitored vehicle seen from the monitoring vehicle is a front face, a left front face, or a right front face.

6. The method of claim 5 , wherein the blind spot monitor determines or supports another device to determine the haphazard vehicle as not located in the blind spot of the monitoring vehicle if (i) first conditions that a first bounding box, among the k bounding boxes, is determined as located in the blind spot formed in a back left area apart from the monitoring vehicle and that the view of the monitored vehicle seen from the monitoring vehicle include the front face or the left front face or (ii) second conditions that a second bounding box, among the k bounding boxes, is determined as located in the blind spot formed in a back right area apart from the monitoring vehicle and that the view of the monitored vehicle seen from the monitoring vehicle include the front face or the right front face are satisfied.

7. The method of claim 1 , wherein, at the step of (c), the blind spot monitor performs or supports another device to perform (i) a process of calculating first overlap ratios which represent pairwise-overlapping ratios among the n bounding boxes and (ii) a process of determining particular bounding boxes confirmed to have the first overlap ratios less than a second threshold among the n bounding boxes as including respective monitored vehicles.

8. The method of claim 1 , wherein, at the step of (a), the blind spot monitor applies or supports another device to apply convolution operation to the rear video image to thereby acquire the feature map.

9. The method of claim 8 , wherein, the blind spot monitor, on condition that a pad is set at zero, applies or supports another device to apply convolution operation to the rear video image or its corresponding feature map, with a filter being slid at a predetermined stride.

10. The method of claim 1 , wherein, at the step of (a), the blind spot monitor applies or supports another device to apply convolution operation to a part of the rear video image corresponding to the blind spot to thereby acquire the feature map.

11. A blind spot monitor for monitoring a blind spot of a vehicle, comprising:
a communication part for acquiring or supporting another device to acquire a rear video image or its corresponding at least one feature map, wherein the rear video image is acquired from a monitoring vehicle in a driving state;
a processor for performing or supporting another device to perform (i) a process of acquiring m proposal boxes corresponding to one or more objects located in the rear video image by using the feature map acquired from the communication part or the feature map calculated by applying one or more convolution operations to the rear video image map acquired from the communication part, (ii) a process of acquiring feature vectors each of which corresponds to each of the m proposal boxes by applying pooling operation to each area, on the feature map, corresponding to the m proposal boxes, (iii) a process of inputting each of the feature vectors corresponding to each of the m proposal boxes into a first FC layer to acquire (iii-1) each of first classification scores for each of first kinds of class, which correspond to each of the m proposal boxes, to confirm whether the objects are monitored vehicles, and (iii-2) eac

Assignees

• Stradvision Inc

Inventors

• Kim Yongjoong
• Nam Woonhyun
• Boo Sukhoon
• Sung Myungchul
• Yeo Donghun
• Ryu Wooju
• Jang Taewoong
• Jeong Kyungjoong
• Je Hongmo
• Cho Hojin

Classifications

• B60R2300/802

  for monitoring and displaying vehicle exterior blind spot views · CPC title

• B60R1/00 Primary

  Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles · CPC title

• G08G1/167 Primary

  Driving aids for lane monitoring, lane changing, e.g. blind spot detection · CPC title

• B60Q9/008

  for anti-collision purposes · CPC title

• G08G1/04

  using optical or ultrasonic detectors · CPC title