Method for monitoring blind spot of monitoring vehicle and blind spot monitor using the same

Patent metadata

Field	Value
Publication number	US-9934440-B1
Application number	US-201715724544-A
Country	US
Kind code	B1
Filing date	Oct 4, 2017
Priority date	Oct 4, 2017
Publication date	Apr 3, 2018
Grant date	Apr 3, 2018

Abstract

A method of monitoring a blind spot of a monitoring vehicle by using a blind spot monitor. The method includes steps of: the blind spot monitor (a) acquiring a feature map from rear video images, on condition that video images with reference vehicles in the blind spot are acquired, reference boxes for the reference vehicles are created, and the reference boxes are set as proposal boxes; (b) acquiring feature vectors for the proposal boxes on the feature map by pooling, inputting the feature vectors into a fully connected layer, acquiring classification and regression information; and (c) selecting proposal boxes by referring to the classification information, acquiring bounding boxes for the proposal boxes by using the regression information, confirming whether the bounding boxes match their corresponding proposal boxes, and determining whether the monitored vehicle is in the proposal boxes to determine the monitored vehicle is in the blind spot.

First claim

What is claimed is:

1. A method of monitoring a blind spot of a monitoring vehicle by using a blind spot monitor, comprising steps of:
(a) the blind spot monitor, if rear video images for testing are acquired from the monitoring vehicle in operation, acquiring or supporting another device to acquire at least one feature map for testing from the rear video images for testing, on condition that the blind spot monitor completes or supports another device to complete (i) a process of acquiring rear video images for sampling with regard to one or more reference vehicles located per distance from the monitoring vehicle and located in the blind spot of the monitoring vehicle, (ii) a process of creating reference boxes corresponding to the reference vehicles in each of the rear video images for sampling, and (iii) a process of setting the reference boxes as m proposal boxes which serve as candidate regions in the blind spot of the monitoring vehicle, wherein the candidate regions have probabilities of detecting at least one monitored vehicle;
(b) the blind spot monitor (i) acquiring or supporting another device to acquire each of feature vectors for testing corresponding to each of the m proposal boxes by applying pooling operation to the m proposal boxes on the feature map for testing, (ii) inputting or supporting another device to input the feature vectors for testing into at least one fully connected layer, (iii) acquiring or supporting another device to acquire classification scores for testing for each class, corresponding to each of the m proposal boxes, and (iv) acquiring or supporting another device to acquire regression information for testing for each class, corresponding to each of the m proposal boxes; and
(c) the blind spot monitor performing or supporting another device to perform (i) a process of selecting j proposal boxes among the m proposal boxes by referring to the classification scores for testing, (ii) a process of acquiring each of bounding boxes corresponding to each of the j proposal boxes by using the regression information for testing corresponding to each of the j proposal boxes, (iii) a process of confirming whether each of the bounding boxes matches its corresponding proposal box among the j proposal boxes to a degree equal to or greater than a first threshold, and (iv) a process of determining whether the monitored vehicle is located in any of the j proposal boxes, to thereby determine whether the monitored vehicle is located in the blind spot of the monitoring vehicle.

2. The method of claim 1 , further comprising a step of:
(d) the blind spot monitor, if at least one specific bounding box that matches its corresponding proposal box among the j proposal boxes to a degree less than a second threshold is determined as found, setting or supporting another device to set the specific bounding box as at least one dynamic proposal box which is added as a new element of a group including the candidate regions, to thereby acquire an updated group.

3. The method of claim 2 , wherein, at the step of (d), if the specific bounding box is acquired from a t frame of the rear video images for testing, the blind spot monitor sets or supports another device to set the dynamic proposal box and the m proposal boxes as being included in the updated group in a t+1 frame of the rear video images for testing.

4. The method of claim 3 , wherein the blind spot monitor determines or supports another device to determine whether the monitored vehicle is located in the m proposal boxes on a t+k+1 frame, on condition that (i) the monitored vehicle is determined as located in the dynamic proposal box from a t+1 frame to a t+(k−1) frame, and (ii) the monitored vehicle is determined as not located in the dynamic proposal box on a t+k frame.

5. The method of claim 1 , further comprising a step of:
(e) the blind spot monitor supporting a control unit to prevent the monitoring vehicle from changing lanes in a direction to the blind spot where the monitored vehicle is determined as located by transmitting information on the monitored vehicle located in the blind spot to the control unit.

6. The method of claim 1 , wherein, at the step of (c), the blind spot monitor (i) calculates or supports another device to calculate first overlapping areas where each of the bounding boxes and its corresponding j proposal boxes overlap each other, and (ii) determines or supports another device to determine a certain proposal box, corresponding to a specific first overlapping area among the first overlapping areas that is equal to or greater than a third threshold, as including the monitored vehicle.

7. The method of claim 1 , wherein, at the step of (c), the blind spot monitor (i) calculates or supports another device to calculate second overlapping areas where the bounding boxes, each of which corresponds to each of the j proposal boxes, overlap each other, (ii) determines or supports another device to determine at least one particular bounding box, corresponding to at least one specific second overlapping area, among the second overlapping areas, confirmed to be equal to or greater than a fourth threshold, as including a single identical monitored vehicle, and (iii) determines or supports another device to determine a selected bounding box among the particular bounding box, having a largest overlapping area with its corresponding proposal box, as including the monitored vehicle.

8. The method of claim 1 , wherein, at the step of (c), the blind spot monitor (i) calculates or supports another device to calculate second overlapping areas where the bounding boxes, each of which corresponds to each of the j proposal boxes, overlap each other, (ii) determines or supports another device to determine especial bounding boxes, corresponding to specific second overlapping areas, among the second overlapping areas, confirmed to be less than a fifth threshold, as including respective monitored vehicles, and (iii) determines or supports another device to determine the especial bounding boxes as including the respective monitored vehicles.

9. The method of claim 1 , wherein, at the step of (a), the blind spot monitor, on condition that a pad is set at zero, performs or supports another device to perform applying convolution operation to the rear video images for testing or to the feature map for testing acquired by using the rear video images, with a filter being slid at a predetermined stride.

10. The method of claim 1 , wherein, at the step of (a), the blind spot monitor acquires or supports another device to acquire the feature map for testing by applying convolution operation to a subject image corresponding to the blind spot in the rear video images for testing.

11. The method of claim 1 , wherein the blind spot monitor performs convolution, classification, and box regression by using at least one convolution parameter, at least one classification parameter, and at least one box regression parameter having been adjusted by a learning device, and
wherein the learning device performs or supports another device to perform
(i) a process of acquiring feature maps for training by applying convolution operation to training images, a process of inputting the feature maps for training into a region proposal network, and a process of acquiring proposal boxes for training corresponding to objects located in the training images, (ii) a process of acquiring each of feature vectors for training corresponding to each of the proposal boxes for training by applying pooling operation to areas of the training images corresponding to the proposal boxes for training, a process of inputting the feature vectors for training into each of the fully connected layers, and a pr

Assignees

• Stradvision Inc

Inventors

• Kim Yongjoong
• Nam Woonhyun
• Boo Sukhoon
• Sung Myungchul
• Yeo Donghun
• Ryu Wooju
• Jang Taewoong
• Jeong Kyungjoong
• Je Hongmo
• Cho Hojin

Classifications

• G06N3/08

  Learning methods · CPC title

• B60R2300/802

  for monitoring and displaying vehicle exterior blind spot views · CPC title

• B60R2300/30

  characterised by the type of image processing · CPC title

• B60Y2400/3015

  Optical cameras · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title