Computing device and method of removing raindrops from video images

Patent metadata

Field	Value
Publication number	US-11615511-B2
Application number	US-202017107586-A
Country	US
Kind code	B2
Filing date	Nov 30, 2020
Priority date	Aug 18, 2020
Publication date	Mar 28, 2023
Grant date	Mar 28, 2023

Abstract

A method of removing raindrops from video images is provided. The method includes the steps of: training a raindrop image recognition model using a plurality raindrop training images labeled in a plurality of rainy-scene images; recognizing a plurality of raindrop images from a plurality of scene images in a video sequence using the raindrop image recognition model; and in response to a specific raindrop image in a current scene image satisfying a predetermined condition, replacing the specific raindrop image in the current scene image with an image region corresponding to the specific raindrop image in a specific scene image prior to the current scene image to generate an output scene image.

First claim

What is claimed is:

1. A method of removing raindrops from video images, comprising:
training a raindrop-image-recognition model using a plurality of raindrop training images labeled in a plurality of rainy scene images;
recognizing a plurality of raindrop images in a plurality of scene images of a video sequence using the raindrop-image-recognition model;
recording a position and size of each recognized raindrop image in the corresponding scene image; and
calculating an intersection over union (IoU) of the specific raindrop image in the current scene image and the specific raindrop image in the specific scene image;
in response to the IoU of a specific raindrop image in the current scene image and the specific raindrop image in the specific scene image being smaller than a predetermined ratio threshold, replacing the specific raindrop image in the current scene image with an image region corresponding to the specific raindrop image in a specific scene image prior to the current scene image to generate an output scene image; and
when the IoUs of the specific raindrop image in the current scene image and the specific raindrop image in multiple previous scene images are smaller than the predetermined ratio threshold, selecting one previous scene image of the multiple previous scene images having the smallest IoU in a revertible candidate list or one previous scene image of the multiple previous scene images having a time point closest to the current scene image in a revertible candidate list.

2. The method as claimed in claim 1 , wherein the specific raindrop image is selected from a region of interest (ROI) in the current scene image.

3. The method as claimed in claim 1 , wherein there are a first number of raindrop images in the current scene image, and the method further comprises:
in response to a number of raindrop images successfully replaced reaching a predetermined ratio of the first number, stopping replacing the remaining raindrop images in the current scene image.

4. A computation apparatus, comprising:
a non-volatile memory, storing a raindrop-removal program; and
a processor, configured to execute the raindrop-removal program to perform the following steps:
training a raindrop-image-recognition model using a plurality of raindrop training images labeled in a plurality of rainy scene images;
recognizing a plurality of raindrop images in a plurality of scene images of a video sequence using the raindrop-image-recognition model;
recording a position and size of each recognized raindrop image in the corresponding scene image; and
calculating an intersection over union (IoU) of the specific raindrop image in the current scene image and the specific raindrop image in the specific scene image;
in response to the IoU of a specific raindrop image in the current scene image and the specific raindrop image in the specific scene image being smaller than a predetermined ratio threshold, replacing the specific raindrop image in the current scene image with an image region corresponding to the specific raindrop image in a specific scene image prior to the current scene image to generate an output scene image; and
when the IOUs of the specific raindrop image in the current scene image and the specific raindrop image in multiple previous scene images are smaller than the predetermined ratio threshold, the processor selects one previous scene image of the multiple previous scene images having the smallest IoU in a revertible candidate list or one previous scene image of the multiple previous scene images having a time point closest to the current scene image in a revertible candidate list.

5. The computation apparatus as claimed in claim 4 , wherein there are a first number of raindrop images in the current scene image, and in response to a number of raindrop images successfully replaced reaching a predetermined ratio of the first number, the processor stops replacing the remaining raindrop images in the current scene image.

Assignees

• Quanta Comp Inc

Inventors

• Lin Chien-Hung
• Wang yang-sheng
• Chen Yu-Ching
• Yeh Chih-Tsang
• Yang Po-An

Classifications

• G06V10/774

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V10/82

  using neural networks · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G06T7/60

  Analysis of geometric attributes · CPC title