Systems and methods of fusing multi-angle view HD images based on epipolar geometry and matrix completion

Patent metadata

Field	Value
Publication number	US-10212410-B2
Application number	US-201615386187-A
Country	US
Kind code	B2
Filing date	Dec 21, 2016
Priority date	Dec 21, 2016
Publication date	Feb 19, 2019
Grant date	Feb 19, 2019

Abstract

Systems and methods for image processing including a memory having stored multi-angled view images of a scene. Each multi-angled view image includes pixels, and at least one multi-angled view image includes a clouded area in the scene, resulting in missing pixels. A processor configured to align three multi-angled view images in the multi-angled view images to a target view angle of the scene, to form a set of aligned multi-angled view images representing a target point of view of the scene, at least one aligned multi-angled view image of the at least three multi-angled view images, has missing pixels due to the clouded area. Form a matrix using vectorized aligned multi-angled view images, wherein the matrix is incomplete due to the missing pixels. Complete the matrix using a matrix completion to combine the aligned multi-angled view images to produce a fused image of the scene without the clouded area.

First claim

What is claimed is:

1. A system for image processing, comprising:
a computer readable memory includes a set of stored multi-angled view images of a scene generated by sensors, such that each multi-angled view image includes pixels, and at least one multi-angled view image includes a clouded area in at least a portion of the scene, resulting in missing pixels;
a processor in communication with the computer readable memory is configured to:
align at least three multi-angled view images in the set of multi-angled view images to a target view angle of the scene, to form a set of aligned multi-angled view images representing a target point of view of the scene, such that at least one aligned multi-angled view image of the at least three multi-angled view images, has missing pixels due to the clouded area,
form a matrix using vectorized aligned multi-angled view images, wherein the matrix is incomplete due to the missing pixels, and
complete the matrix using a matrix completion to combine the aligned multi-angled view images to produce a fused image of the scene without the clouded area.

2. The system according to claim 1 , wherein the set of multi-angled view images are of a three dimensional (3D) scene and each multi-angled view image of the set of multi-angled view images are one of taken at a same time, a different time, or some combination thereof, of the 3D scene with unknown sensors positions relative to the 3D scene.

3. The system according to claim 2 , wherein the aligning of the at least three multi-angled view images to the target point of view of the 3D scene is based on an epipolar geometry of the 3D scene.

4. The system according to claim 1 , wherein the matrix completion is a low-rank matrix completion, such that each column of the low-rank matrix completion corresponds to a vectorized aligned multi-angled view image and the missing pixels of the at least one aligned multi-angled view image corresponds to the clouded area.

5. The system according to claim 1 , wherein the sensors are moving.

6. The system according to claim 1 , wherein the sensors are arranged in a satellite or an airplane.

7. The system according to claim 1 , wherein the scene includes occlusions or the clouded area caused by clouds between the sensors and scene and structures in the scene.

8. The system according to claim 1 , wherein the aligning of the multi-angled view images to the target view angle of the scene forms the set of aligned multi-angled view images representing the target point of view of the scene is based on a fundamental matrix.

9. The system according to claim 8 , wherein the fundamental matrix is estimated by key points in the multi-angled images.

10. The system according to claim 9 , wherein the key points are based on SIFT Matching.

11. The system according to claim 9 , wherein an iterative process including more images improves the fused image that meets a threshold such that the selected image is high correlated to the image to be recovered.

12. The system according to claim 1 , wherein a user interface in communication with the processor and the computer readable memory, acquires and stores the set of multi-angled view images in the computer readable memory upon receiving an input from a surface of the user interface by a user.

13. A method for image processing of high definition (HD) images of a three dimensional (3D) scene, comprising:
acquiring, by a processor, a set of multi-angled view HD images of the 3D scene generated by sensors either by an input interface or from a computer readable memory, in communication with the processor, such that each multi-angled view HD image includes pixels, and at least one multi-angled view HD image includes an occluded area in at least a portion of the 3D scene, resulting in missing pixels;
aligning, by the processor, at least three multi-angled view HD images in the set of multi-angled view HD images to a target view angle of the 3D scene, to form a set of aligned multi-angled view HD images representing a target point of view of the 3D scene, such that at least one aligned multi-angled view HD image of the at least three multi-angled view HD images, includes missing pixels due to the occluded area,
forming, by the processor, a matrix using vectorized aligned multi-angled view HD images, such that the matrix is incomplete due to the missing pixels; and
completing, by the processor, the matrix using a matrix completion to combine the aligned multi-angled view HD images to produce a fused HD image of the 3D scene without the occluded area.

14. The method according to claim 13 , wherein the aligning of the at least three multi-angled view images to the target point of view of the 3D scene is based on an epipolar geometry of the 3D scene.

15. The method according to claim 13 , wherein the matrix completion is a low-rank matrix completion, such that each column of the low-rank matrix completion corresponds to a vectorized aligned multi-angled view image and the missing pixels of the at least one aligned multi-angled view image corresponds to the clouded area.

16. The method according to claim 13 , wherein the sensors are moving and arranged in a satellite or an airplane.

17. A non-transitory computer readable storage medium embodied thereon a program executable by a computer for performing a method, the method for image processing of images of a scene, comprising:
acquiring, by a processor, a set of multi-angled view images of the scene generated by sensors either by an input interface or from a computer readable memory, in communication with the processor, such that each image includes pixels, and at least one image includes an occluded area in at least a portion of the scene, resulting in missing pixels;
aligning, by the processor, at least three images in the set of images to a target view angle of the scene, to form a set of aligned images representing a target point of view of the scene, such that at least one aligned image of the at least three images, includes missing pixels due to the occluded area,
forming, by the processor, a matrix using vectorized aligned images, such that the matrix is incomplete due to the missing pixels; and
completing, by the processor, the matrix using a matrix completion to combine the aligned images to produce a fused image of the scene without the occluded area.

18. The method according to claim 17 , wherein the scene includes occlusions or the clouded area caused by clouds between the sensors and scene and structures in the scene.

19. The method according to claim 17 , wherein the aligning of the multi-angled view images to the target view angle of the scene forms the set of aligned multi-angled view images representing the target point of view of the scene is based on a fundamental matrix.

20. The method according to claim 19 , wherein the fundamental matrix is estimated by key points in the multi-angled images, and the key points are based on SIFT Matching.

Assignees

• Mitsubishi Electric Res Laboratories Inc

Inventors

• Liu Dehong
• Ma Yanting
• Mansour Hassan
• Kamilov Ulugbek
• Taguchi Yuichi
• Boufounos Petros
• Vetro Anthony

Classifications

• H04N13/204

  using stereoscopic image cameras (stereoscopic photography G03B35/00) · CPC title

• H04N13/156 Primary

  Mixing image signals · CPC title

• H04N5/2628 Primary

  Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation · CPC title

• G06T1/00

  General purpose image data processing · CPC title

• G06T2207/30192

  Weather; Meteorology · CPC title