Systems and methods for image processing based on optimal transport and epipolar geometry

Patent metadata

Field	Value
Publication number	US-11842504-B2
Application number	US-202117401577-A
Country	US
Kind code	B2
Filing date	Aug 13, 2021
Priority date	Aug 13, 2021
Publication date	Dec 12, 2023
Grant date	Dec 12, 2023

Abstract

Systems and methods for image processing for determining a registration map between a first image of a scene with a second image of the scene, include solving an optimal transport (OT) problem to produce the registration map by optimizing a cost function that determines a minimum of a ground cost distance between the first and the second images modified with an epipolar geometry-based regularizer including a distance that quantifies the violation of an epipolar geometry constraint between corresponding points defined by the registration map. The ground cost compares a ground cost distance of features extracted within the first image with a ground cost distance of features extracted from the second image.

First claim

What is claimed is:

1. An image processing system for determining a registration map between a first image of a scene with a second image of the scene, comprising: at least one processor; and a memory having instructions stored thereon that, when executed by the at least one processor, cause the image processing system to:
solve an optimal transport (OT) problem to produce the registration map by optimizing a cost function that determines a minimum of a ground cost between the first and the second images modified with an epipolar geometry-based regularizer including a distance that quantifies the violation of an epipolar geometry constraint between corresponding points defined by the registration map;
wherein the ground cost compares a ground cost distance of features extracted within the first image with a ground cost distance of features extracted from the second image.

2. The image processing system of claim 1 , wherein the system is further configured to register the first image and the second image according to the registration map.

3. The image processing system of claim 1 , wherein the ground cost distance of features within the first image is determined based on a similarity measure between pair-wise difference between pixels in the first image and the ground cost distance of features within the second image is determined based on a similarity measure between pair-wise difference between pixels in the second image.

4. The image processing system of claim 3 , wherein the similarity measure between pair-wise difference between pixels in the first image and the similarity measure between pair-wise difference between pixels in the second image are each defined as per Gromov-Wasserstein notion.

5. The image processing system of claim 1 , wherein solving the OT problem comprises determining an OT distance as a Gromov-Wasserstein (GW) distance between vectors of features of the scene in the first and second images.

6. The image processing system of claim 5 , wherein the first and the second images are aerial images with pixel resolutions showing differences in elevation from a ground of different objects in the scene.

7. The image processing system of claim 6 , wherein the epipolar geometry-based regularizer is a function of a fundamental matrix between the first and second images.

8. The image processing system of claim 7 , wherein the epipolar geometry-based regularizer includes a Sampson discrepancy.

9. The image processing system of claim 6 , wherein the processor is further configured to fuse the first and the second images according to the registration map to output a fused image.

10. The image processing system of claim 9 , wherein a modality of the first image is different from a modality of the second image.

11. The image processing system of claim 10 , wherein the modality of the first and the second image is selected from a group consisting of optical color images, optical gray-scale images, depth images, infrared images, and synthetic aperture radar (SAR) images.

12. The image processing system of claim 5 , wherein the first and the second images comprise non-universal features of the scene, and wherein feature vectors of the non-universal features in the first image and feature vectors of the non-universal features in the second image are not defined in a common space.

13. The image processing system of claim 5 , wherein the feature vectors corresponding to the first image and the second image comprise one or more of pixel coordinates and 3-channel intensity values of respective one of the first image and the second image.

14. The image processing system of claim 1 , wherein the OT problem is a function of a cross-image cost matrix of universal features of the first and second images, and wherein feature vectors of the universal features in the first image and feature vectors of the universal features in the second image are defined in a common space.

15. The image processing system of claim 1 , wherein the first and the second images are part of a set of multi-angled view images of the scene generated by sensors, such that each multi-angled view image includes pixels, and at least one multi-angled view image includes a clouded area in at least a portion of the scene, resulting in missing pixels, wherein the processor is further configured to:
align the multi-angled view images based on the registration map to a target view angle of the scene, to form a set of aligned multi-angled view images representing a target point of view of the scene, such that at least one aligned multi-angled view image of the at least three multi-angled view images, has missing pixels due to the clouded area;
form a matrix using vectorized aligned multi-angled view images, wherein the matrix is incomplete due to the missing pixels; and
complete the matrix using a matrix completion to combine the aligned multi-angled view images to produce a fused image of the scene without the clouded area.

16. The image processing system according to claim 15 , wherein the scene is a three dimensional (3D) scene and each multi-angled view image of the set of multi-angled view images are one of taken at the same or different time with unknown sensors positions relative to the 3D scene.

17. The image processing system according to claim 16 , wherein the matrix completion is a low-rank matrix completion, such that each column of the low-rank matrix completion corresponds to a vectorized aligned multi-angled view image and the missing pixels of the at least one aligned multi-angled view image corresponds to the clouded area.

18. The image processing system according to claim 15 , wherein the sensors are movable during the acquisition of the multi-angled view images.

19. The image processing system according to claim 18 , wherein the sensors are arranged in a satellite or an airplane.

20. An image processing method for determining a registration map between a first image of a scene with a second image of the scene, comprising:
solving by a processor, an optimal transport (OT) problem to produce the registration map by optimizing a cost function that determines a minimum of a ground cost between the first and the second images modified with an epipolar geometry-based regularizer including a distance that quantifies the violation of an epipolar geometry constraint between corresponding points defined by the registration map,
wherein the ground cost compares a ground cost distance of features extracted within the first image with a ground cost distance of features extracted from the second image.

Assignees

• Mitsubishi Electric Res Laboratories Inc

Inventors

• Ma Yanting
• Liu Dehong
• Boufounos Petros
• Orlik Philip

Classifications

• G06T7/337 Primary

  involving reference images or patches · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06T2207/10012

  Stereo images · CPC title

• G06T2207/10024

  Color image · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title