System and method for multi-modal image enhancement based on time-adjacent image data

Patent metadata

Field	Value
Publication number	US-12387306-B2
Application number	US-202318109240-A
Country	US
Kind code	B2
Filing date	Feb 13, 2023
Priority date	Feb 13, 2023
Publication date	Aug 12, 2025
Grant date	Aug 12, 2025

Abstract

According to one aspect of the present disclosure, a method of image enhancement is provided. The method may include receiving, by at least one processor, a query for an enhanced image of a queried region obtained by a remote sensor at a queried time. The method may include obtaining, by the at least one processor, an image dataset associated with the queried region from the remote sensor. In response to the image dataset including a plurality of image data associated with the queried region obtained by the remote sensor, the method may include generating, by the at least one processor, the enhanced image of the queried region using an intra-modal regression model.

First claim

What is claimed is:

1. A method of image enhancement, comprising:
receiving, by at least one processor, a query for an enhanced image of a queried region obtained by a remote sensor at a queried time;
obtaining, by the at least one processor, an image dataset associated with the queried region from the remote sensor;
in response to the image dataset including a plurality of image data associated with the queried region obtained by the remote sensor, generating, by the at least one processor, the enhanced image of the queried region using an intra-modal regression model, comprising generating a model pattern based on a first timestamp of the queried time and a second timestamp for each image associated with the queried region;
accessing, by the at least one processor, a training database that includes a plurality of training data images of various regions obtained using different remote sensors;
performing, by the at least one processor, a first training data search associated with the first timestamp of the queried time, as well as the remote sensor and the queried region, and a second training data search associated with the second timestamp for each image in the image dataset associated with the queried region and the remote sensor; and
selecting, by the at least one processor, a training dataset based on the first training data search and the second training data search associated with the queried region.

2. The method of claim 1 , further comprising:
identifying, by the at least one processor, whether any images in the training dataset includes one or more occluded images of the queried region; and
discarding, by the at least one processor, the one or more occluded images of the queried region from the training dataset.

3. The method of claim 1 , further comprising:
generating, by at least one processor, the intra-modal regression model based on the first training data search and the second training data search associated with the queried region.

4. The method of claim 3 , further comprising:
inputting, by the at least one processor, the image dataset associated with the queried region from the remote sensor into the intra-modal regression model; and
receiving, by the at least one processor, the enhanced image of the queried region associated with the queried time and associated with the remote sensor based on an output of the intra-modal regression model.

5. The method of claim 1 , further comprising:
in response to the image dataset including a plurality of image data associated with the queried region obtained by different remote sensors, generating, by the at least one processor, the enhanced image of the queried region using an inter-modal generation model.

6. The method of claim 5 , further comprising:
accessing, by the at least one processor, a training database that includes a plurality of training data images of various regions obtained using the different remote sensors;
performing, by the at least one processor, a training data search for a training data set associated with the queried region captured at any time by any of the different remote sensors; and
selecting, by the at least one processor, a training dataset that includes training data of the queried region captured at different time by the different remote sensors,
wherein the training dataset is not associated with a modality of the remote sensor of the query.

7. The method of claim 6 , further comprising:
inputting, by the at least one processor, the training dataset into the inter-modal generation model, the inter-modal generation model including one or more convolutional layers or residual blocks; and
receiving, by the at least one processor, the enhanced image of the queried region associated with the queried time and associated with a modality of the remote sensor from the query as an output of the inter-modal generation model.

8. The method of claim 7 , wherein:
the training dataset is associated with a first modality or a second modality,
in response to the inter-modal generation model receiving the training dataset associated with the first modality, generating, by the at least one processor, the enhanced image of the second modality of the queried time and region as an output of the inter-modal generation model, and
in response to the inter-modal generation model receiving the training dataset associated with the second modality, generating, by the at least one processor, the enhanced image of the first modality of the queried time and region as the output of the inter-modal generation model.

9. An apparatus for image enhancement, comprising:
at least one processor;
a memory storing instructions which, when executed by the at least one processor, cause the at least one processor to perform:
receiving a query for an enhanced image of a queried region obtained by a remote sensor at a queried time;
obtaining an image dataset associated with the queried region from the remote sensor;
in response to the image dataset including a plurality of image data associated with the queried region obtained by the remote sensor, generating the enhanced image of the queried region using an intra-modal regression model, comprising generating a model pattern based on a first timestamp of the queried time and a second timestamp for each image associated with the queried region;
accessing a training database that includes a plurality of training data images of various regions obtained using different remote sensors;
performing a first training data search associated with the first timestamp of the queried time, as well as the remote sensor and the queried region, and a second training data search associated with the second timestamp for each image in the image dataset associated with the queried region and the remote sensor; and
selecting a training dataset based on the first training data search and the second training data search associated with the queried region.

10. The apparatus of claim 9 , wherein the memory storing instructions which, when executed by the at least one processor, cause the at least one processor to perform:
identifying whether any images in the training dataset includes one or more occluded images of the queried region; and
discarding the one or more occluded images of the queried region from the training dataset.

11. The apparatus of claim 9 , wherein the memory storing instructions which, when executed by the at least one processor, cause the at least one processor to perform:
generating the intra-modal regression model based on the first training data search and the second training data search associated with the queried region.

12. The apparatus of claim 11 , wherein the memory storing instructions which, when executed by the at least one processor, cause the at least one processor to perform:
inputting the image dataset associated with the queried region from the remote sensor into the intra-modal regression model; and
receiving the enhanced image of the queried region associated with the queried time and associated with the remote sensor based on an output of the intra-modal regression model.

13. The apparatus of claim 9 , wherein the memory storing instructions which, when executed by the at least one processor, cause the at least one processor to perform:
in response to the image dataset including a plurality of image data associated with the queried region obtained by different remote sensors, generating the enhanced image of the queried region using an inter-modal generation model;
accessing a training database that includes a plurality of training data images of various regions obtained using the different re

Assignees

• Ping An Tech Shenzhen Co Ltd

Inventors

• Peng Bo
• Lai Jui-Hsin
• Gou Yuchuan
• Zhou Hang
• Liu Hongchen
• Han Mei

Classifications

• G06V10/26

  Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion · CPC title

• G06F16/532 Primary

  Query formulation, e.g. graphical querying · CPC title

• G06F16/583

  using metadata automatically derived from the content · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/10032

  Satellite or aerial image; Remote sensing · CPC title