System, method, and storage medium for distributed joint manifold learning based heterogeneous sensor data fusion

Patent metadata

Field	Value
Publication number	US-12505380-B2
Application number	US-202117563014-A
Country	US
Kind code	B2
Filing date	Dec 27, 2021
Priority date	Jan 23, 2018
Publication date	Dec 23, 2025
Grant date	Dec 23, 2025

Abstract

The present disclosure provide a system, a method, and a storage medium for distributed joint manifold learning (DJML) based heterogeneous sensor data fusion. The system includes a plurality of nodes; and each node includes at least one camera; one or more sensors; at least one memory configured to store program instructions; and at least one processor, when executing the program instructions, configured to obtain heterogeneous sensor data from the one or more sensors to form a joint manifold; determine one or more optimum manifold learning algorithms by evaluating a plurality of manifold learning algorithms based on the joint manifold; compute a contribution of the node based on the one or more optimum manifold learning algorithms; update a contribution table based on the contribution of the node and contributions received from one or more neighboring nodes; and broadcast the updated contribution table to the one or more neighboring nodes.

First claim

What is claimed is:

1 . A system for distributed joint manifold learning (DJML) based heterogeneous sensor data fusion, comprising:
a plurality of nodes,
each node comprising:
at least one camera;
one or more sensors;
at least one memory, configured to store program instructions; and
at least one processor, coupled with the at least one memory and, when executing the program instructions, configured to:
obtain heterogeneous sensor data from two or more sensors among the plurality of nodes to form a joint manifold;
determine one or more optimum manifold learning algorithms by evaluating a plurality of manifold learning algorithms based on the joint manifold;
compute a contribution of the node based on the one or more optimum manifold learning algorithms;
update a contribution table based on the contribution of the node and contributions received from one or more neighboring nodes among the plurality of nodes; and
broadcast the updated contribution table that includes contributions of the node and the one or more neighboring nodes, to the one or more neighboring nodes, wherein:
 the one or more optimum manifold learning algorithms is determined by:
 performing the plurality of manifold learning algorithms to process the joint manifold to generate raw manifold learning results;
 processing the raw manifold learning results to generate intrinsic parameters; and
 determining the one or more optimum manifold learning algorithms by evaluating the plurality of manifold learning algorithms based on the raw manifold learning results and the intrinsic parameters, wherein evaluating the plurality of manifold learning algorithms further includes:
 selecting a subset of the plurality of manifold learning algorithms as one or more candidate manifold learning algorithms by evaluating the raw manifold learning results; and
 selecting the one or more optimum manifold learning algorithms from the one or more candidate manifold learning algorithms by evaluating the intrinsic parameters.

2 . The system according to claim 1 , further comprising: a ground station, wherein a node of the plurality of nodes includes an unmanned aerial vehicle (UAV).

3 . The system according to claim 1 , wherein:
the at least one camera includes one or more infrared (IR) cameras; and the one or more sensors include one or more of an image sensor, an IR sensor, and a radio frequency sensor.

4 . The system according to claim 3 , wherein:
the image sensor is configured to process local camera information of the node to estimate state information of the node.

5 . The system according to claim 1 , wherein:
a pre-defined synchronization time is transmitted to the plurality of nodes.

6 . The system according to claim 1 , wherein the one or more optimum manifold learning algorithms is determined by:
performing the plurality of manifold learning algorithms to process the joint manifold to generate raw manifold learning results; processing the raw manifold learning results to generate intrinsic parameters; and determining the one or more optimum manifold learning algorithms by evaluating the plurality of manifold learning algorithms based on the raw manifold learning results and the intrinsic parameters.

7 . A method for distributed joint manifold learning (DJML) based heterogeneous sensor data fusion, performed by each node of a plurality of nodes in a system, each node comprising at least one camera, one or more sensors, and at least one processor configured to performing:
obtaining heterogeneous sensor data from two or more sensors among the plurality of nodes to form a joint manifold; determining one or more optimum manifold learning algorithms by evaluating a plurality of manifold learning algorithms based on the joint manifold; computing a contribution of the node based on the one or more optimum manifold learning algorithms; updating a contribution table based on the contribution of the node and contributions received from one or more neighboring nodes among the plurality of nodes; and broadcasting the updated contribution table that includes contributions of the node and the one or more neighboring nodes, to the one or more neighboring nodes, wherein:
the one or more optimum manifold learning algorithms is determined by:
performing the plurality of manifold learning algorithms to process the joint manifold to generate raw manifold learning results;
processing the raw manifold learning results to generate intrinsic parameters; and
determining the one or more optimum manifold learning algorithms by evaluating the plurality of manifold learning algorithms based on the raw manifold learning results and the intrinsic parameters, wherein evaluating the plurality of manifold learning algorithms further includes:
selecting a subset of the plurality of manifold learning algorithms as one or more candidate manifold learning algorithms by evaluating the raw manifold learning results; and
selecting the one or more optimum manifold learning algorithms from the one or more candidate manifold learning algorithms by evaluating the intrinsic parameters.

8 . The method according to claim 7 , wherein: the system further includes a ground station, wherein a node of the plurality of nodes includes an unmanned aerial vehicle (UAV).

9 . The method according to claim 7 , wherein:
the at least one camera includes one or more infrared (IR) cameras; and the one or more sensors include one or more of an image sensor, an IR sensor, and a radio frequency sensor.

10 . The method according to claim 9 , wherein:
the image sensor is configured to process local camera information of the node to estimate state information of the node.

11 . The method according to claim 7 , wherein:
a pre-defined synchronization time is transmitted to the plurality of nodes.

12 . The method according to claim 7 , wherein the one or more optimum manifold learning algorithms is determined by:
performing the plurality of manifold learning algorithms to process the joint manifold to generate raw manifold learning results; processing the raw manifold learning results to generate intrinsic parameters; and determining the one or more optimum manifold learning algorithms by evaluating the plurality of manifold learning algorithms based on the raw manifold learning results and the intrinsic parameters.

13 . A non-transitory computer-readable storage medium, containing program instructions for, when being executed by a processor of each node of a plurality of nodes in a system, performing a method for distributed joint manifold learning (DJML) based heterogeneous sensor data fusion, and the method comprising:
obtaining heterogeneous sensor data from two or more sensors among the plurality of nodes to form a joint manifold; determining one or more optimum manifold learning algorithms by evaluating a plurality of manifold learning algorithms based on the joint manifold; computing a contribution of the node based on the one or more optimum manifold learning algorithms; updating a contribution table based on the contribution of the node and contributions received from one or more neighboring nodes among the plurality of nodes; and broadcasting the updated contribution table to the one or more neighboring nodes, wherein: the one or more optimum manifold learning algorithms is determined by: performing the plurality of manifold learning algorithms to process the joint manifold to generate raw manifold learning results; processing the raw manifold learning results to generate intrinsic parameters; and determining the one or more optimum manifold learning algorithms

Assignees

• Intelligent Fusion Tech Inc

Inventors

• Shen Dan
• Zulch Peter
• Disasio Marcello
• Blasch Erik
• Chen Genshe

Classifications

• H04N23/20

  for generating image signals from infrared radiation only · CPC title

• G06F18/251

  of input or preprocessed data · CPC title

• G06F18/25

  Fusion techniques · CPC title

• G06F18/21355

  nonlinear criteria, e.g. embedding a manifold in a Euclidean space · CPC title

• G06V20/13

  Satellite images · CPC title