Methods, systems and media for joint manifold learning based heterogenous sensor data fusion

Patent metadata

Field	Value
Publication number	US-11210570-B2
Application number	US-201815878188-A
Country	US
Kind code	B2
Filing date	Jan 23, 2018
Priority date	Jan 23, 2018
Publication date	Dec 28, 2021
Grant date	Dec 28, 2021

Abstract

The present disclosure provides a method for joint manifold learning based heterogenous sensor data fusion, comprising: obtaining learning heterogeneous sensor data from a plurality sensors to form a joint manifold, wherein the plurality sensors include different types of sensors that detect different characteristics of targeting objects; performing, using a hardware processor, a plurality of manifold learning algorithms to process the joint manifold to obtain raw manifold learning results, wherein a dimension of the manifold learning results is less than a dimension of the joint manifold; processing the raw manifold learning results to obtain intrinsic parameters of the targeting objects; evaluating the multiple manifold learning algorithms based on the raw manifold learning results and the intrinsic parameters to determine one or more optimum manifold learning algorithms; and applying the one or more optimum manifold learning algorithms to fuse heterogeneous sensor data generated by the plurality sensors.

First claim

What is claimed is:

1. A method for joint manifold learning based heterogenous sensor data fusion, comprising:
obtaining learning heterogeneous sensor data from a plurality sensors to form a joint manifold, wherein the plurality sensors include different types of sensors that detect different characteristics of targeting objects;
performing, using a hardware processor, a plurality of manifold learning algorithms to process the joint manifold to obtain raw manifold learning results, wherein a dimension of the manifold learning results is less than a dimension of the joint manifold;
processing the raw manifold learning results to obtain intrinsic parameters of the targeting objects;
evaluating the multiple manifold learning algorithms based on the raw manifold learning results and the intrinsic parameters to determine one or more optimum manifold learning algorithms, wherein evaluating the multiple manifold learning algorithms includes:
selecting a subset of the multiple manifold learning algorithms as one or more candidate manifold learning algorithms by evaluating the raw manifold learning results; and
selecting the one or more optimum manifold learning algorithms from the one or more candidate manifold learning algorithms by evaluating the intrinsic parameters; and

applying the one or more optimum manifold learning algorithms to fuse heterogeneous sensor data generated by the plurality sensors.

2. The method of claim 1 , wherein the plurality sensors at least include a medium wavelength infrared camera and three radio frequency Doppler sensors.

3. The method of claim 1 , wherein the dimension of the joint manifold is at least seven.

4. The method of claim 1 , wherein the plurality of manifold learning algorithms at least include two of:
maximally collapsing metric learning, neighborhood preserving embedding, Isomap, locally linear embedding, Hessian locally linear embedding, Laplacian Eigenmaps, diffusion maps, and local tangent space alignment.

5. The method of claim 1 , wherein performing the plurality of manifold learning algorithms to process the joint manifold includes:
generating, for each of the plurality of manifold learning algorithms, a dimensionality reduction matrix to reduce the dimension of the joint manifold.

6. The method of claim 1 , wherein processing the raw manifold learning results includes:
performing a line regression to the raw manifold learning results for each of the plurality of manifold learning algorithms.

7. The method of claim 6 , wherein performing the line regression includes: generating, for each of the plurality of manifold learning algorithms, a rotating and zooming matrix, and a shifting matrix to lineally transform the raw manifold learning results to obtain the intrinsic parameters.

8. The method of claim 1 , wherein the intrinsic parameters of the targeting objects at least include a position parameter and a velocity parameter of the targeting objects.

9. A system for joint manifold learning based heterogenous sensor data fusion, the system comprising:
a hardware processor; and
a memory storing instructions that, when executed by the hardware processor, cause the hardware processor to:
obtain learning heterogeneous sensor data from a plurality sensors to form a joint manifold, wherein the plurality sensors include different types of sensors that detect different characteristics of targeting objects,
perform a plurality of manifold learning algorithms to process the joint manifold to obtain raw manifold learning results, wherein a dimension of the manifold learning results is less than a dimension of the joint manifold,
process the raw manifold learning results to obtain intrinsic parameters of the targeting objects,
evaluate the multiple manifold learning algorithms based on the raw manifold learning results and the intrinsic parameters to determine one or more optimum manifold learning algorithms, wherein the instructions further cause the hardware processor to:
select a subset of the multiple manifold learning algorithms as one or more candidate manifold learning algorithms by evaluating the raw manifold learning results; and
select the one or more optimum manifold learning algorithms from the one or more candidate manifold learning algorithms by evaluating the intrinsic parameters; and

apply the one or more optimum manifold learning algorithms to fusing heterogeneous sensor data generated by the plurality sensors.

10. The system of claim 9 , wherein the plurality sensors at least include a medium wavelength infrared camera and three radio frequency Doppler sensors.

11. The system of claim 9 , wherein the dimension of the joint manifold is at least seven.

12. The system of claim 9 , wherein the plurality of manifold learning algorithms at least include two of:
maximally collapsing metric learning, neighborhood preserving embedding, Isomap, locally linear embedding, Hessian locally linear embedding, Laplacian Eigenmaps, diffusion maps, and local tangent space alignment.

13. The system of claim 9 , wherein the instructions further cause the hardware processor to:
generate, for each of the plurality of manifold learning algorithms, a dimensionality reduction matrix to reduce the dimension of the joint manifold.

14. The system of claim 9 , wherein the instructions further cause the hardware processor to:
perform a line regression to the raw manifold learning results for each of the plurality of manifold learning algorithms.

15. The system of claim 14 , wherein the instructions further cause the hardware processor to:
generate, for each of the plurality of manifold learning algorithms, a rotating and zooming matrix, and a shifting matrix to lineally transform the raw manifold learning results to obtain the intrinsic parameters.

16. The system of claim 9 , wherein the intrinsic parameters of the targeting objects at least include a position parameter and a velocity parameter of the targeting objects.

17. A non-transitory computer-readable medium containing computer-executable instructions that, when executed by a hardware processor, cause the hardware processor to perform a method for joint manifold learning based heterogenous sensor data fusion, the method comprising:
obtaining learning heterogeneous sensor data from a plurality sensors to form a joint manifold, wherein the plurality sensors include different types of sensors that detect different characteristics of targeting objects;
performing, using a hardware processor, a plurality of manifold learning algorithms to process the joint manifold to obtain raw manifold learning results, wherein a dimension of the manifold learning results is less than a dimension of the joint manifold;
processing the raw manifold learning results to obtain intrinsic parameters of the targeting objects;
evaluating the multiple manifold learning algorithms based on the raw manifold learning results and the intrinsic parameters to determine one or more optimum manifold learning algorithms, wherein evaluating the multiple manifold learning algorithms includes:
selecting a subset of the multiple manifold learning algorithms as one or more candidate manifold learning algorithms by evaluating the raw manifold learning results; and
selecting the one or more optimum manifold learning algorithms from the one or more candidate manifold learning algorithms by evaluating the intrinsic parameters; and

applying the one or more optimum manifold learning algorithms to fuse heterogeneous sensor data generated by the plurality sensors.

Assignees

• Intelligent Fusion Tech Inc

Inventors

• Shen Dan
• Zulch Peter
• Disasio Marcello
• Blasch Erik
• Chen Genshe
• Wang Zhonghai
• Lu Jingyang

Classifications

• G01S13/87 Primary

  Combinations of radar systems, e.g. primary radar and secondary radar · CPC title

• G06V10/80

  Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level (multimodal speaker identification or verification G10L17/10) · CPC title

• G06V10/7715

  Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods · CPC title

• G06F18/25

  Fusion techniques · CPC title

• G06F18/21355

  nonlinear criteria, e.g. embedding a manifold in a Euclidean space · CPC title