Method and system for multi-sensor fusion using transform learning

Patent metadata

Field	Value
Publication number	US-12499388-B2
Application number	US-202117407219-A
Country	US
Kind code	B2
Filing date	Aug 20, 2021
Priority date	Aug 21, 2020
Publication date	Dec 16, 2025
Grant date	Dec 16, 2025

Abstract

This disclosure relates to multi-sensor fusion using Transform Learning (TL) that provides a compact representation of data in many scenarios as compared to Dictionary Learning (DL) and Deep network models that may be computationally intensive and complex. A two-stage approach for better modeling of sensor data is provided, wherein in the first stage, representation of the individual sensor time series is learnt using dedicated transforms and their associated coefficients and in the second stage, all the representations are fused together using a fusing (common) transform and its associated coefficients to effectively capture correlation between the different sensor representations for deriving an inference. The method and system of the present disclosure can find application in areas employing multiple sensors that are mostly heterogeneous in nature.

First claim

What is claimed is:

1 . A processor implemented method comprising the steps of:
receiving, via one or more hardware processors, a plurality of training data (X 1 , X 2 , . . . X n ) from a plurality of sensors connected to a monitored system with a training output (y); performing, via the one or more hardware processors, a joint optimization of a set of parameters including (i) sensor specific transforms (T 1 , T 2 , . . . T n ) and (ii) sensor specific coefficients (Z 1 , Z 2 , . . . Z n ), wherein each of the sensor specific transforms and the sensor specific coefficients correspond to a training data in the plurality of training data (X 1 , X 2 . . . X n ), (iii) a fusing transform (T (f) ), (iv) a fusing coefficient (Z (f) ), and (v) a weight matrix (w), and wherein the joint optimization comprises:
initializing the sensor specific transforms (T 1 T 2 , . . . T n ) and the fusing transform (T (f) ) with a random matrix comprising real numbers between 0 and 1;
estimating the sensor specific coefficients (Z 1 , Z 2 , . . . Z n ) based on the initialized sensor specific transforms (T 1 , T 2 , . . . T n ) and a corresponding training data from the plurality of training data (X 1 , X 2 . . . X n );
estimating the fusing coefficient (Z (f) ) based on the initialized fusing transform (T (f) ) and the estimated sensor specific coefficients (Z 1 , Z 2 , . . . Z n );
estimating the weight matrix (w) based on the training output (y) and the estimated fusing coefficient (Z (f) ); and
iteratively performing joint learning using the initialized parameters and the estimated parameters from the set of parameters, in a first iteration and learnt parameters thereafter until a termination criterion is met, the joint learning comprising:
learning each of the sensor specific transforms (T 1 , T 2 , . . . T n ) based on a corresponding sensor specific coefficient (Z 1 , Z 2 , . . . Z n ) and the plurality of training data (X 1 , X 2 , . . . X n );
learning each of the sensor specific coefficients (Z 1 , Z 2 , . . . Z n ) based on the fusing transform (T (f) ), a corresponding sensor specific transform (T 1 , T 2 , . . . T n ), the fusing coefficient (Z (f) ) and a corresponding training data from the plurality of training data (X 1 , X 2 , . . . X n ), and remaining of the sensor specific coefficients (Z 1 , Z 2 , . . . Z n );
learning the fusing transform (T (f) ) based on the sensor specific coefficient (Z 1 , Z 2 , . . . Z n ) and the fusing coefficient (Z (f) );
learning the fusing coefficient (Z (f) ) based on the fusing transform (T (f) ), the sensor specific coefficient (Z 1 , Z 2 , . . . Z n ), the weight matrix (w) and the training output (y); and
learning the weight matrix (w) based on the fusing coefficient (Z (f) ) and the training output (y);

wherein the termination criterion is one of (i) completion of a predefined number of iterations (Maxiter) and (ii) difference of the fusing transform (T (f) ) of a current iteration and the fusing transform (T (f) ) of a previous iteration being less than an empirically determined threshold value (Tol);

to obtain jointly (i) the learnt sensor specific transforms (T 1 , T 2 , . . . T n ), (ii) the learnt fusing transform (T (f) ) and (iii) the learnt weight matrix (w) for the monitored system being sensed by the plurality of sensors; and
estimating, via the one or more hardware processors, an output (y new ) of the monitored system for a plurality of new data (x 1 , x 2 , . . . x n ), wherein the new data is a time series data and estimating the output represents application of the method to the monitored system using the new data which is different from the training data (X 1 , X 2 , . . . X n ), wherein the monitored system is a Friction Stir Welding (FSW) machine, wherein the output (y) is a value representing an Ultimate Tensile Strength (UTS) indicative of the quality of the weld performed by the FSW machine,
wherein the step of estimating the output (y new ) of the FSW machine comprises:
receiving the plurality of new data (x 1 , x 2 , . . . x n ) from the plurality of sensors connected to the FSW machine, wherein the sensors measure parameters pertaining to a force, a torque, and a power for a welding process implemented in real-time;
estimating the sensor specific coefficients (z 1 , z 2 , . . . z n ) corresponding to the plurality of new data (x 1 , x 2 , . . . x n ) using the received plurality of new data (x 1 , x 2 , . . . x n ) and the learnt sensor specific transforms (T 1 , T 2 , . . . T n ), wherein the sensor specific transforms are learnt to represent corresponding sensor data;
estimating a new fusing coefficient z (f) using the learnt fusing transform (T (f) ) and the estimated sensor specific coefficients (z 1 , z 2 . . . z n ); and
estimating the output (y new ) for the FSW machine based on the learnt weight matrix (w) and the estimated new fusing coefficient z (f) .

2 . The processor implemented method of claim 1 , wherein the joint optimization is represented as

min

T
1

⁢

T
2

,
…

,

T
n

,

T

(
f
)

⁢

Z
1

,

Z
2

,
…

,

Z
n

,

Z

(
f
)

,
w



T
1

⁢

X
1

-

Z
1



F
2

+



T
2

⁢

X
2

Assignees

• Tata Consultancy Services Ltd
• Indian Institute Of Tech Kharagpur

Inventors

• Chandra Mariswamy Girish
• Kumar Achanna Anil
• Kumar Kriti
• Majumdar Angshul
• Mishra Debasish
• PAL Surjya Kanta

Classifications

• G06F17/16

  Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)} · CPC title

• G05B23/0245

  based on a qualitative model, e.g. rule based; if-then decisions · CPC title

• B23K20/123

  Controlling or monitoring the welding process · CPC title

• G06N20/20 Primary

  Ensemble learning · CPC title

• G06N20/00 Primary

  Machine learning · CPC title