Spectral data augmentation for single domain generalization

Patent metadata

Field	Value
Publication number	US-12525006-B2
Application number	US-202318230556-A
Country	US
Kind code	B2
Filing date	Aug 4, 2023
Priority date	Aug 4, 2023
Publication date	Jan 13, 2026
Grant date	Jan 13, 2026

Abstract

A machine learning model is trained using original source domain data through empirical risk minimization and a model sensitivity map is computed. Each sensitive frequency point on the model sensitivity map is targeted. An adversarial technique is employed to generate spectral adversarial images based on the model sensitivity map and an image amplitude spectrum is augmented. The generated spectral adversarial images are mixed with the original source domain data to finetune the machine learning model and deployment of the finetuned machine learning model is facilitated.

First claim

What is claimed is:

1 . A method comprising:
training, using at least one hardware device, a machine learning model using original source domain data through empirical risk minimization; computing, using the at least one hardware device, a model sensitivity map; targeting, using the at least one hardware device, each sensitive frequency point on the model sensitivity map; employing, using the at least one hardware device, an adversarial technique to generate spectral adversarial images based on the model sensitivity map and augmenting an image amplitude spectrum; mixing, using the at least one hardware device, the generated spectral adversarial images with the original source domain data to finetune the machine learning model; and facilitating deployment of the finetuned machine learning model.

2 . The method of claim 1 , wherein the model sensitivity map is a surrogate of model vulnerability in a frequency space.

3 . The method of claim 1 , wherein the employing operation encodes model sensitivity into the spectral adversarial images.

4 . The method of claim 1 , wherein the step of computing the model sensitivity map uses a source domain amplitude spectrum as a domain prior to enhance the model sensitivity map.

5 . The method of claim 1 , further comprising performing inferencing in one or more preferred domains using the deployed finetuned machine learning model.

6 . The method of claim 1 , further comprising repeating the computing, targeting, employing, and mixing operations using the finetuned machine learning model in place of the machine learning model.

7 . The method of claim 1 , wherein the employing of the adversarial technique to generate the spectral adversarial images comprises:
computing a mean amplitude spectrum D by averaging an amplitude spectrum of all images in a source domain; reformulating an original Fourier basis noise N i,j as defined by
N i,j =r·D i,j ·U i,j ; and
computing an enhanced model sensitivity at frequency (i,j) by evaluating a prediction error rate on the spectral adversarial images as defined by

M
S

(

i
,
j

)

=

1
-

Acc

(

x
,
y

)

∈

X
S

⁢

(

F
⁡
(

x
+

r
·

D

i
,
j

·

U

i
,
j

,
y

)

)

,

 where F is a model trained with empirical risk minimization (ERM) by minimizing a cross entropy loss

ℒ
ERM

=

1
-

𝔼

(

x
,
y

)

∈

X
S

⁢

ℓ
CE

(

F
⁡
(
x
)

,
y

)

,

 where U i,j is a Fourier basis image, r is a randomly sampled integer, and X S is a whole dataset.

8 . The method of claim 1 , wherein the employing of the adversarial technique to generate the spectral adversarial images comprises computing an original spectral amplitude A org and a phase P org on a given source domain image x using a Fast Fourier Transform (FFT) as A org ,P org =FFT[x] and initializing the original amplitude spectrum A org with a random perturbation as A 0 =A org ⊙(1+Unif(−ϵ,ϵ))FFT[x] where Unif(−ϵ,ϵ)∈ represents a two-dimensional (2D) matrix with each entry sampled uniformly from [−ϵ,ϵ], and ⊙ denotes a Hadamard product.

9 . The method of claim 1 , wherein the employing of the adversarial technique to generate the spectral adversarial images comprises iteratively optimizing an amplitude spectrum A t+1 by adding a M S -weighted sign gradient of a cross-entropy loss to the amplitude spectrum A t with δ as a perturbation step size to target each sensitive frequency component,

A

t
+
1

=

Assignees

• IBM
• Rensselaer Polytech Inst

Inventors

• Chen Pin-Yu
• Dhurandhar Amit
• Zhang Jiajin
• Chao Hanqing
• Yan Pingkun

Classifications

• G06V10/80

  Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level (multimodal speaker identification or verification G10L17/10) · CPC title

• G06V10/776 Primary

  Validation; Performance evaluation · CPC title

• G06V10/82 Primary

  using neural networks · CPC title