Training machine-learned models for perceptual tasks using biometric data

Patent metadata

Field	Value
Publication number	US-11823439-B2
Application number	US-202017428659-A
Country	US
Kind code	B2
Filing date	Jan 16, 2020
Priority date	Feb 6, 2019
Publication date	Nov 21, 2023
Grant date	Nov 21, 2023

Abstract

Generally, the present disclosure is directed to systems and methods that train machine-learned models (e.g., artificial neural networks) to perform perceptual or cognitive task(s) based on biometric data (e.g., brain wave recordings) collected from living organism(s) while the living organism(s) are performing the perceptual or cognitive task(s). In particular, aspects of the present disclosure are directed to a new supervision paradigm, by which machine-learned feature extraction models are trained using example stimuli paired with companion biometric data such as neural activity recordings (e g electroencephalogram data, electrocorticography data, functional near-infrared spectroscopy, and/or magnetoencephalography data) collected from a living organism (e.g., human being) while the organism perceived those examples (e.g., viewing the image, listening to the speech, etc.).

First claim

What is claimed is:

1. A computer-implemented method to perform multi-modal learning, the method comprising:
accessing, by one or more computing devices, data descriptive of a plurality of training examples, wherein each training example comprises a respective stimulus and a respective set of biometric data collected from a living organism concurrent with exposure of the living organism to the respective stimulus, the living organism having been instructed to perform a perceptual task on the respective stimulus during exposure of the living organism to the respective stimulus; and
for each of one or more of the plurality of training examples:
inputting, by the one or more computing devices, the respective stimulus into a machine-learned stimulus feature extraction model configured to process the respective stimulus to produce a respective stimulus embedding;
receiving, by the one or more computing devices, the respective stimulus embedding as an output of the machine-learned stimulus feature extraction model;
inputting, by the one or more computing devices, the respective set of biometric data into a machine-learned biometric feature extraction model configured to process the respective set of biometric data to produce a respective biometric embedding;
receiving, by the one or more computing devices, the respective biometric embedding as an output of the machine-learned biometric feature extraction model; and
learning, by the one or more computing devices and based at least in part on the respective stimulus embedding and the respective biometric embedding, one or both of: first parameter values of the machine-learned stimulus feature extraction model and second parameter values of the machine-learned biometric feature extraction model.

2. The computer-implemented method of claim 1 , wherein, for each of the plurality of training examples, the respective set of biometric data comprises a respective set of neural recording data descriptive of neural activity of the living organism concurrent with exposure of the living organism to the respective stimulus.

3. The computer-implemented method of claim 2 , wherein, for each of the plurality of training examples, the respective set of neural recording data comprises one or more of:
electroencephalogram data;
electrocorticography data;
magnetoencephalography data; and
functional near-infrared spectroscopy.

4. The computer-implemented method of claim 1 , wherein, for each of the plurality of training examples, the respective stimulus comprises one or more of:
a visual stimulus;
an auditory stimulus;
a haptic stimulus;
an olfactory stimulus; and
a gustatory stimulus.

5. The computer-implemented method of claim 1 , wherein the perceptual task comprises classification of the respective stimulus into one or more of a plurality of classes.

6. The computer-implemented method of claim 5 , wherein:
the respective stimulus comprises an image that depicts an object; and
the perceptual task comprises classification of the object into one or more of a plurality of object classes.

7. The computer-implemented method of claim 5 , wherein:
the respective stimulus comprises audio of human speech;
the plurality of classes comprise one or more of: a plurality of phonemes, a plurality of words, a plurality of semantic concepts, and a plurality of emotions; and
the perceptual task comprises classification of the human speech into one or more of the plurality of classes.

8. The computer-implemented method of claim 1 , wherein the perceptual task comprises detection of one or more items contained within the respective stimulus.

9. The computer-implemented method of claim 1 , wherein learning, by the one or more computing devices and based at least in part on the respective stimulus embedding and the respective biometric embedding, one or both of: the first parameter values of the machine-learned stimulus feature extraction model and the second parameter values of the machine-learned biometric feature extraction model comprises:
determining, by the one or more computing devices, a correlation between the respective stimulus embedding and the respective biometric embedding; and
adjusting, by the one or more computing devices, one or both of: the first parameter values of the machine-learned stimulus feature extraction model and the second parameter values of the machine-learned biometric feature extraction model based at least in part on a gradient of an objective function that seeks to maximize the correlation between the respective stimulus embedding and the respective biometric embedding.

10. The computer-implemented method of claim 1 , wherein learning, by the one or more computing devices and based at least in part on the respective stimulus embedding and the respective biometric embedding, one or both of: the first parameter values of the machine-learned stimulus feature extraction model and the second parameter values of the machine-learned biometric feature extraction model comprises:
providing, by the one or more computing devices, the respective stimulus embedding and the respective biometric embedding to a machine-learned fusion model configured to process the respective stimulus embedding and the respective biometric embedding to produce a prediction that indicates whether the respective stimulus and the respective set of biometric data are associated with each other; and
adjusting, by the one or more computing devices, one or both of: the first parameter values of the machine-learned stimulus feature extraction model and the second parameter values of the machine-learned biometric feature extraction model based at least in part on a gradient of a loss function that compares the prediction produced by the machine-learned fusion model to a ground truth label that indicates whether the respective stimulus and the respective set of biometric data are associated with each other.

11. The computer-implemented method of claim 1 , further comprising, after learning, by the one or more computing devices, the first parameter values of the machine-learned stimulus feature extraction model:
inputting, by the one or more computing devices, an additional stimulus into the machine-learned stimulus feature extraction model;
receiving, by the one or more computing devices, an additional stimulus embedding as an output of the machine-learned stimulus feature extraction model; and
one or more of:
performing, by the one or more computing device, the perceptual task on the additional stimulus based on the additional stimulus embedding;
performing, by the one or more computing device, a second, different perceptual task on the additional stimulus based on the additional stimulus embedding;
clustering, by the one or more computing devices, the additional stimulus with one or more other stimuli based on the additional stimulus embedding; and
identifying, by the one or more computing devices, one or more other stimuli that are similar to the additional stimulus based on the additional stimulus embedding.

12. The computer-implemented method of claim 1 , further comprising, after learning, by the one or more computing devices, the second parameter values of the machine-learned biometric feature extraction model:
inputting, by the one or more computing devices, an additional set of biometric data into the machine-learned biometric feature extraction model;
receiving, by the one or more computing devices, an additional biometric embedding as an output of the machine-learned biometric feature extraction model; and
one or more of:
decoding, by the one or more computing device, the additional biometric embedding to obtain an outcom

Assignees

• Google Llc

Inventors

• Jansen Aren
• Slaney Malcolm

Classifications

• G06V10/774 Primary

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V10/811

  the classifiers operating on different input data, e.g. multi-modal recognition · CPC title

• G06V40/15

  Biometric patterns based on physiological signals, e.g. heartbeat, blood flow · CPC title

• G06F18/2413 Primary

  based on distances to training or reference patterns · CPC title

• G06F2218/12

  Classification; Matching · CPC title