Self-Supervised Learning for Temporal Counterfactual Estimation

Patent metadata

Field	Value
Publication number	US-2025111285-A1
Application number	US-202418902137-A
Country	US
Kind code	A1
Filing date	Sep 30, 2024
Priority date	Sep 28, 2023
Publication date	Apr 3, 2025
Grant date	—

Abstract

A machine-learned model includes an encoder having a feature block configured to embed input data into a plurality of features in an embedding space. The input data includes multiple components such as covariate, treatment, and output components. The encoder includes one or more encoding layers, each including a temporal attention block and a feature-wise attention block. The temporal attention block is configured to obtain the embedded input data and apply temporal causal attention along a time dimension in parallel for each feature of the plurality of features to generate temporal embeddings. The feature-wise attention block is configured to obtain the temporal embeddings and generate component representations such as a covariate representation, a treatment representation, and an output representation.

First claim

What is claimed is:

1 . A system, comprising:
one or more processors; and one or more non-transitory computer-readable media that collectively store a machine-learned model including an encoder, the encoder comprising:
a feature block configured to embed input data into a plurality of features in an embedding space, the input data including multiple components; and
one or more encoding layers, each encoding layer including a temporal attention block and a feature-wise attention block, the temporal attention block configured to obtain the embedded input data and apply temporal causal attention along a time dimension in parallel for each feature of the plurality of features to generate temporal embeddings, the feature-wise attention block configured to obtain the temporal embeddings and generate component representations of the input data, the component representations including a respective representation for each of the multiple components.

2 . The system of claim 1 , wherein:
the multiple components of the input data include a covariate component, a treatment component, and an outcome component; and the component representations include a covariate representation, a treatment representation, and an outcome representation.

3 . The system of claim 1 , wherein:
the input data includes sequential data.

4 . The system of claim 1 , wherein:
the input data includes observed historical sequence data.

5 . The system of claim 1 , wherein:
the multiple components of the input data are time-varying components of the input data.

6 . The system of claim 1 , wherein the machine-learned model includes a supervised counterfactual transformer.

7 . The system of claim 1 , wherein the encoder comprises:
one or more pooling layers configured to obtain the component representations and generate an overall representation of the input data.

8 . The system of claim 1 , wherein the feature-wise attention block models interactions among different features of the plurality of features.

9 . The system of claim 1 , wherein:
each embedded input includes a sum of input feature projection and feature positional encoding.

10 . The system of claim 1 , wherein:
the temporal attention block is configured to capture temporal dependencies within each feature.

11 . The system of claim 1 , wherein:
the feature-wise attention block is configured to determine full self-attention along a feature dimension in a plurality of time steps.

12 . The system of claim 1 , wherein:
the plurality of features includes feature embeddings in an embedding space.

13 . The system of claim 1 , wherein:
the component representations generated by the feature-wise attention block include propagated embeddings of time-varying features.

14 . A computer-implemented method to perform outcome estimation, the method comprising:
obtaining, by a computing system comprising one or more computing devices, input data including multiple components; obtaining, by the computing system, a machine-learned model including an encoder; providing, by the computing system, the input data as one or more inputs to the machine-learned model; and pre-training, by the computing system, the encoder using a self-supervised learning loss having component-wise losses including a respective loss for each of the multiple components of the input data.

15 . The computer-implemented of claim 14 , wherein
the multiple components of the input data include a covariate component, a treatment component, and an outcome component; and the component-wise losses include a covariate contrastive loss for a covariate representation of a covariate component of the input data, a treatment contrastive loss for a treatment representation of a treatment component of the input data, and an outcome contrastive loss for an output representation of an outcome component of the input data.

16 . The computer-implemented of claim 14 , wherein the machine-learned model includes a supervised counterfactual transformer.

17 . The computer-implemented method of claim 14 , wherein the input data includes observed historical sequence data including a plurality of sequential data points, each sequential data point including a respective covariate component, a respective treatment component, and

18 . One or more non-transitory computer-readable media that collectively store a self-supervised counterfactual transformer including an encoder, the encoder comprising:
a feature block configured to embed input data into a plurality of features in an embedding space, the input data including multiple components; and one or more encoding layers, each encoding layer including a temporal attention block and a feature-wise attention block, the temporal attention block configured to obtain the embedded input data and apply temporal causal attention along a time dimension in parallel for each feature of the plurality of features to generate temporal embeddings, the feature-wise attention block configured to obtain the temporal embeddings and generate component representations of the input data, the component representations including a covariate representation, a treatment representation, and an output representation, the component representations including a respective representation for each of the multiple components.

19 . The system of claim 18 , wherein:
the multiple components of the input data include a covariate component, a treatment component, and an outcome component; and the component representations include a covariate representation, a treatment representation, and an outcome representation.

20 . The system of claim 18 , wherein the encoder comprises:
one or more pooling layers configured to obtain the component representations and generate an overall representation of the input data.

Assignees

• Google Llc

Inventors

• Liu Yan
• Meng Chuizheng
• Dong Yihe
• Arik Sercan Omer
• Pfister Tomas

Classifications

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N20/00 Primary

  Machine learning · CPC title