Interpretability-aware redundancy reduction for vision transformers

Patent metadata

Field	Value
Publication number	US-12154307-B2
Application number	US-202117559053-A
Country	US
Kind code	B2
Filing date	Dec 22, 2021
Priority date	Dec 22, 2021
Publication date	Nov 26, 2024
Grant date	Nov 26, 2024

Abstract

A sequence of patch tokens representing an image can be received. A network can be trained to learn informative patch tokens and uninformative patch tokens in the sequence of patch tokens, in learning to recognize an object in the image. The sequence of patch tokens can be reduced by removing the uninformative patch tokens from the sequence of patch tokens. The reduced sequence of patch tokens can be input to an attention-based deep learning neural network. The attention-based deep learning neural network can be fine-tuned to recognize the object in the image using the reduced sequence of patch tokens.

First claim

What is claimed is:

1. A computer-implemented method comprising:
receiving a sequence of patch tokens representing an image;
training a network to learn informative patch tokens and uninformative patch tokens in the sequence of patch tokens, in learning to recognize an object in the image, wherein the network includes a linear layer, wherein the sequence of patch tokens is input to the linear layer and the linear layer outputs a binary vector by applying a policy token to the sequence of patch tokens and feeding result of applying the policy token to the patch tokens into an activation function, the binary vector indicating whether the sequence of patch tokens input to the linear layer is activated or deactivated, wherein a value of the policy token is learned based on reinforcement learning during the training of the network;
reducing the sequence of patch tokens by removing the uninformative patch tokens from the sequence of patch tokens; and
inputting the reduced sequence of patch tokens to an attention-based deep learning neural network;
fine-tuning the attention-based deep learning neural network to recognize the object in the image using the reduced sequence of patch tokens,
wherein the attention-based deep learning neural network is divided into D number of groups, each group in the D number of groups including the network's linear layer and L blocks of multi-head self-attention layer and feed-forward network.

2. The method of claim 1 , wherein the network includes a multi-headed module connected to the attention-based deep learning neural network.

3. The method of claim 1 , wherein the attention-based deep learning neural network includes a vision transformer.

4. The method of claim 1 , wherein the network includes a linear layer, wherein the sequence of patch tokens input to the linear layer is activated or deactivated based on applying an activation function and a policy token.

5. The method of claim 1 , wherein training of the network and fine-tuning the attention-based deep learning neural network are performed together, wherein parameters learned by the network is used in fine-tuning the attention-based deep learning neural network.

6. The method of claim 1 , wherein the sequence of patch tokens have positional embeddings.

7. The method of claim 1 , wherein the network is optimized using reinforcement learning based on a prediction of the attention-based deep learning neural network.

8. A system comprising:
a processor; and
a memory device coupled with the processor;
the processor configured to at least:
receive a sequence of patch tokens representing an image;
train a network to learn informative patch tokens and uninformative patch tokens in the sequence of patch tokens, in learning to recognize an object in the image, wherein the network includes a linear layer, wherein the sequence of patch tokens is input to the linear layer and the linear layer outputs a binary vector by applying a policy token to the sequence of patch tokens and feeding result of applying the policy token to the patch tokens into an activation function, the binary vector indicating whether the sequence of patch tokens input to the linear layer is activated or deactivated, wherein a value of the policy token is learned based on reinforcement learning during the training of the network;
reduce the sequence of patch tokens by removing the uninformative patch tokens from the sequence of patch tokens; and
input the reduced sequence of patch tokens to an attention-based deep learning neural network;
fine-tune the attention-based deep learning neural network to recognize the object in the image using the reduced sequence of patch tokens,

wherein the attention-based deep learning neural network is divided into D number of groups, each group in the D number of groups including the network's linear layer and L blocks of multi-head self-attention layer and feed-forward network.

9. The system of claim 8 , wherein the network includes a multi-headed module connected to the attention-based deep learning neural network.

10. The system of claim 8 , wherein the attention-based deep learning neural network includes a vision transformer.

11. The system of claim 8 , wherein the network includes a linear layer, wherein the sequence of patch tokens input to the linear layer is activated or deactivated based on applying an activation function and a policy token.

12. The system of claim 8 , wherein the processor is configured to train the network and fine-tune the attention-based deep learning neural network together, wherein parameters learned by the network is used in fine-tuning the attention-based deep learning neural network.

13. The system of claim 8 , wherein the sequence of patch tokens have positional embeddings.

14. The system of claim 8 , wherein the network is optimized using reinforcement learning based on a reward determined based on a prediction of the attention-based deep learning neural network.

15. A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions readable by a device to cause the device to:
receive a sequence of patch tokens representing an image;
train a network to learn informative patch tokens and uninformative patch tokens in the sequence of patch tokens, in learning to recognize an object in the image, wherein the network includes a linear layer, wherein the sequence of patch tokens is input to the linear layer and the linear layer outputs a binary vector by applying a policy token to the sequence of patch tokens and feeding result of applying the policy token to the patch tokens into an activation function, the binary vector indicating whether the sequence of patch tokens input tothe linear layer is activated or deactivated, wherein a value of the policy token is learned based on reinforcement learning during the training of the network;
reduce the sequence of patch tokens by removing the uninformative patch tokens from the sequence of patch tokens; and
input the reduced sequence of patch tokens to an attention-based deep learning neural network;
fine-tune the attention-based deep learning neural network to recognize the object in the image using the reduced sequence of patch tokens,
wherein the attention-based deep learning neural network is divided into D number of groups, each group in the D number of groups including the network's linear layer and L blocks of multi-head self-attention layer and feed-forward network.

16. The computer program product of claim 15 , wherein the network includes a multi-headed module connected to the attention-based deep learning neural network.

17. The computer program product of claim 15 , wherein the attention-based deep learning neural network includes a vision transformer.

18. The computer program product of claim 15 , wherein the network includes a linear layer, wherein the sequence of patch tokens input to the linear layer is activated or deactivated based on applying an activation function and a policy token.

19. The computer program product of claim 15 , wherein the device is caused to train the network and fine-tune the attention-based deep learning neural network together, wherein parameters learned by the network is used in fine-tuning the attention-based deep learning neural network.

20. The computer program product of claim 15 , wherein the device is caused to optimize the network using reinforcement learning based on a reward determined based on a prediction of the attent

Assignees

• IBM
• Massachusetts Inst Technology

Inventors

• Pan Bowen
• Panda Rameswar
• Feris Rogerio Schmidt
• Oliva Aude Jeanne

Classifications

• G06T7/136

  involving thresholding · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T7/11

  Region-based segmentation · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06N3/08

  Learning methods · CPC title