Controlling interactive agents using multi-modal inputs

Patent metadata

Field	Value
Publication number	US-12482464-B2
Application number	US-202218077194-A
Country	US
Kind code	B2
Filing date	Dec 7, 2022
Priority date	Dec 7, 2021
Publication date	Nov 25, 2025
Grant date	Nov 25, 2025

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for controlling agents. In particular, an interactive agent can be controlled based on multi-modal inputs that include both an observation image and a natural language text sequence.

First claim

What is claimed is:

1 . A method performed by one or more computers and for controlling an agent interacting with an environment, the method comprising, at each of a plurality of time steps:
receiving an observation image characterizing a state of the environment at the time step; receiving a natural language text sequence for the time step that characterizes a task being performed by the agent in the environment at the time step; processing the observation image using an image embedding neural network to generate a plurality of image embeddings that represent the observation image; processing the natural language text sequence using a text embedding neural network to generate a plurality of text embeddings that represent the natural language text sequence; processing an input comprising the image embeddings, the text embeddings, and a set of one or more dedicated embeddings using a multi-modal Transformer neural network to generate an aggregated embedding, wherein the multi-modal Transformer neural network is configured to (i) apply self-attention over at least the text embeddings and the image embeddings to generate respective updated embeddings for at least the plurality of dedicated embeddings and (ii) generate the aggregated embedding from at least the respective updated embeddings for the dedicated embeddings; selecting, using the aggregated embedding, one or more actions to be performed by the agent in response to the observation image; and causing the agent to perform the one or more selected actions.

2 . The method of claim 1 , wherein the multi-modal Transformer neural network comprises one or more self-attention layers that each have one or more self-attention heads, and wherein applying self-attention comprises processing the input through the one or more self-attention layers.

3 . The method of claim 2 , wherein the set of one or more dedicated embeddings are the same for each of the plurality of time steps and are not dependent on the observation images at the plurality of time steps.

4 . The method of claim 3 , wherein applying self-attention comprises generating respective updated embeddings for the text embeddings and the dedicated embeddings without updating the image embeddings.

5 . The method of claim 3 , wherein each self-attention head of each self-attention layer is configured to:
receive a head input comprising (i) the image embeddings generated by the image embedding neural network and (ii) respective current embeddings for the text embeddings and the dedicated embeddings; generate, from the respective current embeddings, a respective query corresponding to each text embedding and each dedicated embedding; generate, from the image embeddings and the respective current embeddings, a respective key corresponding to each image embedding, each text embedding, and each dedicated embedding; generate, from the image embeddings and the respective current embeddings, a respective value corresponding to each image embedding, each text embedding, and each dedicated embedding; and apply query-key-value attention over the respective queries, keys, and values to generate a respective initial updated embedding for each text embedding and each dedicated embedding without updating the image embeddings.

6 . The method of claim 3 , wherein generating the aggregated embedding comprises:
aggregating the respective updated embeddings for the text embeddings and the dedicated embeddings to generate an initial aggregated embedding; and combining the respective updated embeddings for the dedicated embeddings with the initial aggregated embedding to generate the aggregated embedding.

7 . The method of claim 6 , wherein the combining comprises concatenating each respective updated embedding for each dedicated embedding and the initial aggregated embedding.

8 . The method of claim 1 , wherein selecting, using the aggregated embedding, one or more actions to be performed by the agent in response to the observation image comprises:
generating a state representation from the aggregated embedding; and selecting the one or more actions using the state representation.

9 . The method of claim 8 , wherein generating the state representation comprises processing the aggregated embedding using a memory neural network.

10 . The method of claim 9 , wherein the memory neural network is a recurrent neural network.

11 . The method of claim 8 , further comprising:
processing the state representation using a natural language generation neural network to generate an output text sequence for the time step.

12 . The method of claim 11 , wherein the natural language text sequence is generated by transcribing a verbalized utterance from another agent in the environment, and wherein the method further comprises:
generating speech representing the output text sequence for the time step; and causing the agent to verbalize the generated speech.

13 . The method of claim 11 , further comprising:
processing the state representation using a text no-op neural network to generate an indication of whether text should be generated at the time step; and wherein processing the state representation for the time step using a natural language generation neural network to generate an output text sequence for the time step comprises: only generating the output text sequence when the indication indicates that text should be generated at the time step.

14 . The method of claim 8 , wherein selecting the one or more actions using the state representation comprises:
processing the state representation using an action policy neural network to select a single action to be performed in response to the image observation.

15 . The method of claim 8 , wherein selecting the one or more actions using the state representation comprises:
processing the state representation to select a sequence of a plurality of actions to be performed in response to the image observation, the sequence comprising a respective action at each of a plurality of positions.

16 . The method of claim 15 , wherein processing the state representation comprises:
processing the state representation using a high-level controller neural network to generate a respective low-level input for each position in the sequence; and for each position, processing the respective low-level input for the position using a policy neural network to select the action to be performed by the agent at the position in the sequence.

17 . The method of claim 16 , wherein the high-level controller neural network auto-regressively generates the respective low-level inputs for each position in the sequence after receiving as input the state representation.

18 . The method of claim 17 , wherein the high-level controller neural network is a recurrent neural network.

19 . The method of claim 8 , further comprising:
processing the state representation using an action no-op neural network to generate an indication of whether any actions should be performed at the time step; and wherein causing the agent to perform the one or more actions comprises: only causing the agent to perform the actions when the indication indicates that actions should be performed at the time step.

20 . The method of claim 1 , wherein the natural language text sequence is generated by transcribing a verbalized utterance from another agent in the environment.

21 . A method performed by one or more computers and for controlling an agent interacting with an envir

Assignees

• Deepmind Tech Ltd

Inventors

• Abramson Joshua Simon
• Ahuja Arun
• Carnevale Federico Javier
• Georgiev Petko Ivanov
• Hung Chia-Chun
• Lillicrap Timothy Paul
• Muldal Alistair Michael
• Santoro Adam Anthony
• Von Glehn Tamara Louise
• Landon Jessica Paige
• Wayne Gregory Duncan
• Yan Chen
• Zhu Rui

Classifications

• G10L2015/223

  Execution procedure of a spoken command · CPC title

• G10L15/16

  using artificial neural networks · CPC title

• G10L15/063

  Training · CPC title

• G10L13/02

  Methods for producing synthetic speech; Speech synthesisers · CPC title

• G06V10/774

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title