Image analysis by prompting of machine-learned models using chain of thought

Patent metadata

Field	Value
Publication number	US-12346828-B2
Application number	US-202418967327-A
Country	US
Kind code	B2
Filing date	Dec 3, 2024
Priority date	Jun 3, 2022
Publication date	Jul 1, 2025
Grant date	Jul 1, 2025

Abstract

An example technique for image analysis is provided. An example image analysis method includes obtaining an instructive sequence descriptive of an instructive query, an instructive response, and an instructive trace of intermediate states from the instructive query to the instructive response. The example image analysis method includes inputting, to a machine-learned model, the instructive sequence and an operative image processing query that comprises image data, wherein the machine-learned model is configured to process the operative query with attention over the instructive sequence. The example method can include generating, using the machine-learned model and responsive to the operative query, an operative image processing response that comprises an analysis of the image data.

First claim

What is claimed is:

1. A computer-implemented method for performing image analysis, the method comprising:
obtaining, by a computing system comprising one or more processors, an instructive sequence descriptive of an instructive query, an instructive response, and an instructive trace of intermediate states from the instructive query to the instructive response;
inputting, by the computing system and to a machine-learned model, the instructive sequence and an operative image processing query comprising image data, wherein the machine-learned model is configured to process the operative image processing query with attention over the instructive sequence; and
generating, by the computing system, using the machine-learned model and responsive to the operative image processing query, an operative image processing response.

2. The computer-implemented method of claim 1 , comprising:
generating, by the computing system, using the machine-learned model and responsive to the operative image processing query, an operative trace of intermediate states from the operative query to the operative image processing response.

3. The computer-implemented method of claim 1 , wherein the instructive sequence is prepended to the operative image processing query.

4. The computer-implemented method of claim 2 , wherein the instructive trace comprises a chain of intermediate responses to intermediate queries.

5. The computer-implemented method of claim 1 , wherein the instructive sequence comprises an input flag and an output flag.

6. The computer-implemented method of claim 1 , wherein the instructive sequence comprises a tokenized representation of a natural language.

7. The computer-implemented method of claim 1 , wherein the instructive trace comprises one or more intermediate states of one or more variables declared by a computer-executable coding language.

8. The computer-implemented method of claim 1 , wherein generating the operative response comprises:
generating, by the computing system and using the machine-learned model, a plurality of operative responses; and
determining, by the computing system, the operative image processing response based on a sample of the plurality of operative responses.

9. The computer-implemented method of claim 8 , wherein determining the operative image processing response comprises:
determining, by the computing system, a consistency metric based on the sample of the plurality of operative responses.

10. The computer-implemented method of claim 8 , wherein the sample is based on respective probabilities associated with the plurality of operative responses.

11. The computer-implemented method of claim 9 , wherein the consistency metric comprises at least one of: a plurality vote, or a majority vote.

12. The computer-implemented method of claim 9 , wherein the consistency metric comprises a vote based on operative responses respectively associated with diverse operative traces.

13. The computer-implemented method of claim 1 , wherein the operative image processing query is a first query component and the operative image processing response is a first response component, and wherein the method comprises:
inputting, by the computing system and to the machine-learned model, the instructive sequence, the first query component, the first response component, and a second query component; and
generating, by the computing system, using the machine-learned model and responsive to the second query component, a second response component.

14. The computer-implemented method of claim 13 , comprising:
generating, by the computing system and responsive to a target query, one or more query components.

15. The computer-implemented method of claim 13 , comprising:
inputting, by the computing system and to the machine-learned model, a preliminary instructive sequence comprising a preliminary instructive query and a preliminary instructive response, wherein the preliminary instructive response comprises a plurality of preliminary instructive query components.

16. The computer-implemented method of claim 13 , wherein the first query component and the second query component are generated with a different machine-learned model other than the machine-learned model used to obtain the first response component and the second response component.

17. The computer-implemented method of claim 14 , wherein the second query component corresponds to the target query.

18. The computer-implemented method of claim 13 , comprising, for a plurality of iterations:
generating, by the computing system, an updated instructive sequence based on combining one or more prior input sequences with one or more output sequences respectively corresponding thereto;
inputting, by the computing system and to the machine-learned model, the updated instructive sequence and an additional query component; and
generating, by the computing system, using the machine-learned model and responsive to the additional query component, an additional response component.

19. One or more memory devices storing non-transitory computer-readable instructions executable to cause one or more processors to perform operations for performing image analysis, the operations comprising:
obtaining an instructive sequence descriptive of an instructive query, an instructive response, and an instructive trace of intermediate states from the instructive query to the instructive response;
inputting, to a machine-learned model, the instructive sequence and an operative image processing query comprising image data, wherein the machine-learned model is configured to process the operative image processing query with attention over the instructive sequence; and
generating, using the machine-learned model and responsive to the operative image processing query, an operative image processing response.

20. A computing system for performing image analysis, the system comprising:
one or more processors; and
one or more memory devices storing non-transitory computer-readable instructions that are executable to cause the one or more processors to perform operations, the operations comprising:
obtaining an instructive sequence descriptive of an instructive query, an instructive response, and an instructive trace of intermediate states from the instructive query to the instructive response;
inputting, to a machine-learned model, the instructive sequence and an operative image processing query comprising image data, wherein the machine-learned model is configured to process the operative image processing query with attention over the instructive sequence; and
generating, using the machine-learned model and responsive to the operative image processing query, an operative image processing response.

Assignees

• Google Llc

Inventors

• Wei Jason Weng
• Zhou Dengyong
• Schuurmans Dale Eric
• Le Quoc V
• Bosma Maarten Paul
• Chi Ed Huai-Hsin
• Bousquet Olivier Jean Andrè
• Hou Le
• Scales Nathan
• Bieber David J
• Sutton Charles Aloysius
• Schärli Nathanael
• Odena Augustus Quadrozzi
• Narang Sharan
• Gur-Ari Krakover Guy
• Chowdhery Aakanksha
• Lewkowycz Aitor
• Luan Jiageng
• Dohan David Martin
• Michalewski Henryk
• Austin Jacob
• Andreassen Anders Johan
• Nye Maxwell
• Wang Xuezhi

Classifications

• G06N3/042

  Knowledge-based neural networks; Logical representations of neural networks · CPC title

• G06N3/096

  Transfer learning · CPC title

• G06N5/022 Primary

  Knowledge engineering; Knowledge acquisition · CPC title

• G06N3/0455 Primary

  Auto-encoder networks; Encoder-decoder networks · CPC title