Self-improving LLMs through consistency-based self-generated demonstrations

Patent metadata

Field	Value
Publication number	US-12468897-B2
Application number	US-202318128450-A
Country	US
Kind code	B2
Filing date	Mar 30, 2023
Priority date	Jan 20, 2023
Publication date	Nov 11, 2025
Grant date	Nov 11, 2025

Abstract

Aspects of the disclosure are directed to automatically selecting examples in a prompt for an LLM to demonstrate how to perform tasks. Aspects of the disclosure can select and build a set of examples from LLM zero-shot outputs via predetermined criteria that can combine consistency, diversity, and repetition. In the zero-shot setting for three different LLMs, using only LLM predictions, aspects of the disclosure can improve performance up to 15% compared to zero-shot baselines and can match or exceed few-shot base-lines for a range of reasoning tasks.

First claim

The invention claimed is:

1 . A method for consistency based self-adaptive prompting, comprising:
generating, by one or more processors, a pool of demonstrations using a large language model (LLM) for a plurality of test queries by running chain-of-thought (CoT) over the plurality of test queries; determining, by the one or more processors, a self-consistency score for respective demonstrations in the pool of demonstrations; selecting, by the one or more processors, a set of demonstrations from the pool of demonstrations based on the self-consistency scores; prepending, by the one or more processors, the set of demonstrations to the plurality of test queries; and generating, by the one or more processors, a plurality of predictions based on the test queries prepended with the set of demonstrations using the LLM.

2 . The method of claim 1 , further comprising receiving, by the one or more processors, the plurality of test queries, each test query being concatenated with a trigger phrase or a labeled demonstration.

3 . The method of claim 1 , wherein the CoT comprises Zero-shot CoT or Few-shot CoT.

4 . The method of claim 1 , wherein the CoT is run multiple times using the LLM to generate multiple reasoning paths and different predictions for each test query.

5 . The method of claim 1 , wherein selecting a set of demonstrations further comprises computing a majority vote prediction and retaining only reasoning paths that result in a majority vote prediction.

6 . The method of claim 5 , wherein the majority vote prediction is computed based on entropy and repetitiveness.

7 . The method of claim 1 , wherein prepending the set of demonstrations further comprises adaptively allocating a number of demonstrations per test query that is proportional to an entropy of the test query.

8 . The method of claim 1 , wherein generating the plurality of predictions further comprises querying the test queries prepended with the set of demonstrations multiple times using the LLM to generate multiple predictions for each test query.

9 . The method of claim 8 , further comprising:
selecting, by the one or more processors, a prediction of the multiple predictions for each test query based on a majority voting; and outputting, by the one or more processors, the selected prediction for each test query.

10 . A system comprising:
one or more processors; and one or more storage devices coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations for consistency based self-adaptive prompting, the operations comprising:
generating a pool of demonstrations using a large language model (LLM) for a plurality of test queries by running chain-of-thought (CoT) over the plurality of test queries;
determining a self-consistency score for respective demonstrations in the pool of demonstrations;
selecting a set of demonstrations from the pool of demonstrations based on the self-consistency scores;
prepending the set of demonstrations to the plurality of test queries; and
generating a plurality of predictions based on the test queries prepended with the set of demonstrations using the LLM.

11 . The system of claim 10 , wherein the CoT is run multiple times using the LLM to generate multiple reasoning paths and different predictions for each test query.

12 . The system of claim 10 , wherein selecting a set of demonstrations further comprises computing a majority vote prediction and retaining only reasoning paths that result in a majority vote prediction, the majority vote prediction being computed based on entropy and repetitiveness.

13 . The system of claim 10 , wherein generating the plurality of predictions further comprises querying the test queries prepended with the set of demonstrations multiple times using the LLM to generate multiple predictions for each test query.

14 . The system of claim 13 , wherein the operations further comprise selecting a prediction of the multiple predictions for each test query based on a majority voting.

15 . A non-transitory computer readable medium for storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations for consistency based self-adaptive prompting, the operations comprising:
generating a pool of demonstrations using a large language model (LLM) for a plurality of test queries by running chain-of-thought (CoT) over the plurality of test queries; determining a self-consistency score for respective demonstrations in the pool of demonstrations; selecting a set of demonstrations from the pool of demonstrations based on the self-consistency scores; prepending the set of demonstrations to the plurality of test queries; and generating a plurality of predictions based on the test queries prepended with the set of demonstrations using the LLM.

16 . The non-transitory computer readable medium of claim 15 , wherein the CoT is run multiple times using the LLM to generate multiple reasoning paths and different predictions for each test query.

17 . The non-transitory computer readable medium of claim 15 , wherein selecting a set of demonstrations further comprises computing a majority vote prediction and retaining only reasoning paths that result in a majority vote prediction, the majority vote prediction being computed based on entropy and repetitiveness.

18 . The non-transitory computer readable medium of claim 15 , wherein generating the plurality of predictions further comprises querying the test queries prepended with the set of demonstrations multiple times using the LLM to generate multiple predictions for each test query.

19 . The non-transitory computer readable medium of claim 18 , wherein the operations further comprise selecting a prediction of the multiple predictions for each test query based on a majority voting.

Assignees

• Google Llc

Inventors

• Sun Ruoxi
• Wan Xingchen
• Dai Hanjun
• Arik Sercan Omer
• Pfister Tomas

Classifications

• G06F16/3344 Primary

  using natural language analysis · CPC title

• G06N3/096

  Transfer learning · CPC title

• G06N3/088

  Non-supervised learning, e.g. competitive learning · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06F40/40 Primary

  Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) · CPC title