Computer implemented methods for the automated analysis or use of data, including use of a large language model

Patent metadata

Field	Value
Publication number	US-11989507-B2
Application number	US-202318301615-A
Country	US
Kind code	B2
Filing date	Apr 17, 2023
Priority date	Aug 24, 2021
Publication date	May 21, 2024
Grant date	May 21, 2024

Abstract

Methods are provided, such as a method of interacting with a large language model (LLM), including the step of a processing system using a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language, to provide new context data for the LLM, in order to improve the output, such as continuation text output, generated by the LLM in response to a prompt; and such as a method of interacting with a LLM, including the step of providing continuation data generated by the LLM to a processing system that uses a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language, in which the processing system is configured to analyse the continuation output generated by the LLM in response to a prompt to enable an improved version of that continuation output to be provided to a user. Related computer systems are provided.

First claim

The invention claimed is:

1. A method for ensuring that a large language model (LLM) generates original text, comprising the steps of:
(i) providing a first database of previous text that the LLM should not generate;
(ii) performing a beam search;
(iii) checking potential continuations generated by the LLM against the first database;
(iv) when a potential continuation generated by the LLM matches non-original text in the first database, adjusting the potential continuation generated by the LLM to no longer match non-original text in the first database;
the method further including a method for adding citations to text generated by the LLM comprising the steps of:
(v) providing a second database of text used to train the LLM which includes sources associated with each section of text stored;
(vi) checking sections of the potential continuation generated by the LLM against the second database; and
(vii) retrieving respective sources where respective sections of the potential continuation generated by the LLM match respective text contained within the second database.

2. The method of claim 1 where adjusting the potential continuation includes the step of selecting a different sequence from the beam search.

3. The method of claim 1 , further comprising the step of providing a classifier that can distinguish between prompts that require original text as a continuation and those that do not and applying the classifier to a prompt of the prompts.

4. The method of claim 1 , where checking potential continuations against the first database includes close but imperfect matches to the previous text.

5. The method of claim 1 , further comprising the steps of:
providing a classifier to determine whether a section of the potential continuation is appropriate for citing references against;
utilising the classifier when matching sections of the potential continuation.

6. The method of claim 1 , further comprising the step of:
adjusting the potential continuation to appropriately cite at least some of the respective sources.

7. The method of claim 1 , including a method of interacting with the large language model (LLM), including the step of providing continuation data generated by the LLM to a processing system that uses a structured, machine-readable representation of data that conforms to a machine-readable language, in which the processing system is configured to analyse the continuation data generated by the LLM in response to a prompt to enable an improved version of that continuation data to be provided to a user.

8. The method of claim 7 , in which the processing system translates at least some of the continuation data into the structured, machine-readable representation of data to analyse the translated continuation data.

9. The method of claim 7 , in which the processing system translates the continuation data into the structured, machine-readable representation of data to analyse the translated continuation data.

10. The method of claim 8 , in which the processing system analyses the translated continuation data for at least one of factual inaccuracies, contradictions and scope, and, where necessary or beneficial, provides corrected output for use in the continuation data provided by the LLM, wherein the processing system provides an updated prompt, wherein the corrected output is presented to a user.

11. The method of claim 8 , in which the processing system analyses the translated continuation data and, where necessary or beneficial, provides corrected output for use in the continuation data, wherein the corrected output is presented to a user.

12. The method of claim 7 , in which the processing system analyses internal, logical self-consistency, including a correct time ordering of events, of the continuation data and, where necessary or beneficial, provides logically self-consistent output, for use in the continuation data wherein the logically self-consistent output is provided to a user.

13. The method of claim 7 , in which the processing system analyses internal, logical self-consistency of the continuation data and, where necessary or beneficial, provides logically self-consistent output, for use in the continuation data wherein the logically self-consistent output is provided to a user.

14. The method of claim 7 , in which the processing system analyses correspondence of the continuation data generated by the LLM to how people understand the real world or reason in the real world and, where necessary or beneficial, provides reasoned output, corresponding to how people understand the real world or reason in the real world, for use in the continuation data wherein the reasoned output is provided to a user.

15. The method of claim 7 ,
(a) in which the processing system analyses the continuation data for bias and, where necessary or beneficial, provides reduced bias or bias-free output, for use in the continuation data that is provided to a user; or
(b) in which the processing system analyses the continuation data for an absence of relevant dynamic or real-time information and, where necessary or beneficial, provides any missing or useful dynamic or real-time information for use in the continuation data that is provided to a user; or
(c) in which the processing system analyses the continuation output for any reasoned text and, where necessary or beneficial, provides improved reasoned text for use in the continuation data that is provided to a user; or
(d) in which the LLM is answering a question and in which the processing system analyses an answer to that question in the continuation data and where necessary or beneficial, provides an improved answer for use in the continuation data that is provided to a user; or
(e) in which a classifier operates to identify when a prompt is likely to result in a continuation output where accuracy is important, and/or when accuracy is important in the continuation output, and to then use the processing system to improve factual accuracy and/or factual scope of that continuation output; or
(f) in which the continuation data from the LLM is a partial continuation, namely an output made before the LLM has stopped generating or whilst the LLM is still generating; or
(g) in which a classifier operates to identify when a prompt is likely to result in a continuation where accuracy is important; or
(h) when used to improve one or more of: accuracy of output of the LLM; factual accuracy and/or factual scope of output of the LLM; internal, logical self-consistency of output of the LLM; correspondence of output generated by the LLM to how people understand the real world or reason in the real world; bias reduction or removal in output of the LLM; inclusion of dynamic or real-time information; or
(i) when used to improve one or more parameters of output of the LLM: level of formality, level of brevity, suitability for speaking via a text to speech system, other style language, level of certainty; or
(j) in which the LLM is a generative AI based system; or
(k) in which the LLM is an autoregressive language model; or
(l) in which the LLM is an autoregressive language model, which is a Generative Pre-trained Transformer; or
(m) when used for: generation of program code; or
(n) when used for: solution of any problem that can be described in natural language; or
(o) when used for any of the following: generation of poetry, lyrics, creative writing, generation of other forms of writing, writing essays, writing summaries of knowledge, writing summaries of longer texts, writing scientific papers; internet search.

16. A method for ensuring that a large la

Assignees

• Unlikely Artificial Intelligence Ltd

Inventors

• Tunstall-Pedoe William
• Heywood Robert
• Warren Seth
• Benn Paul
• Reynolds Duncan
• Shah Ayush
• Krnic Luci
• Zhu Ziyi

Classifications

• G06F40/30

  Semantic analysis · CPC title

• G06F40/58

  Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation · CPC title

• G06F40/20 Primary

  Natural language analysis (semantic analysis of natural language G06F40/30) · CPC title

• G06F16/3344 Primary

  using natural language analysis · CPC title

• G06F40/56

  Natural language generation · CPC title