Computer implemented methods for the automated analysis or use of data, including use of a large language model

Patent metadata

Field	Value
Publication number	US-12430504-B2
Application number	US-202418914659-A
Country	US
Kind code	B2
Filing date	Oct 14, 2024
Priority date	Aug 24, 2021
Publication date	Sep 30, 2025
Grant date	Sep 30, 2025

Abstract

There is provided a method of improving the operation of a generative AI large language model (LLM)-based data processing system, by operating the LLM-based system in conjunction with a non-LLM data processing system; in which the LLM-based system sends a continuation as an input to the non-LLM system; and in which the non-LLM system (a) uses symbolic representations to (i) generate factual assertions and/or (ii) generate non-statistical reasoning steps, in each case by processing the input sent from the LLM-based system and (b) stores the factual assertions and/or non-statistical reasoning steps (“stored facts and reasoning data”) in a memory for long term re-use by the LLM-based system and/or the non-LLM system.

First claim

The invention claimed is:

1. A method of improving the operation of a generative AI large language model (LLM)-based data processing system, by operating the LLM-based system in conjunction with a non-LLM data processing system, in which a large language model is a deep learning model capable of processing natural language;
in which the LLM-based system sends a continuation as an input to the non-LLM system;
and in which the non-LLM system (a) uses symbolic representations to (i) generate factual assertions and/or (ii) generate non-statistical reasoning steps, in each case by processing the input sent from the LLM-based system and (b) stores the factual assertions and/or non-statistical reasoning steps (“stored facts and reasoning data”) in a memory for long term re-use by the LLM-based system and/or the non-LLM system.

2. The method of claim 1 in which the non-LLM system provides a semantic backbone or support for the LLM-based system.

3. The method of claim 1 in which the non-LLM system is programmed to use a non-statistical symbolic representation of one or more of the following: things, relations between things, facts, relations between facts.

4. The method of claim 1 in which the non-LLM system uses a symbolic representation to generate the stored facts and reasoning data, and stored facts and reasoning data are sent to the LLM-based system, where the sent stored facts and reasoning data are used to improve one or more of the following parameters of new continuations or other outputs of the LLM-based system: factual accuracy and/or factual scope of new continuations or other output; internal, logical self-consistency of new continuations or other outputs; correspondence of new continuations or other outputs to how people understand the real world or reason in the real world; bias reduction or removal in new continuations or other outputs; inclusion of dynamic or real-time information.

5. The method of claim 1 in which the non-LLM system uses a symbolic representation to provide the stored facts and reasoning data to the LLM-based system, where the stored facts and reasoning data includes one or more the following: (i) dynamic or real-time information; (ii) reasoned text; (iii) an answer to a question.

6. The method of claim 1 in which the non-LLM system provides stored facts and reasoning data to the LLM-based system, where the provided stored facts and reasoning data is labelled with a level of certainty or uncertainty, or trust or lack of trust.

7. The method of claim 1 in which the input from the LLM-based system to the non-LLM system is a partial continuation, namely a continuation or other output made before the LLM-based system has stopped generating or whilst the LLM-based system is still generating.

8. The method of claim 1 in which a classifier operates to (a) identify when a prompt or other input to the LLM-based system is likely to result in a continuation or other output from the LLM-based system where accuracy is important, and (b) to then use the non-LLM system to improve the factual accuracy and/or factual scope of that continuation or other output.

9. The method of claim 1 in which the method includes the step of providing an input to the non-LLM system, where the non-LLM system uses symbolic representations to analyse the input to the non-LLM system and then accesses or searches an authoritative knowledge or data source external to the LLM-based system to enable an enhanced or augmented version of the input to the non-LLM system to be generated and provided to the LLM-based system as an augmented prompt or other context.

10. The method of claim 9 in which the authoritative knowledge or data source external to the LLM-based system is one or more of: the web, social media, and document stores;
and the non-LLM system provides the augmented prompt or other context to the LLM-based system to improve one or more of the following parameters of a new continuation or other output of the LLM-based system: factual accuracy and/or factual scope of a continuation or other output; internal, logical self-consistency of the continuation or other output; correspondence of the continuation or other output to how people understand the real world or reason in the real world; bias reduction or removal in the continuation or other output; inclusion of dynamic or real-time information.

11. The method of claim 9 in which the non-LLM system analyses the input to the non-LLM system by extracting from the input to the non-LLM system a collection of one or more factual assertions, and checks one or more of the one or more factual assertions for accuracy, and then outputs, or makes available, results of the checking to enable the enhanced or augmented version of the input to the non-LLM system to be provided to the LLM-based system as a prompt or other context.

12. The method of claim 9 in which the non-LLM system includes a reasoning system able to reason with a symbolic representation of the world, and the LLM-based system provides the continuation input to the non-LLM system which then (i) translates the continuation input to the non-LLM system into one or more assertions in a form compatible with the reasoning system and (ii) utilises the reasoning system to identify one or more assertions which are true or false.

13. The method of claim 9 in which the non-LLM system includes a symbolic reasoning system able to reason with symbolic representations, and to translate assertions into questions and to answer those questions.

14. The method of claim 1 which includes the step of the non-LLM system providing labels or other data to the LLM-based system that the LLM-based system uses to control an output generated by the LLM-based system, the labels or other data including one or more of: labels that relate to certainty, brevity, expectation that a response will be spoken, level of formality; use or non-use of profanity, context of an age or other details of a person being addressed, emotion to be conveyed in a response.

15. The method of claim 1 in which the method includes the step of the non-LLM system performing a symbolic computation on the input using multi-dimensional vectors.

16. The method of claim 1 in which the method includes the step of the non-LLM system providing the stored facts and reasoning data as training data to the LLM-based system.

17. The method of claim 16 in which the training data is reasoned text derived using a non-statistical, symbolic reasoning process.

18. The method of claim 16 in which if the LLM-based system is generating an answer to a question, then the training data provided to the LLM-based system by the non-LLM system is an answer to that question derived using a non-statistical, symbolic reasoning process.

19. The method of claim 16 in which the training data provided to the LLM-based system includes dynamic or real-time information.

20. The method of claim 16 in which the training data provided to the LLM-based system is labelled with a level of certainty or uncertainty, or trust or lack of trust.

21. The method of claim 16 in which the non-LLM system provides training data to the LLM-based system to improve any of the following aspects of output of the LLM-based system: (i) factual accuracy and/or factual scope; (ii) internal, logical self-consistency; (iii) correspondence to how people understand the real world or reason in the real world; (iv) bias.

22. The method of claim 1 in which the LLM-based system provides a continuation input to the

Assignees

• Unlikely Artificial Intelligence Ltd

Inventors

• Tunstall-Pedoe William
• Heywood Robert
• Warren Seth
• Benn Paul
• Reynolds Duncan
• Shah Ayush
• Krnic Luci
• Zhu Ziyi

Classifications

• G06F40/56

  Natural language generation · CPC title

• G06F16/3344 Primary

  using natural language analysis · CPC title

• G06F40/20 Primary

  Natural language analysis (semantic analysis of natural language G06F40/30) · CPC title

• G06F40/30

  Semantic analysis · CPC title

• G06F40/58

  Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation · CPC title