Computer implemented methods for the automated analysis or use of data, including use of a large language model

Patent metadata

Field	Value
Publication number	US-12456008-B2
Application number	US-202418914717-A
Country	US
Kind code	B2
Filing date	Oct 14, 2024
Priority date	Aug 24, 2021
Publication date	Oct 28, 2025
Grant date	Oct 28, 2025

Abstract

There is provided a method of improving the operation of a generative AI large language model (LLM)-based data processing system, by operating the LLM-based system in conjunction with a non-LLM data processing system; and in which (a) the LLM-based system sends a continuation as an input to the non-LLM system, and (b) the non-LLM system (i) uses symbolic representations to perform non-statistical reasoning on the input from the LLM-based system and (ii) generates a reasoned prompt or other context.

First claim

The invention claimed is:

1. A method of improving the operation of a generative AI large language model (LLM)-based data processing system, by operating the LLM-based system in conjunction with a non-LLM data processing system, in which a large language model is a deep learning model capable of processing natural language;
and in which (a) the LLM-based system sends a continuation as an input to the non-LLM system, and (b) the non-LLM system (i) uses symbolic representations to perform non-statistical reasoning on the input from the LLM-based system and (ii) generates a reasoned prompt or other context.

2. The method of claim 1 in which the non-LLM system provides a semantic backbone or support for the LLM-based system.

3. The method of claim 1 in which the non-LLM system is programmed to use a non-statistical symbolic representation of one or more of the following: things, relations between things, facts, relations between facts.

4. The method of claim 1 in which the non-LLM system uses a symbolic representation to provide the reasoned prompt or other context to the LLM-based system, where the reasoned prompt or other context improves one or more of the following parameters of a new continuation or other output of the LLM-based system: factual accuracy and/or factual scope of the new continuation or other output; internal, logical self-consistency of the new continuation or other output; correspondence of the new continuation or other output to how people understand the real world or reason in the real world; bias reduction or removal in the new continuation or other output; inclusion of dynamic or real-time information.

5. The method of claim 1 in which the non-LLM system uses a symbolic representation to provide the reasoned prompt or other context to the LLM-based system, where the reasoned prompt or other context includes one or more the following: (i) dynamic or real-time information; (ii) reasoned text; (iii) an answer to a question.

6. The method of claim 1 in which the reasoned prompt or other context is labelled with a level of certainty or uncertainty, or trust or lack of trust.

7. The method of claim 1 in which the input from the LLM-based system that is sent to the non-LLM system is a partial continuation, namely a continuation or other output made before the LLM-based system has stopped generating or whilst the LLM-based system is still generating.

8. The method of claim 1 in which a classifier operates to (a) identify when the input from the LLM-based system is one for which factual accuracy and/or factual scope is important, and (b) to then use the non-LLM system to provide the reasoned prompt or other context to the LLM-based system to improve the factual accuracy and/or factual scope of a new continuation or other output from the LLM-based system.

9. The method of claim 1 in which the method includes the step of providing the input to the non-LLM system, where the non-LLM system uses symbolic representations to analyse the input to the non-LLM system and then accesses or searches an authoritative knowledge or data source external to the LLM-based system to generate an enhanced or augmented version of the input to the non-LLM system and to provide the enhanced or augmented version of the input to the LLM-based system as an augmented, reasoned prompt or other context.

10. The method of claim 9 in which the authoritative knowledge or data source external to the LLM-based system is one or more of: the web, social media, and document stores;
and the non-LLM system provides the augmented, reasoned prompt or other context to the LLM-based system to improve one or more of the following parameters of a new continuation or other output of the LLM-based system: factual accuracy and/or factual scope of the new continuation or other output; internal, logical self-consistency of the new continuation or other output; correspondence of the new continuation or other output to how people understand the real world or reason in the real world; bias reduction or removal in the new continuation or other output; inclusion of dynamic or real-time information.

11. The method of claim 9 in which the non-LLM system analyses the input to the non-LLM system by extracting from the input to the non-LLM system a collection of one or more factual assertions, and checks one or more of the one or more factual assertions for accuracy, and then outputs, or makes available, results of the checking as a reasoned prompt or other context to the LLM-based system.

12. The method of claim 9 in which the non-LLM system includes a reasoning system able to reason with a symbolic representation of the world, and the LLM-based system provides the continuation input to the non-LLM system which then (i) translates the continuation input to the non-LLM system into one or more assertions in a form compatible with the reasoning system and (ii) utilises the reasoning system to identify one or more assertions which are true or false.

13. The method of claim 9 in which the non-LLM system includes a symbolic reasoning system able to reason with symbolic representations, and to translate assertions into questions and to answer those questions.

14. The method of claim 1 which includes the step of the non-LLM system providing labels or other data to the LLM-based system that the LLM-based system uses to control output generated by the LLM-based system, the labels or other data including one or more of: labels that relate to certainty, brevity, expectation that a response will be spoken, level of formality; use or non-use of profanity, context of an age or other details of a person being addressed, emotion to be conveyed in a response.

15. The method of claim 1 in which the method includes the step of the non-LLM system performing a symbolic computation on the input using multi-dimensional vectors.

16. The method of claim 1 in which the method includes the step of the non-LLM system providing the reasoned prompt or other context as training data to the LLM-based system.

17. The method of claim 16 in which the training data is reasoned text derived using a non-statistical, symbolic reasoning process.

18. The method of claim 16 in which if the LLM-based system is generating an answer to a question, then the training data provided to the LLM-based system by the non-LLM system is an answer to the question derived using a non-statistical, symbolic reasoning process.

19. The method of claim 16 in which the training data provided to the LLM-based system includes dynamic or real-time information.

20. The method of claim 16 in which the training data provided to the LLM-based system is labelled with a level of certainty or uncertainty, or trust or lack of trust.

21. The method of claim 16 in which the non-LLM system provides training data to the LLM-based system to improve any of the following aspects of output of the LLM-based system: (i) factual accuracy and/or factual scope; (ii) internal, logical self-consistency; (iii) correspondence to how people understand the real world or reason in the real world; (iv) bias.

22. The method of claim 1 in which the LLM-based system provides the continuation input to the non-LLM system that then (a) automatically analyses the continuation input using a symbolic process to generate factually accurate training data, and (b) provides the factually accurate training data back to the LLM-based system to train the LLM-based system.

23. The method of claim 22 in which the

Assignees

• Unlikely Artificial Intelligence Ltd

Inventors

• Tunstall-Pedoe William
• Heywood Robert
• Warren Seth
• Benn Paul
• Reynolds Duncan
• Shah Ayush
• Krnic Luci
• Zhu Ziyi

Classifications

• G06F40/56

  Natural language generation · CPC title

• G06F16/3344 Primary

  using natural language analysis · CPC title

• G06F40/20 Primary

  Natural language analysis (semantic analysis of natural language G06F40/30) · CPC title

• G06F40/30

  Semantic analysis · CPC title

• G06F40/58

  Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation · CPC title