Malicious prompt management for large language models

Patent metadata

Field	Value
Publication number	US-12457239-B1
Application number	US-202418651643-A
Country	US
Kind code	B1
Filing date	Apr 30, 2024
Priority date	Apr 30, 2024
Publication date	Oct 28, 2025
Grant date	Oct 28, 2025

Abstract

A method includes receiving, at a server from a user device, a user prompt segment to a large language model (LLM), obtaining an additional prompt segment from a prompt data source, identifying a electronic address in the prompt segment, replacing the electronic address with a placeholder to generate a updated prompt segment, generating a LLM prompt comprising the updated prompt segment and the user prompt segment, and sending the LLM prompt to the LLM. The method further includes receiving a response to the LLM prompt from the LLM, the response comprising the placeholder, replacing the placeholder with the electronic address to generate an updated response, and sending the updated response to the user device.

First claim

What is claimed is:

1. A method comprising:
receiving, at a server from a first user device, a first user prompt segment to a large language model (LLM);
obtaining a first additional prompt segment from a first prompt data source;
for each prompt segment of a first plurality of prompt segments in the LLM prompt:
obtaining a length value and a class,
validating that the length value satisfies a threshold length value for the class, wherein the first plurality of prompt segments comprises the first additional prompt segment and the first user prompt segment;

making a determination, at least in part based on the length value satisfying the threshold length value for each prompt segment, that the first plurality of prompt segments does not correspond to a prompt injection event;
identifying a first electronic address in the first user prompt segment;
replacing the first electronic address with a first placeholder to generate a first updated prompt segment;
generating a first LLM prompt comprising the first updated prompt segment and the first user prompt segment;
sending, responsive to the determination, the first LLM prompt to the LLM;
receiving a first response to the first LLM prompt from the LLM, the first response comprising the first placeholder;
replacing the first placeholder with the first electronic address to generate a first updated response; and
sending the first updated response to the first user device.

2. The method of claim 1 , further comprising:
validating the first electronic address prior to replacing the first placeholder with the first electronic address.

3. The method of claim 2 , wherein validating the first electronic address comprising comparing the first electronic address to a stored set of malicious addresses.

4. The method of claim 1 , further comprising:
classifying each of the first plurality of prompt segments to obtain the class for each prompt segment of the first plurality of prompt segments.

5. The method of claim 4 , wherein classifying the first plurality of prompt segments into a plurality of classes is according to a property of a plurality of prompt data sources, the plurality of prompt data sources comprising the first prompt data source, and the plurality of classes comprises the class.

6. The method of claim 1 , further comprising:
receiving, at the server from a second user device, a second user prompt segment to the LLM;
obtaining a second additional prompt segment from a second prompt data source;
obtaining a second length value and a second class for a prompt segment in a plurality of prompt segments, wherein the plurality of prompt segments comprises the second additional prompt segment and the second user prompt segment;
detecting that the second length value fails to satisfy a second threshold length value for the second class; and
outputting the prompt injection event responsive to the length value failing to satisfy the second threshold length value.

7. The method of claim 6 , further comprising:
obtaining a plurality of length values for a plurality of historical prompt segments assigned to the second class;
calculating a distribution of the plurality of length values;
determining the second threshold length value corresponding to a predefined quantile of the distribution; and
assigning the second threshold length value to the second class.

8. A system comprising:
at least one computer processor; and
a large language model (LLM) prompt manager executing on the at least one computer processor and configured to:
receive, at a server from a first user device, a first user prompt segment to the LLM, obtain a first additional prompt segment from a first prompt data source,
for each prompt segment of a first plurality of prompt segments in the LLM prompt:
obtaining a length value and a class,
validating that the length value satisfies a threshold length value for the class wherein the first plurality of prompt segments comprises the first additional prompt segment and the first user prompt segment,

making a determination, at least in part based on the length value satisfying the threshold length value for each prompt segment, that the first plurality of prompt segments does not correspond to a prompt injection event,
identify a first electronic address in the first user prompt segment,
replace the first electronic address with a first placeholder to generate a first updated prompt segment,
generate a first LLM prompt comprising the first updated prompt segment and the first user prompt segment,
send, responsive to the determination, the first LLM prompt to the LLM,
receive a first response to the first LLM prompt from the LLM, the first response comprising the first placeholder,
replace the first placeholder with the first electronic address to generate a first updated response, and
send the first updated response to the first user device.

9. The system of claim 8 , wherein the LLM prompt manager is further configured to:
validate the first electronic address prior to replacing the first placeholder with the first electronic address.

10. The system of claim 9 , wherein validating the first electronic address comprising comparing the first electronic address to a stored set of malicious addresses.

11. The system of claim 8 , wherein the LLM prompt manager is further configured to:
classify each of the first plurality of prompt segments to obtain the class for each prompt segment of the first plurality of prompt segments.

12. The system of claim 11 , wherein classifying the first plurality of prompt segments into a plurality of classes is according to a property of a plurality of prompt data sources, the plurality of prompt data sources comprising the first prompt data source, and the plurality of classes comprises the class.

13. The system of claim 8 , wherein the LLM prompt manager is further configured to:
receive, at the server from a second user device, a second user prompt segment to the LLM;
obtain a second additional prompt segment from a second prompt data source;
obtain a second length value and a second class for a prompt segment in a plurality of prompt segments, wherein the plurality of prompt segments comprises the second additional prompt segment and the second user prompt segment;
detect that the second length value fails to satisfy a second threshold length value for the second class; and
output the prompt injection event responsive to the length value failing to satisfy the second threshold length value.

14. The system of claim 13 , wherein the LLM prompt manager is further configured to:
obtain a plurality of length values for a plurality of historical prompt segments assigned to the second class;
calculate a distribution of the plurality of length values;
determine the second threshold length value corresponding to a predefined quantile of the distribution; and
assign the second threshold length value to the second class.

15. A method comprising:
obtaining a plurality of length values for a plurality of historical prompt segments assigned to a class;
calculating a distribution of the plurality of length values;
determining a threshold length value corresponding to a predefined quantile of the distribution;
assigning the threshold length value to the class;
receiving, at a server from a user device, a user prompt segment to a large language model (LLM);
obtaining an additional prompt segment from a prompt data source;
obtaining a length value and the class for a prompt segment in a plurality of prompt segments, wherein the plurality of prompt segments comprises the

Assignees

• Intuit Inc

Inventors

• Mantin Itsik Yizhak
• Bitton Ron
• Shtar Guy
• Mathov Gome Yael
• Venturelli Henry

Classifications

• H04L63/1466

  Active attacks involving interception, injection, modification, spoofing of data unit addresses, e.g. hijacking, packet injection or TCP sequence number attacks · CPC title

• H04L63/1483

  service impersonation, e.g. phishing, pharming or web spoofing (detection of rogue wireless access points H04W12/12) · CPC title

• H04L63/1441 Primary

  Countermeasures against malicious traffic (countermeasures against attacks on cryptographic mechanisms H04L9/002) · CPC title