Methods and systems for generation of text using large language model with indications of unsubstantiated information

Patent metadata

Field	Value
Publication number	US-12468878-B2
Application number	US-202318180518-A
Country	US
Kind code	B2
Filing date	Mar 8, 2023
Priority date	Jan 31, 2023
Publication date	Nov 11, 2025
Grant date	Nov 11, 2025

Abstract

Methods and systems for prompting a large language model (LLM) to generate a description of an object with indications of any unsubstantiated information are disclosed. A prompt is generated to a LLM to generate a description of an object, where the prompt includes one or more object attributes to include in the generated description. The prompt also includes an instruction for the LLM to annotate any portions of the generated description that are, involve, and/or include unsubstantiated information according to a defined format. The prompt is provided to the LLM and the generated description is received. The generated description is parsed to identify, based on the defined format, one or more annotated portions indicating unsubstantiated information. The generated description is presented for display via a user device.

First claim

The invention claimed is:

1 . A system comprising:
a processing unit configured to execute computer-readable instructions to cause the system to:
generate a prompt to a large language model (LLM) to generate a description of an object, the prompt including one or more object attributes to include in the generated description, and also including an instruction for the LLM to annotate, according to a defined format, any portions of the generated description that include unsubstantiated information;
provide the prompt to the LLM;
causing the LLM to generate the generated description; receive the generated description;
parse the generated description to identify, based on the defined format, one or more annotated portions indicating unsubstantiated information; and
present the generated description for display via a user device.

2 . The system of claim 1 , wherein the generated description is presented in a user interface (UI) in which at least one of the identified one or more annotated portions is modifiable.

3 . The system of claim 2 , wherein the processing unit is configured to execute computer-readable instructions to further cause the system to, prior to presenting the UI:
identify, for a given one annotated portion in the generated description, an unsubstantiated object attribute; query a database to search for the unsubstantiated object attribute; modify the generated description by replacing the given one annotated portion with a found object attribute received in a response to the query; and present the modified generated description in the UI.

4 . The system of claim 3 , wherein there is a plurality of found object attributes received in the response to the query, and wherein the generated description is modified by replacing the given one annotated portion with a UI element for selecting one of the plurality of found object attributes.

5 . The system of claim 2 , wherein the UI further includes an input field for receiving user input to edit the one or more annotated portions.

6 . The system of claim 5 , wherein the processing unit is configured to execute computer-readable instructions to further cause the system to:
identify at least two annotated portions in the generated description requiring a same user input; provide one input field for receiving user input to edit the at least two annotated portions; and update the at least two annotated portions with information inputted in the one input field.

7 . The system of claim 1 , wherein the prompt to the LLM includes at least one example of an annotation according to the defined format.

8 . The system of claim 7 , wherein the at least one example is generated by:
retrieving, from a database of the system, an example object description and a set of example object attributes for an example object; modifying the example object description by replacing one selected example object attribute in the example object description with an annotation in accordance with the defined format; modifying the set of example object attributes by removing the one selected example object attribute; and generating the at least one example to include the modified set of example object attributes and the modified example object description.

9 . The system of claim 1 , wherein the processing unit is configured to execute computer-readable instructions to further cause the system to provide the prompt to the LLM as a set of tokens.

10 . The system of claim 1 , wherein the LLM is a trained generative LLM.

11 . The system of claim 1 , wherein the prompt to the LLM includes instructions to generate a product description for a product, and the generated description is the generated product description.

12 . The system of claim 11 , wherein the generated product description is used to update a product page related to the product.

13 . The system of claim 12 , wherein the generated product description is used to update the product page related to the product responsive to an approval received from a user device.

14 . A method comprising:
generating a prompt to a large language model (LLM) to generate a description of an object, the prompt including one or more object attributes to include in the generated description, and also including an instruction for the LLM to annotate, according to a defined format, any portions of the generated description that include unsubstantiated information;
providing the prompt to the LLM;
causing the LLM to generate the generated description; receiving the generated description;
parsing the generated description to identify, based on the defined format, one or more annotated portions indicating unsubstantiated information; and
presenting the generated description for display via a user device.

15 . The method of claim 14 , wherein the generated description is presented in a user interface (UI) in which at least one of the identified one or more annotated portions is modifiable.

16 . The method of claim 15 , further comprising, prior to presenting the UI:
identifying, for a given one annotated portion in the generated description, an unsubstantiated object attribute; querying a database to search for the unsubstantiated object attribute; modifying the generated description by replacing the given one annotated portion with a found object attribute received in a response to the query; and presenting the modified generated description in the UI.

17 . The method of claim 16 , wherein there is a plurality of found object attributes received in the response to the query, and wherein the generated description is modified by replacing the given one annotated portion with a UI element for selecting one of the plurality of found object attributes.

18 . The method of claim 17 , further comprising:
identifying at least two annotated portions in the generated description requiring a same user input; providing one input field for receiving user input to edit the at least two annotated portions; and updating the at least two annotated portions with information inputted in the one input field.

19 . The method of claim 14 , wherein the prompt to the LLM includes at least one example of an annotation according to the defined format.

20 . The method of claim 19 , wherein the at least one example is generated by:
retrieving, from a database, an example object description and a set of example object attributes for an example object; modifying the example object description by replacing one selected example object attribute in the example object description with an annotation in accordance with the defined format; modifying the set of example object attributes by removing the one selected example object attribute; and generating the at least one example to include the modified set of example object attributes and the modified example object description.

21 . The method of claim 14 , wherein the prompt to the LLM includes instructions to generate a product description for a product, and the generated description is the generated product description.

22 . The method of claim 21 , wherein the generated product description is used to update a product page related to the product.

23 . The method of claim 22 , wherein the generated product description is used to update the product page related to the product responsive to an approval received from a user device.

24 . A non-transitory computer readable med

Assignees

• Shopify Inc

Inventors

• Maschmeyer Russ
• Beauchamp Daniel

Classifications

• G06F40/40

  Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) · CPC title

• G06F40/205 Primary

  Parsing · CPC title

• G06F16/3328

  using graphical result space presentation or visualisation · CPC title

• G06F40/169 Primary

  Annotation, e.g. comment data or footnotes · CPC title