Generative summaries for search results

Patent metadata

Field	Value
Publication number	US-11769017-B1
Application number	US-202318123861-A
Country	US
Kind code	B1
Filing date	Mar 20, 2023
Priority date	Dec 30, 2022
Publication date	Sep 26, 2023
Grant date	Sep 26, 2023

Abstract

At least selectively utilizing a large language model (LLM) in generating a natural language (NL) based summary to be rendered in response to a query. In some implementations, in generating the NL based summary additional content is processed using the LLM. The additional content is in addition to query content of the query itself and, in generating the NL based summary, can be processed using the LLM and along with the query content—or even independent of the query content. Processing the additional content can, for example, mitigate occurrences of the NL based summary including inaccuracies and/or can mitigate occurrences of the NL based summary being over-specified and/or under-specified.

First claim

What is claimed is:

1. A method implemented by one or more processors, the method comprising:
receiving a query associated with a client device;
generating large language model (LLM) output based on processing input using an LLM, the input being based on the query and/or corresponding content from one or more search result documents that are responsive to the query;
generating a natural language (NL) based summary using the LLM output;
causing the NL based summary to be rendered, at the client device, along with corresponding links to one or more of the search result documents that are responsive to the query;
determining, subsequent to the NL based summary being rendered at the client device, occurrence of an interaction with a given search result document of the search result documents;
based on determining the occurrence of the interaction with the given search result document:
generating revised LLM output based on processing revised input using the LLM or an additional LLM, the revised input reflecting occurrence of the interaction with the given search result document, and the input not reflecting the occurrence of the interaction with the given search result document;
generating a revised NL based summary using the revised LLM output; and
causing the revised NL based summary to be rendered at the client device.

2. The method of claim 1 , wherein the interaction with the given search result document includes viewing of at least part of the given search result document.

3. The method of claim 2 , wherein the interaction with the given search result document includes viewing of at least part of the given search result document for at least a threshold duration of time.

4. The method of claim 2 , wherein causing the revised NL based summary to be rendered at the client device comprises causing the revised NL based summary to, after rendering of at least part of the given search result document at the client device, supplant the NL based summary in a graphical interface that initially rendered the NL based summary.

5. The method of claim 2 , wherein the NL based summary is rendered, along with the search result document, during at least an initial duration of the interaction with the given search result document, and wherein causing the revised NL based summary to be rendered at the client device comprises causing the revised NL based summary to, after at least the initial duration of the interaction with the given search result document, supplant the NL based summary.

6. The method of claim 2 , wherein causing the revised NL based summary to be rendered at the client device comprises causing the revised NL based summary to be rendered responsive to receiving, after rendering of at least part of the given search result document on the client device, another occurrence of the query in association with the client device.

7. The method of claim 2 , wherein causing the revised NL based summary to be rendered at the client device comprises causing the revised NL based summary to be rendered responsive to receiving, after the viewing of at least part of the given search result document, an additional query that is formulated based on user interface input at the client device and that is determined to be similar to the query.

8. The method of claim 1 ,
wherein generating the revised LLM output comprises processing the revised input using the LLM, wherein the revised input comprises a revised prompt that reflects familiarity with given content of the given search result document; and
wherein the input, processed using the LLM to generate the LLM output, lacks any prompt that reflects familiarity with given content of the given search result document.

9. The method of claim 1 ,
wherein generating the revised LLM output comprises processing the revised input using the additional LLM, wherein the additional LLM is fine-tuned based on a prompt that reflects familiarity with content; and
wherein the LLM is not fine-tuned based on any prompt that reflects familiarity with content.

10. The method of claim 1 ,
wherein the input, processed using the LLM in generating the LLM output, is based on the corresponding content from the one or more search result documents that are responsive to the query.

11. The method of claim 10 , further comprising:
selecting the one or more search result documents, wherein selecting the one or more search result documents including selecting, for inclusion among the one or more search result documents, a plurality of query-responsive search result documents based on the query-responsive search result documents being responsive to the query.

12. The method of claim 11 ,
wherein selecting the one or more search result documents including selecting, for inclusion among the one or more search result documents, a related-query-responsive search result document based on the related-query-responsive search result document being responsive to a related query determined to have a correlation to the query.

13. A method implemented by one or more processors, the method comprising:
receiving a query associated with a client device;
selecting a set of search result documents, selecting the set of search result documents including selecting, for inclusion in the set, a plurality of query-responsive search result documents based on the query-responsive search result documents being responsive to the query;
generating large language model (LLM) output based on processing, using an LLM, corresponding content from each of the search result documents of the set; and
generating a natural language (NL) based summary using the LLM output;
generating a confidence measure for a portion of the NL based summary; and
causing the NL based summary to be rendered at the client device in response to the query, including causing the portion of the NL based summary to be rendered with a given confidence annotation, of a plurality of candidate annotations, based on the given confidence annotation corresponding to the confidence measure for the portion of the NL based summary.

14. The method of claim 13 , wherein selecting, for inclusion in the set, the query-responsive search result documents includes selecting the query-responsive search result documents from a superset of search result documents that are responsive to the query, and based on one or more corresponding features of each of the query-responsive search result documents.

15. The method of claim 14 , wherein the one or more corresponding features of each of the query-responsive search result documents include one or more query-dependent measures for the query-responsive search result document, one or more query-independent measures for the query-responsive search result document, and/or one or more user-dependent measures for the query-responsive search result document.

16. The method of claim 14 , wherein selecting the set of search result documents further comprises:
selecting, for inclusion in the set, a related-query-responsive search result document based on the related-query-responsive search result document being responsive to a related query determined to have a correlation to the query.

17. The method of claim 16 , wherein selecting the related-query-responsive search result document is based on determining that a magnitude of the correlation satisfies a threshold.

18. The method of claim 13 , further comprising:
causing a selectable link, to a given document of the search result documents of the set, to be rendered at the client device along with rendering of the NL based summ

Assignees

• Google Llc

Inventors

• Gray Matthew K
• Blitzer John
• Herrick Corinn
• Venkatachary Srinivasan
• Madhavan Jayant
• Oates Sam
• Parakh Phiroze
• Shah Aditya
• Rofouei Mahsan
• Badr Ibrahim

Classifications

• G06F40/40 Primary

  Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) · CPC title

• G06F16/3328

  using graphical result space presentation or visualisation · CPC title

• G06F40/56 Primary

  Natural language generation · CPC title

• G06F16/345

  Summarisation for human users · CPC title

• G06F16/3344

  using natural language analysis · CPC title