Interactive assistance for executing natural language queries to data sets

Patent metadata

Field	Value
Publication number	US-12007988-B2
Application number	US-202318182303-A
Country	US
Kind code	B2
Filing date	Mar 10, 2023
Priority date	Mar 31, 2021
Publication date	Jun 11, 2024
Grant date	Jun 11, 2024

Abstract

Interactive assistances for executing natural language queries to data sets may be performed. A natural language query may be received. Candidate entity linkages may be determined between an entity recognized in the natural language query and columns in data sets. The candidate linkages may be ranked according to confidence scores which may be evaluated to detect ambiguity for an entity linkage. Candidate entity linkages may be provided to a user via an interface to select an entity linkage to use as part of completing the natural language query.

First claim

What is claimed is:

1. A system, comprising:
one or more processors; and
a memory, that stores program instructions that, when executed by the at least one processor, cause the one or more processors to implement a business intelligence service, the business intelligence service configured to:
receive a natural language query via an graphical interface for the business intelligence service that provides access to a plurality of different data sets;
identify, using one or more machine learning models trained for natural language processing, a plurality of candidate entity linkages between an entity recognized in the natural language query and respective columns from one or more of the plurality of different data sets;
responsive to a determination that respective confidence scores determined by the one or more machine learning models for the candidate entity linkages identified for the entity are not above a minimum confidence threshold:
underline, in the graphical interface, the entity in the natural language query to indicate that the entity is ambiguous; and
provide, in the graphical interface, a prompt to resolve the ambiguity of the entity before a result of the natural language query is provided.

2. The system of claim 1 , wherein the prompt comprises one or more of the plurality of candidate entity linkages as selectable suggestions.

3. The system of claim 1 , wherein the business intelligence service is further configured to:
execute the natural language query to return a first result via the graphical interface before the prompt is provided; and
execute the natural language query again to return a second result via the graphical interface based on a response to the prompt received via the graphical interface.

4. The system of claim 1 , wherein the business intelligence service is further configured to execute the natural language query to return a result via the graphical interface based on a response to the prompt received via the graphical interface.

5. The system of claim 4 , wherein the result comprises a visualization.

6. The system of claim 4 , wherein the result comprises a restatement of the natural language query.

7. The system of claim 1 , wherein the business intelligence service is further configured to provide via the interface one or more names for one or more data sets as part of an auto completion of the natural language query.

8. A method, comprising:
receiving a natural language query via an graphical interface for a business intelligence service that provides access to a plurality of different data sets;
identifying, by the business intelligence service using one or more machine learning models trained for natural language processing, a plurality of candidate entity linkages between an entity recognized in the natural language query and respective columns from one or more of the plurality of different data sets;
responsive to a determination that respective confidence scores determined by the one or more machine learning models for the candidate entity linkages identified for the entity are not above a minimum confidence threshold:
underlining, in the graphical interface, the entity in the natural language query to indicate that the entity is ambiguous; and
providing, in the graphical interface, a prompt to resolve the ambiguity of the entity before a result of the natural language query is provided.

9. The method of claim 8 , wherein the prompt comprises one or more of the plurality of candidate entity linkages as selectable suggestions.

10. The method of claim 8 , further comprising:
executing, by the business intelligence service, the natural language query to return a first result via the graphical interface before the prompt is provided; and
executing, by the business intelligence service, the natural language query again to return a second result via the graphical interface based on a response to the prompt received via the graphical interface.

11. The method of claim 8 , further comprising executing, by the business intelligence service, the natural language query to return a result via the graphical interface based on a response to the prompt received via the graphical interface.

12. The method of claim 11 , wherein the result comprises a visualization.

13. The method of claim 11 , wherein the result comprises a restatement of the natural language query.

14. The method of claim 8 , further comprising providing via the interface one or more names for one or more data sets as part of an auto completion of the natural language query.

15. One or more non-transitory computer-readable storage media storing program instructions that, when executed on or across one or more computing devices, cause the one or more computing devices to implement:
receiving a natural language query via an graphical interface for a business intelligence service that provides access to a plurality of different data sets;
identifying, using one or more machine learning models trained for natural language processing, a plurality of candidate entity linkages between an entity recognized in the natural language query and respective columns from one or more of the plurality of different data sets;
responsive to a determination that respective confidence scores determined by the one or more machine learning models for the candidate entity linkages identified for the entity are not above a minimum confidence threshold:
underlining, in the graphical interface, the entity in the natural language query to indicate that the entity is ambiguous; and
providing, in the graphical interface, a prompt to resolve the ambiguity of the entity before a result of the natural language query is provided.

16. The one or more non-transitory computer-readable storage media of claim 15 , wherein the prompt comprises one or more of the plurality of candidate entity linkages as selectable suggestions.

17. The one or more non-transitory computer-readable storage media of claim 15 , storing further program instructions that when executed on or across the one or more computing devices, cause the one or more computing devices to further implement:
executing the natural language query to return a first result via the graphical interface before the prompt is provided; and
executing the natural language query again to return a second result via the graphical interface based on a response to the prompt received via the graphical interface.

18. The one or more non-transitory computer-readable storage media of claim 15 , storing further program instructions that when executed on or across the one or more computing devices, cause the one or more computing devices to further implement executing, by the business intelligence service, the natural language query to return a result via the graphical interface based on a response to the prompt received via the graphical interface.

19. The one or more non-transitory computer-readable storage media of claim 18 , wherein the result comprises a visualization.

20. The one or more non-transitory computer-readable storage media of claim 15 , storing further program instructions that when executed on or across the one or more computing devices, cause the one or more computing devices to further implement providing via the interface one or more names for one or more data sets as part of an auto completion of the natural language query.

Assignees

• Amazon Tech Inc

Inventors

• Nallapati Ramesh M
• Wang Zhiguo
• Xiang Bing
• Ng Patrick
• Wang Yung Haw
• Karnik Mukul
• Li Nanyan
• Revadigar Sharanabasappa Parashuram
• Jones Timothy
• Ash Stephen Michael
• Sengupta Sudipta
• Adams Gregory David
• Murthy Deepak Shantha
• Cerny Douglas Scott
• Weeks Stephanie
• Li Hanbo

Classifications

• G06N20/00

  Machine learning · CPC title

• G06F40/295 Primary

  Named entity recognition · CPC title

• G06F16/2423

  Interactive query statement specification based on a database schema · CPC title

• G06F16/243

  Natural language query formulation · CPC title

• G06N3/08

  Learning methods · CPC title