Cognitive horizon surveillance

Patent metadata

Field	Value
Publication number	US-11663273-B2
Application number	US-202016917638-A
Country	US
Kind code	B2
Filing date	Jun 30, 2020
Priority date	Jun 30, 2020
Publication date	May 30, 2023
Grant date	May 30, 2023

Abstract

A method for ranking relevance of documents includes using a set of queries, searching a corpus of documents for a set of candidate documents with information relevant to the set of queries. The method further includes ranking the set of candidate documents by a deep learning processing system according to relevance to respective ones of the set of queries. The method additionally includes responsive to user input, revising the ranked set of candidate documents to produce a revised ranked set of candidate documents. The method further includes using the revised ranked set of candidate documents to retrain the deep learning processing system. The method still further includes performing a categorization of the set of candidate documents by the retrained deep learning processing system.

First claim

What is claimed is:

1. A method for ranking relevance of documents comprising:
using a set of queries, searching a corpus of documents for a set of candidate documents with information relevant to the set of queries, wherein the set of queries comprise keywords;
ranking the set of candidate documents by a deep learning processing system, wherein the ranking comprises:
for each document of the set of candidate documents:
identifying relevant locations in a text of the document by searching the keywords in the text;
determining at least one local relevance of at least one keyword of the keywords by capturing at least one context where the at least one keyword is used; and
assigning a ranking score to the document by aggregating the at least one local relevance;

responsive to user input relating to selections via a user interface for user preferences, revising the ranked set of candidate documents to produce a revised ranked set of candidate documents;
using the revised ranked set of candidate documents to retrain the deep learning processing system including feeding back adjudication from the user input for improvement of performance of the deep learning processing system for a next iteration of processing by rejecting a majority of the set of candidate documents; and
performing a categorization of the set of candidate documents by the retrained deep learning processing system, wherein the categorization comprises displaying a ranked list of different categories that are most likely for a candidate document selected using user input from the user interface, and assigning at least one of the different categories selected using further user input from the user interface to the candidate document, and wherein the ranked list provided via the user interface includes, for each category of the different categories, a visual indication of a likelihood, predicted by the retrained deep learning processing system, that the category is suitable for the candidate document, and a sort button that sorts the different categories based on each score corresponding to each category of the different categories that is indicative of a likelihood, predicted by the retrained deep learning processing system, that the category is suitable for the candidate document.

2. The method of claim 1 , wherein the deep learning processing system finds documents to reject for determining candidate documents for automatic query refinement.

3. The method of claim 1 , further comprising:
updating the set of queries based on a periodic schedule; and
expanding the set of queries by utilizing a dictionary to find additional keywords that match some of the keywords, wherein the expanded set of queries includes the additional keywords.

4. The method of claim 1 , wherein the categorization is based according to a set of user defined categories, the user defined categories are displayed in the ranked list, and background for candidate documents are displayed with the ranked list.

5. The method of claim 4 , further comprising:
responsive to user input, revising the set of user defined categories.

6. The method of claim 1 , further comprising:
responsive to user input, selecting one of the set of queries; and
displaying, using a user interface, categorized documents which satisfy the selected query.

7. The method of claim 1 , wherein the method is repeated on a periodic basis, such that the deep learning processing system is continually retrained.

8. A computer program product for ranking relevance of documents, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to:
using a set of queries, search, by the processor, a corpus of documents for a set of candidate documents with information relevant to the set of queries, wherein the set of queries comprise keywords;
rank, by the processor, the set of candidate documents by a deep learning processing system, wherein the ranking comprises:
for each document of the set of candidate documents:
identifying relevant locations in a text of the document by searching the keywords in the text;
determining at least one local relevance of at least one keyword of the keywords by capturing at least one context where the at least one keyword is used; and
assigning a ranking score to the document by aggregating the at least one local relevance;

responsive to user input relating to selections via a user interface for user preferences, revise, by the processor, the ranked set of candidate documents to produce a revised ranked set of candidate documents;
using the revised ranked set of candidate documents, retrain, by the processor, the deep learning processing system including feeding back adjudication from the user input for improvement of performance of the deep learning processing system for a next iteration of processing by rejecting a majority of the set of candidate documents; and
perform, by the processor, a categorization of the set of candidate documents by the retrained deep learning processing system using further user input from the user interface, wherein the categorization comprises displaying a ranked list of different categories that are most likely for a candidate document selected using user input from the user interface, and assigning at least one of the different categories selected using further user input from the user interface to the candidate document, and wherein the ranked list provided via the user interface includes, for each category of the different categories, a visual indication of a likelihood, predicted by the retrained deep learning processing system, that the category is suitable for the candidate document, and a sort button that sorts the different categories based on each score corresponding to each category of the different categories that is indicative of a likelihood, predicted by the retrained deep learning processing system, that the category is suitable for the candidate document.

9. The computer program product of claim 8 , wherein the deep learning processing system finds documents to reject for determining candidate documents for automatic query refinement.

10. The computer program product of claim 8 , wherein the program instructions executable by the processor further cause the processor to:
update, by the processor, the set of queries based on a periodic schedule; and
expand, by the processor, the set of queries by utilizing a dictionary to find additional keywords that match some of the keywords, wherein the expanded set of queries includes the additional keywords.

11. The computer program product of claim 8 , wherein the categorization is based according to a set of user defined categories, the user defined categories are displayed in the ranked list, and background for candidate documents are displayed with the ranked list.

12. The computer program product of claim 11 , wherein the program instructions executable by the processor further cause the processor to:
responsive to user input, revise, by the processor, the set of user defined categories.

13. The computer program product of claim 8 , wherein:
the program instructions executable by the processor further cause the processor to:
responsive to user input, select, by the processor, one of the set of queries; and
using a user interface, display, by the processor, categorized documents which satisfy the selected query.

14. The computer program product of claim 8 , wherein the program instructions are repeated on a periodic basis, such that the deep learning processing sys

Assignees

• IBM

Inventors

• Gruhl Daniel
• Kato Linda Ha
• Ristoski Petar
• Welch Steven R
• Deluca Chad Eric
• Gentile Anna Lisa
• Alba Alfredo
• Zubarev Dmitry
• Narayan Chandrasekhar
• Park Nathaniel H

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/091

  Active learning · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06F16/24578 Primary

  using ranking · CPC title