System and method to teach and evaluate image grading performance using prior learned expert knowledge base

Patent metadata

Field	Value
Publication number	US-10984674-B2
Application number	US-201715814590-A
Country	US
Kind code	B2
Filing date	Nov 16, 2017
Priority date	Mar 15, 2017
Publication date	Apr 20, 2021
Grant date	Apr 20, 2021

Abstract

A learning sub-system models search patterns of multiple experts in analyzing an image using a recurrent neural network (RNN) architecture, creates a knowledge base that models expert knowledge. A teaching sub-system teaches the search pattern captured by the RNN model and presents to a learning user the information for analyzing an image. The teaching sub-system determines the teaching image sequence based on a difficulty level identified using image features, audio cues, expert confidence and time taken by experts. An evaluation sub-system measures the learning user's performance in terms of search strategy that is evaluated against the RNN model and provides feedback on overall sequence followed by the learning user and time spent by the learning user on each region in the image.

First claim

We claim:

1. A method of teaching image analysis and evaluating analysis results, the method executed by at least one hardware processor, the method comprising:
retrieving an image from a database of images and presenting the image on a user interface displayed on a display device;
transmitting a signal to an eye tracker comprising at least a camera coupled to the hardware processor, the signal representing a notification to the eye tracker to monitor eye movements of a learning user analyzing the image and generating a sequence of eye movements based on the eye tracker monitoring the eye movements;
receiving via the user interface, annotations on the image input by the learning user;
receiving via a microphone coupled to the hardware processor, audio data associated with the image spoken by the learning user, and translating the audio data into text and extracting keywords from the text;
correlating the sequence of eye movements, the annotations and the keywords according to their time of occurrence;
extracting image features from the image and mapping the image features with the sequence of eye movements, the annotations and the keywords that are correlated;
generating a search pattern of the learning user based on the image features mapped with the sequence of eye movements, the annotations and the keywords that are correlated;
retrieving from a knowledgebase stored in a storage device, a recurrent neural network model that predicts a likelihood of an expert image analyzer focusing on a feature in the image and time spent by the expert image analyzer on the feature;
generating an expert's search pattern of the image by executing the recurrent neural network model, and displaying the expert's search pattern on the user interface while playing associated audio cues retrieved from the knowledgebase, and further zooming in the feature predicted by the recurrent neural network model on the user interface.

2. The method of claim 1 , further comprising measuring the search pattern of the learning user based on the expert's search pattern of the image.

3. The method of claim 2 , further comprising evaluating a time spent on the feature by the learning user against the time spent by the expert image analyzer on the feature.

4. The method of claim 3 , further including providing the learning user with feedback comprising a learning user's perceived importance of different regions in the image based on the search pattern of the learning user.

5. The method of claim 1 , wherein the learning user is allowed a fixed amount of time to analyze the image and input the annotations on the image and input the audio data.

6. The method of claim 5 , further comprising allocating a session identifier to associate with a session comprising the learning user analyzing the image during the fixed amount of time.

7. The method of claim 4 , wherein the feedback further includes that the learning user identified landmarks in the image.

8. The method of claim 1 , wherein the image includes a retinal image.

9. The method of claim 1 , wherein the displaying of the expert's search pattern further includes automatically highlighting a region in the image, which is labeled as being interesting.

10. The method of claim 1 , wherein the recurrent neural network model is trained based on training data including at least image filters learned via a convolutional neural network.

11. The method of claim 2 , further including retrieving a next image from the database of images for presenting on the user interface, wherein the next image is selected based on the user's performance measured against the expert's search pattern of the image.

Assignees

• IBM

Inventors

• Garnavi Rahil
• Mahapatra Dwarikanath
• Roy Pallab K
• Tennakoon Ruwan B

Classifications

• G06V10/7788

  the supervisor being a human, e.g. interactive learning with a human teacher · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06N3/042

  Knowledge-based neural networks; Logical representations of neural networks · CPC title

• G06F18/41

  Interactive pattern learning with a human teacher · CPC title