Automatically evaluating likely accuracy of event annotations in field data

Patent metadata

Field	Value
Publication number	US-10114807-B2
Application number	US-201313964811-A
Country	US
Kind code	B2
Filing date	Aug 12, 2013
Priority date	Aug 10, 2012
Publication date	Oct 30, 2018
Grant date	Oct 30, 2018

Abstract

Embodiments operate in contexts where field data have been generated from a field event, and annotations have been generated from the field data, which purport to identify events within the field data, such as CPR compressions and ventilations. Metrics are generated from the annotations, which are used in training. In such contexts, a grade may be assigned that reflects how well the annotations meet one or more accuracy criteria. The grade may be used in a number of ways. Reviewers may opt to disregard field data and metrics that have a low grade. Expert annotators may be guided as to precisely which annotations to revise, saving time. A low grade may decide that the results are not emailed to reviewers, but to annotators. A learning medical device can use the grade internally to adjust its own internal parameters so as to improve its annotating algorithms.

First claim

What is claimed is:

1. A device comprising:
a processor; and
a non-transitory storage medium communicatively coupled to the processor, the storage medium configured to store one or more programs which, when executed by the processor, cause the device to:
receive field data, derived from a Cardio Pulmonary Resuscitation (CPR) session, including events of at least two different types occurring to a patient over time, the event types including at least one of chest compressions and ventilations within the CPR session;
receive annotations that have been previously generated from field data, the annotations identifying at least some of the events;
calculate at least one of a relative timing of at least two events of the field data based on the annotations identifying the at least two events;
obtain at least one accuracy criterion, the accuracy criterion indicating an expected event sequence and the relative timing of the at least two events;
compute at least one accuracy score for the annotations based on the accuracy criterion;
assign, out of a plurality of possible grades, at least one grade based on the accuracy score, the assigned grade indicating an accuracy with which the annotations identify the events identified by the annotations, and
output a user signal that includes the at least one grade for the annotations.

2. The device of claim 1 , in which
the field data is arranged along a time dimension, and
the annotations include events identified as occurring in sequence at different times along the time dimension.

3. The device of claim 1 , in which
the events result in repeating patterns within the field data.

4. The device of claim 1 , in which
assigning the at least one grade is accomplished by selecting one of a plurality of bands with boundaries that contain the accuracy score, and assigning a grade corresponding to the selected band.

5. The device of claim 1 , in which
the grade is encoded to be heard with a sound different than respective sounds of the other possible grades.

6. The device of claim 1 , in which
the grade is encoded to appear with a color or an image different than respective colors or images of the other possible grades.

7. The device of claim 1 , in which
the user signal also includes, in association with the grade, metrics that are computed from the annotations.

8. The device of claim 1 , in which executing the one or more programs further causes the device to:
receive adjusted annotations generated from the field data after the annotations;
assign an adjusted grade reflecting an accuracy with which the adjusted annotations identify events in the field data based on the accuracy criterion; and
output a user signal that encodes the adjusted grade.

9. The device of claim 1 , in which executing the one or more
programs further causes the device to:
receive an override input; and
output a user signal that encodes a new grade different from the assigned grade responsive to receiving the override input.

10. The device of claim 1 , in which
a flag is issued when at least one aspect of the annotations does not meet the at least one accuracy criterion, and
the grade is assigned based on a number of issued flags.

11. The device of claim 10 , in which
a flag is issued when at least one aspect of the annotations does not meet a plurality of accuracy criteria.

12. The device of claim 10 , in which
the at least one accuracy criterion specifies that no event is missing in a series of successive events.

13. The device of claim 12 , in which
the grade and the accuracy score are computed based on a number of identified events and a number of expected events.

14. The device of claim 10 , in which
the annotations include events identified as occurring in sequence at different times, and
the at least one accuracy criterion specifies that no isolated identified event occurs after an idle interval of a first duration and before another idle interval of a second duration.

15. The device of claim 10 , in which
the annotations include events identified as occurring in sequence at different times, and
the at least one accuracy criterion specifies that identified events do not occur faster than a maximum repetition rate, or slower than a minimum repetition rate.

16. The device of claim 10 , in which
the annotations include events identified as occurring in sequence at different times,
an average time interval between the occurrences of two successive events is determined, and
the at least one accuracy criterion specifies that an event should be followed with a pause in an expected time interval that is determined by the average time interval.

17. The device of claim 10 , in which
a flag is further issued when at least one aspect of the field data does not meet at least one validity criterion.

18. The device of claim 17 , in which
the at least one validity criterion specifies that the field data represents a signal that is larger than a threshold.

19. A method comprising:
receiving field, derived from a Cardio Pulmonary Resuscitation (CPR) session, including events of at least two different types occurring to a patient over time, the event types including at least one of chest compressions and ventilations within the CPR session;
receiving annotations that have been previously generated from field data, the annotations identifying at least some of the events;
calculate at least one of a relative timing of at least two events of the field data based on the annotations identifying the at least two events;
obtain at least one accuracy criterion, the accuracy criterion indicating an expected event sequence and the relative timing of the at least two events;
computing at least one accuracy score for the annotations based on the accuracy criterion;
assigning, out of a plurality of possible grades, at least one grade based on the accuracy score, the assigned grade indicating an accuracy with which the annotations identify events identified by the annotations; and
outputting a user signal that includes the at least one grade for the annotations.

20. A non-transitory computer-readable storage medium storing one or more programs which, when executed by at least one device, they result in:
receiving field data, derived from a Cardin Pulmonary Resuscitation (CPR) session, including events occurring to a patient over time, the events including at least one of chest compressions and ventilations within the CPR session;
receiving annotations that have been previously generated from field data, the annotations identifying at least some of the events;
calculate at least one of a relative timing of at least two events of the field data based on the annotations identifying the at least two events;
obtain at least one accuracy criterion, the accuracy criterion indicating an expected event sequence and the relative timing of the at least two events;
assigning, out of a plurality of possible grades, at least one grade for the annotations based on the accuracy criterion by comparing the field data associated with the annotations with the expected event sequence, the assigned grade indicating an accuracy with which the annotations identify events identified by the annotations; and
outputting a user signal that encodes the grade.

21. A medical device, comprising:
a sensor configured to sense one or more patient parameters used to sense field data about a patient, the field data derived from a Cardio Pulmonary Resuscitation (CPR) sessi

Assignees

• Physio Control Inc

Inventors

• Esibov Alexander
• Banville Isabelle
• Marx Robert
• Chapman Fred
• Merry Randy

Classifications

• G06F40/169 Primary

  Annotation, e.g. comment data or footnotes · CPC title

• A61N1/39044

  in combination with cardiopulmonary resuscitation [CPR] therapy · CPC title

• G06F17/27

  Physics · mapped topic

• G06F17/241 Primary

  Physics · mapped topic