A Human-in-the-Loop Method for Annotation of Events in Biomedical Signals.

OBJECTIVE

Building large-scale data bases of biomedical signal recordings for training artificial-intelligence systems involves substantial human effort in data processing and annotation. In the case of event detection, experts need to exhaustively scroll through the recordings and highlight events of interest.

METHODS

We propose an iterative annotation support algorithm with a human in the loop to improve the efficiency of the annotation process. Our algorithm generates proposal events based on an event detection model trained on incomplete annotations. The human only needs to verify candidate events proposed by the tool instead of scrolling through the entire data set. Our algorithm iterates between proposal generation and verification to leverage the human-in-the-loop feedback to obtain a growing set of event annotations.

RESULTS

Our algorithm finds a substantial amount of events at a fraction of the human time spent when comparing with a benchmark method and the normal manual process, finding all events in one data set and 70% of events in another with the human-in-the-loop only viewing 20% of the data.

CONCLUSION

Our results show that combining human and computer effort can substantially speed up the annotation process for events in biomedical signal processing.

SIGNIFICANCE

Due to its simplicity and minimal reliance on task-specific information, our algorithm is broadly applicable, unlocking substantial improvements in the scalability and efficiency of biomedical signal annotation.