Centers for Disease Control and Prevention, Atlanta, GA, United States of America.
PLoS One. 2019 Sep 25;14(9):e0222907. doi: 10.1371/journal.pone.0222907. eCollection 2019.
The Centers for Disease Control and Prevention (CDC) coordinates a labor-intensive process to measure the prevalence of autism spectrum disorder (ASD) among children in the United States. Random forests methods have shown promise in speeding up this process, but they lag behind human classification accuracy by about 5%. We explore whether more recently available document classification algorithms can close this gap.
Using data gathered from a single surveillance site, we applied 8 supervised learning algorithms to predict whether children meet the case definition for ASD based solely on the words in their evaluations. We compared the algorithms' performance across 10 random train-test splits of the data, using classification accuracy, F1 score, and number of positive calls to evaluate their potential use for surveillance.
Across the 10 train-test cycles, the random forest and support vector machine with Naive Bayes features (NB-SVM) each achieved slightly more than 87% mean accuracy. The NB-SVM produced significantly more false negatives than false positives (P = 0.027), but the random forest did not, making its prevalence estimates very close to the true prevalence in the data. The best-performing neural network performed similarly to the random forest on both measures.
The random forest performed as well as more recently available models like the NB-SVM and the neural network, and it also produced good prevalence estimates. NB-SVM may not be a good candidate for use in a fully-automated surveillance workflow due to increased false negatives. More sophisticated algorithms, like hierarchical convolutional neural networks, may not be feasible to train due to characteristics of the data. Current algorithms might perform better if the data are abstracted and processed differently and if they take into account information about the children in addition to their evaluations.
Deep learning models performed similarly to traditional machine learning methods at predicting the clinician-assigned case status for CDC's autism surveillance system. While deep learning methods had limited benefit in this task, they may have applications in other surveillance systems.
疾病控制与预防中心（CDC）协调了一项劳动密集型工作，以衡量美国儿童自闭症谱系障碍（ASD）的流行率。随机森林方法已显示出在加快这一过程中的潜力，但它们的分类准确性比人类低约 5%。我们探讨了最近可用的文档分类算法是否可以缩小这一差距。
使用从单个监测站点收集的数据，我们应用了 8 种监督学习算法，仅根据评估中儿童的单词来预测他们是否符合 ASD 的病例定义。我们比较了算法在数据的 10 次随机训练-测试分割中的性能，使用分类准确性、F1 分数和阳性预测值来评估它们在监测中的潜在用途。
在 10 次训练-测试循环中，随机森林和带有朴素贝叶斯特征的支持向量机（NB-SVM）的平均准确率均略高于 87%。NB-SVM 产生的假阴性明显多于假阳性（P = 0.027），但随机森林没有，因此其患病率估计值非常接近数据中的真实患病率。表现最好的神经网络在这两个指标上的表现与随机森林相似。
随机森林的表现与最近可用的模型（如 NB-SVM 和神经网络）一样好，并且它也产生了良好的患病率估计值。由于假阴性的增加，NB-SVM 可能不是全自动监测工作流程的理想候选者。由于数据的特点，更复杂的算法（如层次卷积神经网络）可能无法训练。如果对数据进行抽象和处理，并考虑到儿童的信息，而不仅仅是他们的评估，那么当前的算法可能会表现得更好。
深度学习模型在预测 CDC 自闭症监测系统的临床医生分配病例状态方面的表现与传统机器学习方法相似。虽然深度学习方法在这项任务中没有带来很大的好处，但它们可能在其他监测系统中有应用。
