This study aimed to develop Natural Language Processing (NLP) approaches to supplement manual outcome validation, specifically to validate pneumonia cases from chest radiograph reports.We trained one NLP system, ONYX, using radiograph reports from children and adults that were previously manually reviewed. We then assessed its validity on a test set of 5000 reports. We aimed to substantially decrease manual review, not replace it entirely, and so, we classified reports as follows: (1) consistent with pneumonia; (2) inconsistent with pneumonia; or (3) requiring manual review because of complex features. We developed processes tailored either to optimize accuracy or to minimize manual review. Using logistic regression, we jointly modeled sensitivity and specificity of ONYX in relation to patient age, comorbidity, and care setting. We estimated positive and negative predictive value (PPV and NPV) assuming pneumonia prevalence in the source data.Tailored for accuracy, ONYX identified 25% of reports as requiring manual review (34% of true pneumonias and 18% of non-pneumonias). For the remainder, ONYX's sensitivity was 92% (95% CI 90-93%), specificity 87% (86-88%), PPV 74% (72-76%), and NPV 96% (96-97%). Tailored to minimize manual review, ONYX classified 12% as needing manual review. For the remainder, ONYX had sensitivity 75% (72-77%), specificity 95% (94-96%), PPV 86% (83-88%), and NPV 91% (90-91%).For pneumonia validation, ONYX can replace almost 90% of manual review while maintaining low to moderate misclassification rates. It can be tailored for different outcomes and study needs and thus warrants exploration in other settings. Copyright ? 2013 John Wiley & Sons, Ltd.
展开▼
机译：这项研究旨在开发自然语言处理（NLP）方法以补充人工结局验证，特别是从胸部X光片报告中验证肺炎病例。我们使用以前手动检查过的儿童和成人的X光片报告训练了一种NLP系统ONYX。然后，我们在5000个报告的测试集中评估了其有效性。我们的目标是大幅减少人工检查，而不是完全取代人工检查，因此，我们将报告分类如下：（1）与肺炎一致; （2）与肺炎不一致;或（3）由于功能复杂而需要人工审核。我们开发了量身定制的流程，以优化准确性或最大程度地减少人工检查。使用逻辑回归，我们共同模拟了ONYX的敏感性和特异性与患者年龄，合并症和护理环境的关系。我们在源数据中假设肺炎患病率时估计了阳性和阴性预测值（PPV和NPV），为求准确性，ONYX确认25％的报告需要人工检查（34％的真实肺炎和18％的非肺炎）。在其他情况下，ONYX的敏感性为92％（95％CI 90-93％），特异性87％（86-88％），PPV 74％（72-76％）和NPV 96％（96-97％）。为了最大程度地减少人工审核，ONYX将12％归类为需要人工审核。在其他情况下，ONYX的敏感性为75％（72-77％），特异性为95％（94-96％），PPV为86％（83-88％），NPV为91％（90-91％）。 ，ONYX可以取代几乎90％的人工审核，同时保持较低至中等的误分类率。可以针对不同的结果和学习需求进行量身定制，因此值得在其他环境中进行探索。版权？ 2013 John Wiley＆Sons，Ltd.
展开▼
