iMRMC: Software for the statistical analysis of multi-reader multi-case reader studies
Catalog of Regulatory Science Tools to Help Assess New Medical Devices
Technical Description
The primary objective of the iMRMC statistical software is to assist investigators with analyzing and sizing multi-reader multi-case (MRMC) reader studies that compare the difference in the area under Receiver Operating Characteristic curves (AUCs) from two modalities. The iMRMC application is a software package that includes simulation tools to characterize bias and variance of the MRMC variance estimates.
The core elements of this application include the ability to perform MRMC variance analysis and the ability to size an MRMC trial, including study designs that are not "fully-crossed."
- The core iMRMC application is a stand-alone, precompiled, license-free Java application and source code. It can be used in GUI mode or on the command line.
- There is also an R package that utilizes the core Java application. Examples for using the programs can be found in the R help files.
Intended Purpose
The iMRMC package analyzes data from MRMC studies. MRMC stands for Multiple Readers and Multiple Cases (MRMC). MRMC studies are often imaging studies where clinicians (readers) evaluate patient images (cases). The MRMC methods apply to any scenario in which clinicians interpret data to make decisions.
The iMRMC package calculates the reader-averaged area under the curve (AUC) of the receiver operating characteristic curve (ROC). AUC is a diagnostic performance measure. Additional functions analyze other endpoints, such as binary performance and score differences. This package also estimates variances, confidence intervals, and p-values. These uncertainty characteristics are needed for hypothesis tests to size and assess the efficacy of diagnostic imaging devices and computer aids, including artificial intelligence-based devices.
Many imaging studies are designed so that every reader reads every case in all modalities, or in a fully-crossed study. In this case, the data is cross-correlated, and we consider the readers and cases to be cross-correlated random effects. An MRMC analysis accounts for the variability and correlations from the readers and cases when estimating variances, confidence intervals, and p-values. In addition, the functions in this package can treat arbitrary study designs and studies with missing data, not just fully-crossed study designs.
The package permits industry statisticians to use a validated statistical analysis method without having to develop and validate it themselves.
Citation
Please cite these references when you use the iMRMC software.
- Please reference the use of this tool using https://doi.org/10.5281/zenodo.6628838.
- Gallas, B. D., Bandos, A., Samuelson, F., & Wagner, R. F. (2009). A framework for random-effects ROC analysis: Biases with the bootstrap and other variance estimators. Commun Stat A-Theory, 38(15), 2586–2603. https://doi.org/10.1080/03610920802610084
Testing
The tool has been characterized through simulations (bias and variance of the MRMC variance estimates) and was found to have similar characteristics as other methods in the literature. The simulations model the data collected in reader studies: the diagnostic scores from readers interpreting images. The reader-study data yields a reader-averaged AUC and an MRMC variance estimate for each simulated reader study. Repeating this many times using a Monte Carlo simulation allows for the estimation of the bias and variance of the MRMC variance estimates.
The testing is summarized in the following articles:
- Gallas, B. D. (2006). One-shot estimate of MRMC variance: AUC. Acad Radiol, 13(3), 353–362. https://doi.org/10.1016/j.acra.2005.11.030
- Original description of method and validation with simulations. Results comparable to jacknife resampling technique.
- Gallas, B. D., Pennello, G. A., & Myers, K. J. (2007). Multireader multicase variance analysis for binary data. Journal of the Optical Society of America. A, Optics, Image Science, and Vision, 24(12), B70-80. https://doi.org/10.1364/josaa.24.000b70
- Generalize method to binary performance measures.
- Gallas, B. D., & Brown, D. G. (2008). Reader studies for validation of CAD systems. Neural Networks Special Conference Issue, 21(2), 387–397. https://doi.org/10.1016/j.neunet.2007.12.013
- Generalize method to treat arbitrary study designs.
- Gallas, B. D., Bandos, A., Samuelson, F., & Wagner, R. F. (2009). A framework for random-effects ROC analysis: Biases with the bootstrap and other variance estimators. Commun Stat A-Theory, 38(15), 2586–2603. https://doi.org/10.1080/03610920802610084
- Provide framework for understanding method and comparing to other methods analytically and with simulations.
- Obuchowski, N. A., Gallas, B. D., & Hillis, S. L. (2012). Multi-Reader ROC studies with Split-Plot Designs: A Comparison of Statistical Methods. Academic Radiology, 19(12), 1508–1517. https://doi.org/10.1016/j.acra.2012.09.012
- This study compares the tool method to two alternative methods in the literature in collaboration with the developers of the two alternative methods.
- Gallas, B. D., Chen, W., Cole, E., Ochs, R., Petrick, N., Pisano, E. D., Sahiner, B., Samuelson, F. W., & Myers, K. J. (2019). Impact of prevalence and case distribution in lab-based diagnostic imaging studies. Journal of Medical Imaging, 6(1), 015501. https://doi.org/10.1117/1.JMI.6.1.015501
- Study that uses the software and related research methods and study designs in a large study. Supplementary materials include data and scripts to reproduce study results.
Limitations
The tool may produce negative variance estimates for studies where the dataset is small.
Supporting Doumentation
Tool websites:
- Primary: GitHub - DIDSR/iMRMC: iMRMC: Software to do multi-reader multi-case analysis of reader studies
- Secondary: CRAN - Package iMRMC (r-project.org)
- User manual for java app (PDF)
- User manual for R package
- FAQs (GitHub)
Supplementary materials:
Related FDA Product Codes
Related FDA product codes include:
- KPS: System, Tomography, Computed, Emission
- LLZ: System, Image Processing, Radiological
- PAA: Automated Breast Ultrasound
- POK: Computer-Assisted Diagnostic Software For Lesions Suspicious For Cancer
- QDQ: Radiological Computer Assisted Detection/Diagnosis Software For Lesions Suspicious For Cancer
- QPN: Software Algorithm Device To Assist Users In Digital Pathology
Related Work
- Chen, W., Gong, Q., Gallas, B.D. (2018). Paired split-plot designs of multireader multicase studies. Journal of Medical Imaging 5, 031410. https://doi.org/10.1117/1.JMI.5.3.031410
- Gallas, B.D., Chan, H.-P., D’Orsi, C.J., Dodd, L.E., Giger, M.L., Gur, D., Krupinski, E.A., Metz, C.E., Myers, K.J., Obuchowski, N.A., Sahiner, B., Toledano, A.Y., Zuley, M.L. (2012). Evaluating imaging and computer-aided detection and diagnosis devices at the FDA. Acad Radiol 19, 463–477. https://doi.org/10.1016/j.acra.2011.12.016
- Gallas, B. D., & Hillis, S. L. (2014). Generalized Roe and Metz ROC model: Analytic link between simulated decision scores and empirical AUC variances and covariances. J Med Img, 1(3), 031006. https://doi.org/10.1117/1.JMI.1.3.031006
Current research
- High-Throughput Truthing (HTT) project: NCI Hub - Group: eeDAP studies ~ Wiki: HTT - What is HTT?
- The goal of the High-Throughput Truthing (HTT) project is to produce a validation dataset established by pathologist annotations for artificial intelligence algorithms analyzing digital scans of pathology slides: data (images + annotations). We are pursuing the qualification of the final validation dataset as an FDA-qualified medical device development tool MDDT to become a high-value public resource that can be used in AI/ML algorithm submissions and guide others to develop quality validation datasets.
- Digital Pathology Program: Research on Digital Pathology Medical Devices
- Medical Imaging and Diagnostics Program: Research on Medical Imaging and Diagnostic Devices
Contact
Tool Reference
In addition to citing relevant publications, please reference the use of this tool using DOI: 10.5281/zenodo.6628838.