The FDA’s Center for Drug Evaluation and Research’s (CDER) Office of Surveillance and Epidemiology (OSE) has developed the Information Visualization Platform (InfoViP). InfoViP incorporates artificial intelligence (AI) and advanced visualizations to support OSE safety reviewer’s examination of data or content to discover deeper insights, make predictions, or generate recommendations - to support postmarket safety surveillance.

Oanh Dang, PharmD, Board Certified Pharmacotherapy Specialist, of OSE’s Regulatory Science Staff, shares how the Safety Surveillance tool demonstrates the value of AI for increasing the efficiency of FDA’s reviews and how InfoViP impacts public health.

Oanh, before we get into InfoViP, can you tell us about OSE’s safety surveillance activities?

A key aspect of OSE’s mission is to monitor and assess the safety of marketed drugs and therapeutic biologics. We work to detect, assess, prevent, and manage risks associated with medication use to advance public health. Our safety surveillance activities assist with the identification and evaluation of  safety concerns that are identified after a product is approved and many more people are using it than in the (pre-approval) clinical trial setting.

OSE uses multiple approaches and data sources to identify and assess adverse effects, unexpected or unintended experiences, or other adverse events associated with human use of a drug or therapeutic biologic. The FDA Adverse Event Reporting System (FAERS) is one tool that OSE safety reviewers use. FAERS contains millions of individual case safety reports (ICSR) including mandatory safety reports from regulated industry such as pharmaceutical companies and voluntary MedWatch reports from patients and health care professionals. I want to point out here that while pharmaceutical companies are required to report adverse events to FAERS, other stakeholders are not — and we appreciate everyone’s diligence in doing so. After the reports are submitted, OSE safety reviewers analyze ICSRs in FAERS to identify whether the reported adverse events might be caused by the drug or therapeutic biologic.

Why was there a need for InfoViP?

The volume of data we collect in FAERS is enormous. In recent years, FAERS has received over 2 million reports each year. As of 2021, FAERS had over 23 million ICSRs. The large and increasing volume of ICSRs makes it difficult for OSE pharmacovigilance safety reviewers to efficiently detect and evaluate safety concerns. We realize AI has the potential to help us in the review and analysis of the growing amount of ICSR data by increasing efficiency in the labor-intensive ICSR review and evaluation process. By increasing automation, we also hope to help standardize the process of ICSR evaluation, leading to general improvements in decision-making.

The combination of the increasing volume and complexity of data collected in FAERS contributes to the time it takes us to review ICSRs and apply our clinical expertise to evaluate them. For one, ICSR data contain written text narratives that require us to manually review them to assess the relationship between the adverse event and the drug. This written narrative can range from one sentence to multiple pages of text, and the quality of the information in text can vary substantially. Some ICSRs may have sufficient information while others lack clinically relevant details needed to assess causality. FAERS also contains duplicate ICSRs describing the same patient and adverse events that different people (e.g., patient, pharmaceutical company representatives) independently submit. These challenges led OSE to consider AI technologies.

How does InfoViP use Artificial Intelligence for Individual Case Safety Reports within FAERS?

InfoViP incorporates natural language processing (NLP) capabilities (the ability for a computer program to understand spoken and written human language), machine learning (ML) (a technique that can be used to design and train software algorithms to learn from and act on data*), and advanced data visualizations (images created from raw data to allow humans to process information). InfoViP was developed as a decision-support software tool - to support safety reviewers, analysts and decision makers in making better and faster decisions.

Safety reviewers manually review ICSR narratives to identify and extract clinical concepts relevant to post-market safety surveillance, including the drug, adverse effect, medical histories, and sequence of events. Now InfoViP’s NLP technology introduces the capability to automatically scan the narratives in ICSRs to find and visually display relevant clinical information. The NLP automatically identifies information from ICSRs to create a timeline of events, including when the drug was taken in relation to the adverse effect.   

High-quality ICSRs, i.e., those with sufficient clinical details to conduct a causality assessment between the drug and adverse event, help reviewers identify safety concerns more quickly. The current, manual process of finding high-quality ICSRs is time intensive. Whereas InfoViP’s ML approach reduces the time to find high-quality ICSRs because it can simultaneously look at multiple data points contained in each ICSR and classify them based on their level of information quality. Safety reviewers can then use this classification to triage high-quality ICSRs for priority review to detect safety concerns more rapidly.

The presence of duplicate ICSRs in the FAERS database results in falsely elevated report counts, which can give the impression that there are more unique reports of a safety concern than there actually are. One of the first steps in evaluating a safety concern is to identify and remove duplicates. The current, multi-step process of finding duplicates involves the manual identification of ICSRs that look similar followed by a comparison of each ICSRs’ narratives. On the other hand, InfoViP’s NLP-based “deduplication” algorithm can efficiently scan, extract, and compare numerous data points among a large group of ICSRs to detect duplicates automatically and present them to the safety reviewer for confirmation. We anticipate the integration of InfoViP with the new FDA FAERS 2 system will further improve the work process of FDA safety reviewers.

What insights can you share from CDER’s work implementing AI in a regulatory setting?

While we usually think of AI as something that helps us automate work, humans are still an essential part of AI tool development and its application in the regulatory setting. AI tools such as InfoViP are decision-support tools that help, but don’t replace, pharmacovigilance safety reviewers in their investigations of safety concerns that affect drugs and therapeutic biologics used by millions of people.

AI tool development involves thoughtful and frequent engagement and trust building with end users, and this was evident throughout the time we spent developing InfoViP. InfoViP development occurred with years of applied research and development and is an extension of work begun in the Center for Biologics Evaluation and Research (CBER) to summarize vaccine adverse event reports that is still being used to help with the review of reports submitted to CBER for COVID-19 vaccines. The work capitalizes on the expertise of collaborators at leading universities in close collaboration with the end users, OSE safety reviewers. Years of close collaboration and communication among InfoViP developers and subject matter expert end users were required to understand problems, incorporate feedback that met user-specific needs, and fit InfoViP into existing business workflows. For instance, when developing the deduplication algorithm, all end users were invited to interact with the AI tool and apply InfoViP’s deduplication algorithm to 26 ICSR datasets comprising over 10,000 ICSRs so they could see how the automated process could support their workflow.

In the regulatory setting, there is a need for AI tools to be explainable and have a “human-in-the-loop" component, so that human experts can understand and validate the quality of the AI tools’ outputs.  For instance, within InfoVIP human experts can readily locate the data from the ICSR that the AI tool extracted to verify the accuracy of the NLP-derived timeline of events and the classification of a report as high or low quality. We also found in our experience that the ICSR data elements used in InfoViP’s deduplication algorithm often align with those used by safety reviewers to manually find duplicate ICSRs. AI technologies support and strengthen InfoViP to develop insights from data for safety reviewers to consider, validate, and ultimately use, to inform decisions about the safety of drugs and therapeutic biologics. Furthermore, InfoViP supports a CDER strategic goal of improving drug and biologic operations through the application of AI technologies.
 

*US FDA.  Artificial Intelligence and Machine Learning (AI/ML) Software as a Medical Device Action Plan.