A Report by the CFSAN Risk Analysis Working Group

March 2002

RECOMMENDATIONS

The CFSAN Risk Analysis Working Group recommends the following:

Risk analysis is a powerful tool that should be used to enhance the scientific basis of regulatory decisions. It should be conducted within CFSAN through the efforts of risk assessment, risk management, and risk communication teams. Risk assessment should be conducted in an iterative manner that allows refinement of the risk assessment question(s), key assumptions, and data used in the model. The exchange of information (communication) within and between the risk analysis teams, with other agencies, and stakeholders (including industry, consumer groups, and other interested parties) should be encouraged by active participation in the process and collaboration, when appropriate.

To support and promote the use of a risk analysis framework for initiating and conducting 'major' risk assessments, CFSAN should:

1. Adopt a decision-based approach to identify and select risk assessments conducted by CFSAN, particularly those that are 'major' (complex and impact or involve multiple offices). Available resources, regulatory needs, and public health concerns should be considered in the selection of risk assessments. This approach should be implemented for microbial risk assessments now, and later expanded to include chemical and other non-microbial hazards.
2. Establish a procedure for the conduct of risk assessment within a risk analysis framework. The procedure should identify the boundaries and responsibilities of key participants in the risk analysis process.
3. Develop criteria to evaluate the quality of data used for risk assessments and specify what information is needed to scientifically evaluate the usefulness of a study or data set used for risk assessment.
4. Develop guidelines to evaluate risk assessments and supporting data developed by stakeholders and submitted to the Center.
5. Formalize a peer review process that will encourage critical review and evaluation of CFSAN's risk assessments by government and non-government experts in a manner that improves the science and acceptance of complex risk assessments.
6. Build capacity to conduct complex risk assessments by providing training opportunities for current staff, hiring new staff or using contractors (as needed), and acquiring additional resources such as computers, software, and dedicated workspace.

---

CONTRIBUTORS

CFSAN Risk Analysis Working Group Members

Co-Chairs:

Robert Buchanan, Senior Science Advisor, Director Office of Science

Janice F. Oliver, Deputy Director CFSAN

Group Facilitator:

Susan Santos, FOCUS GROUP Risk Communication and Environmental

Management Consultants

Working Group Members:

Philip Bolger, OPDFB

Karen Carson, OPDFB

Marjorie Davidson, FSS

Sherri Dennis, OSci

Patricia Hansen, OSci

Kathy Gombas, OFP

Donald Kraemer, OS

John Kvenberg, OFP

Wesley Long, OSci

Marianne Miliotis, OARSA

Arthur Miller, OSci

Patricia Schwartz, OS

Philip Spiller, OS

Terry Troxell, OPDFB

Richard Whiting, OPDFB

Richard Williams, OSAS

Working Group Meeting Organizers:

Catherine DeRoever, OSci

Linda Hayden, OSci

Carloyn Jeletic, OSci

Acknowledgments

In the development of a risk analysis framework for the initiation and conduct of risk assessments, we evaluated and adopted concepts and processes that have been described by others. As appropriate, the principles of risk analysis developed by other federal agencies and international organizations were adapted to be compatible to the specific requirements for CFSAN.

The CFSAN Risk Analysis Working Group thanks the following for their contributions to this effort:

Catherine Bailey, EOS
Mary Bender, ONPLDS
John Bowers, OSAS
Donald Burr, OARSA
Richard Canady, OPDFB
Clark Carrington, OPDFB
Anthony Hitchins, OPDFB
Jennifer Cleveland, visiting scientist from Rutgers University
Ensook Jhon, visiting scientist from the Korea Food and Drug Administration
Clark Nardinelli, OSAS
Richard Raybourne, OARSA
Marianne Ross, OSAS
Mark Walderhaug, OPDFB

Special thanks to Dr. Ensook Jhon for her expertise in collecting and evaluating various approaches to identifying priority risk assessments and assisting in the development of the decision-based approach described in this document.

We are especially grateful to the Listeria monocytogenes and Vibrio parahaemolyticus risk assessment teams for freely sharing their experiences in conducting risk assessments and for reviewing and commenting on early drafts of this document.

The group also extends its appreciation to Susan Santos for her expertise in facilitating the working group meetings and helping us find common ground needed to develop a framework and for her thoughtful comments and suggestions to improve earlier drafts of this document.

This is a working document and it will be revised as needed, based on the continuing experience using these procedures for risk assessment in a risk analysis framework.

---

Table of Contents

• Background
• Part I: Overview of the CFSAN Risk Analysis Framework
• Part II A Decision-Based Approach to Identify and Select Risk Assessments
• Part III Conducting Risk Assessment within the Risk Analysis Framework
• Part IV: Conclusions, Recommendations, and Next Steps
• Appendix A: Components of a Risk Assessment
• Appendix B: Example Questions: Communication Between the Risk Managers and Risk Assessors
• Appendix C: Risk Communication Activities
• Appendix D: Criteria for Identifying Candidate Risk Assessments
• Appendix E: Criteria for Evaluating Technical Feasibility

Background

Charter: CFSAN's Risk Analysis Working Group.

A working group was formed at the request of the Center for Food Safety and Applied Nutrition (CFSAN) senior management to examine lessons learned from conducting quantitative risk assessments. Dr. Susan Santos, FOCUS GROUP Risk Communication and Environmental Management Consultants, facilitated the work and direction of this group.

The overall goal of the CFSAN Risk Analysis Working Group was to improve the quality and consistency of risk assessments conducted within the Center. Specific tasks of the working group were as follows.

• Identify boundaries and responsibilities of key participants in the risk analysis process.
• Develop a process for identifying and selecting risk assessments conducted within the Center.
• Establish procedures for the conduct of risk assessments within a risk analysis framework.

The CFSAN Risk Analysis Working Group discussed and developed proposals for each of these tasks. Most importantly, the group supported the concept that CFSAN's risk assessments should be conducted within a risk analysis framework and developed a list of guiding principles (see below). Specific recommendations for implementing this framework were proposed including: a decision-based approach for selecting risk assessments; a step-by-step procedure for conducting the risk assessment; development of criteria to evaluate the quality of data; development of guidelines to evaluate risk assessments; formalizing a process for review of assessments, and building capacity to conduct risk assessments. These recommendations are provided on page ii.

This document provides a systematic overview of the procedures needed to initiate and conduct risk assessments within a risk analysis framework. Part I of this document is a description of the CFSAN risk analysis framework, including roles and responsibilities of participants. In Part II, the proposed decision-based approach to identify and select risk assessments to be conducted by CFSAN is described in detail. Procedures and activities needed to plan, perform, review, and issue/ publish risk assessments within the risk analysis framework are described in Part III. The implementation of this framework is discussed in Part IV.

The specific roles and responsibilities of risk managers and risk communicators as they relate to risk assessment activities are also described in this document. However, other risk analysis activities of the risk management and risk communication that are conducted independent of the risk assessment are not within the scope of this document. For example, it was not the intent of this document to explain how risk managers develop an action plan to evaluate and propose changes needed to implement or redirect the Center's current regulatory strategies.

While the Risk Analysis Working Group specifically wanted to build on the lessons learned during the conduct of the Listeria monocytogenes, Vibrio parahaemolyticus, and methyl mercury risk assessments it was the group's expectation that the developed procedures should be used by the Center for all hazards, including chemical. However, it was felt that this process was generally only necessary for 'major' risk assessments.

What are 'Major' Risk Assessments?

As defined by CFSAN's Risk Analysis Working Group, all 'major' risk assessments include those that are:

• non-routine and complex;
• may be qualitative or quantitative;
• involve multiple program offices and are cross-cutting in nature; and
• require a commitment of significant resources to complete.

This definition of 'major' risk assessments would not include a variety of routine safety/risk assessments, such as those currently conducted for food additives or botanicals because these are generally conducted within or for a single CFSAN office. It would also not include activities the CFSAN Cancer Assessment Committee (CAC) and the Quantitative Risk Assessment Committee (QRAC). The activities of these standing committees are described in their charter and standard operating practices (SOP) documents.

The following guiding principles were developed based on the discussions during the work group meetings.

Guiding Principles--Risk Assessment within a Risk Analysis Framework

1. Risk analysis is a valuable tool for CFSAN to use to enhance the scientific basis of regulatory decisions. All 'major' risk assessments conducted by CFSAN should be performed within the risk analysis framework. Using this framework, project goals are accomplished through the efforts of risk assessment, risk management, and risk communication teams.
2. Risk assessment of complex topics should be conducted in an iterative manner that allows refinement of the risk assessment question(s), key assumptions, and data used in the model. An iterative process requires active participation and collaboration among the various risk analysis teams and with other interested parties.
3. An open exchange of information and ideas (communication) within and among the risk assessment, risk management, and risk communication teams is critical for successful conduct of risk assessment projects. As such, stakeholders including consumer groups, industry, and other agencies should be identified early in the process and communication should occur frequently with them during the assessment.
4. Risk assessments conducted by CFSAN should be identified and selected using a decision-based approach that considers the appropriate use of the Center's resources to provide the best scientific analysis to solve high priority public health problems.
5. Resources needed to effectively and efficiently conduct risk assessments must be identified and allocated prior to commissioning the assessment.
6. Realistic timeframes with intermediate milestones must be established and agreed to by all participants when the risk assessment is commissioned. Timeframes must consider the iterative nature of risk assessments.
7. The risk assessment should be as simple as possible while providing risk managers with the information needed to make decisions.
8. The risk assessment process must be transparent. All assumptions, data, and decisions that impact the risk assessment conclusions and risk management actions must be clearly documented and shared with interested parties.
9. Draft risk assessment documents and models must undergo peer review by government and non-government experts using a process that allows for extensive evaluation and critical review of the assessment.
10. The Center must continue to build capacity to conduct risk assessments through research, training, hiring, and other means such as using contractors.

back to table of contents

Part I: Overview of the CFSAN Risk Analysis Framework

In this section of the document, risk analysis is defined and the characteristics that describe CFSAN's approach to conducting high quality risk assessments are outlined. The organization of the risk analysis teams and the risk assessment activities as well as the interaction of risk assessors with risk management and risk communication teams are described. A brief description of the processes associated with risk assessment including planning, performing, reviewing, and issuing/publication a risk assessment is provided.

What is Risk Analysis?

Risk analysis is a tool to enhance the scientific basis of regulatory decisions. It includes risk assessment, risk management and risk communication activities. As shown in Figure I-1, risk analysis is often illustrated using three circles, equal in size and overlapping to indicate both independent and dependent activities of the three risk analysis components. Each component has unique responsibilities:

• Risk assessment provides information on the extent and characteristics of the risk attributed to a hazard.
• Risk management includes the activities undertaken to control the hazard.
• Risk communication involves an exchange of information and opinion concerning risk and risk-related factors among the risk assessors, risk managers, and other interested parties.

Figure I-1. The Three Components of Risk Analysis

Characteristics of CFSAN's Risk Assessments

The key terms that describe CFSAN's approach to conducting the highest quality risk assessments possible are: transparent, team-oriented, and iterative. These characteristics should also be applied to the overall risk analysis activities.

Transparent. Transparency includes stating any biases that impact the risk assessment, clearly and concisely documenting the assessment, and using a participatory process. Transparency includes clearly stating all assumptions used in the assessment, providing the scientific rationale, and documenting the data used to estimate the impact of the various factors influencing the risk.

• Reveal biases. All assumptions used in the assessment, and the scientific rationale and data used to estimate the impact of the various factors influencing the risk must be clearly stated. Transparency ensures that biases will be eliminated or minimized, and that any introduced biases are clearly identified.
• Clarity. Transparency includes ensuring the document is clear and understandable. In attempting to provide all of the information used in the risk assessment model, the resulting document tends to be lengthy and very technical. Documents that are so technical that only a few experts can understand them lack transparency. Therefore, CFSAN prepares an interpretive summary document as a companion to each technical risk assessment. The summary document provides a non-technical explanation of the assessment process, results, and conclusions in a manner that a non-scientist can understand.
• Participatory process. The process for initiating, performing and finalizing a risk assessment should also be transparent. CFSAN invites public comment on planned assessments and encourages stakeholders to submit scientific data and information to support the assessment. The advice and opinions of advisory committees are solicited as well as peer review from experts within and outside of the Center. As part of its commitment to transparency and use of the best information available, CFSAN makes the draft risk assessment documents available to the public for technical review and comment. Assessments are posted on the CFSAN web pages and printed copies are also available through CFSAN's Outreach and Information Center.

Team-oriented. Quantitative risk assessments are too complex to be conducted by a single person. They require the input and critical evaluation of experts in a number of scientific disciplines including microbiology, chemistry, epidemiology, medicine, mathematics, statistics, toxicology, and food science, i.e., a multidisciplinary team.

Iterative. Risk assessment for complex topics should be an iterative process, and this iteration will require communication and collaboration among the various risk analysis teams and other interested individuals.

As additional scientific knowledge about the hazard, additional exposure or dose-response data, or improved modeling techniques become available, the assessment and its conclusions may have to be reevaluated or updated. Risk assessments can always be improved and the uncertainty in the model may be reduced when new data or modeling techniques become available. On the other hand, this makes it difficult to know when to stop and give the results to the risk manager. A general guideline is that the risk assessment should be kept as simple as possible, while providing the risk managers with the information they need to make decisions.

Similarly, the risk assessor often must choose between assumptions or models to use to describe exposure or dose-response. The risk management team should be consulted regarding influential choices. They may be able to make all necessary risk management decisions despite this model uncertainty, or they may choose to engage other resources to clarify the choice before proceeding further with the risk assessment (e.g., directed research, expert panels).

Risk analysis should also be conducted and thought of as a dynamic, iterative process. Key elements of successful risk analyses are clear understanding of the questions to be addressed by the risk assessors and wide acceptance of the assumptions used in risk modeling. Communication of the questions to be addressed often involves an initial dialogue and framing of the issues followed by clarification and elaboration once screening-level risk analysis has been carried out.

The iterative nature of risk analysis is illustrated in Figure I-2 by the use of double-headed arrows between risk management, risk assessment, and risk communication teams. The iterative nature of risk analysis might include a situation where:

1. The answer provided by a risk assessment results in a new risk management question; or
2. A risk management question (or new data) stimulates a new risk assessment model.

Note: Because a model may influence the experimental design and the type of data collected, research also has an important role in risk analysis, as noted in the section below entitled, "The Role of Research."

The double-headed arrows shown in Figure I-2 imply a dialogue among the risk analysis teams, in contrast to a monologue (i.e., from risk assessors to risk managers). In a monologue-driven process, the risk management decision is conceived to be one that progresses in a single direction from analysis to policy. In a dialogue, policy is determined based on information exchanged between the risk assessors and the risk managers.

By focusing on the risk assessor—risk manager—risk communicator dialog, the risk assessment may be designed to evaluate the effectiveness of a putative action. That is, the risk management question is not just about the magnitude of the anticipated harm - it is also about how we might mitigate the risk.

Figure I-2. Risk Analysis is an Iterative Process

Other Desired Characteristics of Risk Assessment

Structured. The generally accepted paradigm for microbial, as well as chemical food contaminates, includes separating the assessment activities into four components: hazard identification, exposure assessment, dose-response assessment, and risk characterization. See Appendix A for a description of these components.

Descriptive. An important part of risk assessment is determining the degree of uncertainty in relation to the results and distinguishing this from the variation that is inherent in any biological system. The accuracy of a risk assessment is dependent on the quality of the available data. When definitive data are lacking, the uncertainty about the available information is represented in the risk assessment using a range of possible data values. One way to decrease uncertainty is to conduct research to provide improved data. However, this does not mean that if one collects enough data that a single, "right" value will be reached. One of the challenges in using a risk analysis approach is that an "accurate" risk assessment captures the variability inherent in the food safety system instead of generating a single value.

Flexible. The risk assessment models should be flexible, such that they can be easily revised when new data or information become available.

Based on sound science. The risk assessment should be based on sound science that is decision driven and supported by systematic analysis that maintain integrity and protects the risk assessment from political and other pressure. The standards for sound science differ across different types of research. The risk assessment team should specify what it considers sound or competent science for each type of research that is used in the risk assessment.

The Role of Research. A risk assessment should be based on sound science. Where data are lacking, assumptions must be made. Risk managers must provide a major contribution to those assumptions. There is a high likelihood that a risk assessment will raise uncertainties that might be addressed through further research. At times a risk management decision can be postponed until further research can be conducted, but it is more likely that the issues that gave rise to the present assessment are likely to persist or rise again. However, a risk assessment can be useful in planning the research that will have the most impact on future regulatory decisions. This relationship is shown in Figure I-3 with the placement of research (science) between risk assessment and risk management.

Figure I-3. Role of Research in Risk Analysis

Risk assessments bring together research from many fields, including:

• Microbiology
• Toxicology
• Medicine
• Food Technology
• Epidemiology
• Behavioral Science
• Environmental Sciences
• Mathematics
• Statistics
• Economics
• Chemistry

Supporting research could be conducted before, during, and after any quantitative risk assessment. Research carried out before the risk assessment could be the reason for the request for a risk assessment. For example, an outbreak of a particular food-borne illness may lead to epidemiological and medical research that points to the need for a risk assessment. This same early research will also play a large role in the hazard identification stage of the risk assessment. Research on environmental, technological, and behavioral sources of risk could all be part of the hazard identification.

During the conduct of the risk assessment, risk assessors will most likely discover missing information and request research. For example, the risk assessors may be unable to estimate exposure to a foodborne hazard without additional research on food consumption or food handling practices. They may ask for a survey on consumer practices to get the desired information. Or they may discover that the lethality of the time and temperature treatments of one path from processor to consumer is not well known. The risk assessors would then ask for laboratory research to simulate the properties of the path in order to estimate the lethality of the actual path. Yet another important type of research is on potential risk mitigations. The risk assessors might ask for research on the technical feasibility of some proposals for mitigation. They might ask for research comparing the lethality of chemicals, heat treatments, and other mitigations. Another line of research might be to ask economists to consider the incentives that industry or consumers have to voluntarily adopt the mitigations.

After the risk assessment is complete, several types of research might be in order. Economists might be asked to evaluate or identify the costs associated with risk mitigations identified by the risk assessment. The study of strategic behavior (mathematical game theory), for example, might prove useful in predicting how producers will react to regulations based on the risk assessment. Epidemiological studies, perhaps in conjunction with statistical and mathematical studies, could confirm or refute the range of predictions of the risk assessment. As new methods of hazard control appear, research will be needed to incorporate the potential mitigations into the risk assessment.

Research provides data and information needed to carry out assessments; risk assessments provide information risk managers need to make decisions. The link between research, risk assessment, and policy decisions means that decisions on the amount and type of research should be tied to decisions on the amount and type of information provided by the risk assessment. At various stages of the risk assessment risk managers will consult risk assessors to see what information is lacking. The risk assessment team should let risk management know how much each additional type of research can be expected to reduce uncertainties in the risk assessment, or how it will fill needs for information after the risk assessment. The risk manager will then determine if the research should be carried out. How much and what types of research and when to carry out that research, will depend on the importance of the policy that the risk assessment is designed to support.

Risk Analysis Teams

Risk analysis activities are accomplished through the efforts of risk management, risk assessment and risk communication teams. The risk analysis teams are functionally separate but interact on a frequent and regular basis.

• The Risk Management Team (RM) is responsible for formulating the risk assessment questions to be addressed, providing key assumptions and oversight of the assessment, and developing a management action plan.
• The Risk Assessment Team (RA) is responsible for conducting the assessment, and refining as necessary, the assumptions provided by the risk management team, explaining the uncertainty of the results and the impact of assumptions on the results.
• The Risk Communication Team (RC) is responsible for providing input to the RA and RM teams based on the identification and understanding of stakeholder concerns, information needs, and perceptions; promoting an ongoing exchange of information about the project with all interested parties; and developing public health messages based on the assessment results and management plans.

A team leader should be identified for each of the risk analysis teams. This person is responsible for ensuring that all activities of the team are conducted in a timely manner. The team leader(s) should act to ensure that the risk analysis process is conducted in an iterative manner, including actively seeking participation and collaboration among the team members and with other interested parties.

In addition to the three risk analysis teams, CFSAN's risk analysis framework includes three unique positions: a science advisor, a risk analysis coordinator, and a risk assessment project manager.

Science Advisor. The role of the science advisor is to ensure that the science of the assessment is not compromised by the policy needs of the risk management team. The science advisor must act to preserve credibility and transparency in all decisions and is responsible for resolving any science vs. policy issues.

Risk Analysis Coordinator. This individual assists the risk management, assessment, and communication teams by coordinating and managing the activities and helping to facilitate communication within and among these groups.

Risk Assessment Project Manager. This individual is responsible for administrative and technical management of the process. In contrast, the risk assessment team leader is mainly responsible for the technical conduct of the assessment. For risk assessments that are more limited in scope and complexity, a single individual might function as both the risk assessment project manager and the risk assessment team leader.

The relationships and interaction of the risk analysis teams, the science advisor, the risk analysis coordinator, and the risk assessment project manager are indicted in Figure I-4.

Figure I-4. Organization of CFSAN's Risk Analysis Components (the dark arrows indicate specific points of contact and interaction between two components; the shaded arrows indicate dialog among the teams)

Roles and Responsibilities

The various roles and responsibilities of participants in risk analysis are summarized in Table I-1.

Boundaries between the Risk Analysis Teams

One difficulty in risk analysis is delineating the functional boundaries between assessors, managers, and communicators. A key aspect is that boundaries must be maintained but functions should be interdependent so as to provide appropriate checks and balances to the process. By clearly defining responsibilities of the risk analysis participants, CFSAN believes that appropriate boundaries can be maintained. The risk analysis coordinator should assist the risk analysis participants in understanding their respective roles and responsibilities.

While the goal of a risk assessment is to assist risk managers in making a decision, it is only one of many sources of information that CFSAN's risk managers use to make policy decisions. As such, it is the risk managers who must set the objectives of the risk assessment and provide the overall scope, including key assumptions, which should be used in the absence of data. It is, however, the responsibility of the risk assessors to do the following:

• ensure that the assessment is of high scientific quality and consistent with existing scientific practices for the conduct of risk assessments;
• ensure that the assessment is responsive to the needs of risk management;
• ensure that risk managers understand the impact or limitations of available data on the outcome of the risk assessment; and
• scribe how assumptions affect the results of the assessment including how they affect the level of uncertainty associated with the risk estimate.

The risk analysis teams must be willing to exchange information and ideas and collaborate on decisions. In this way, the key objectives, modeling assumptions, and data used in the model will be refined as more information is gathered and exchanged. A continual dialog helps to ensure that the risk managers' expectations about the use of the risk assessment results are met and are understood by all participants. It also means that the risk assessors must ensure that the risk management team members understand the limitations of the assessment.

The following are the general areas or topics that the risk analysis teams should discuss at various points in the process.

During the Planning of the Risk Assessment

• The risk management question and its significance to the specific risk assessment question(s) and how the assessment answers relate to the risk management options
• How the teams should interact and at which specific points in the assessment process
• Interaction and communications with other interested parties, including other agencies
• The type of risk assessment (quantitative vs. qualitative; risk ranking, product pathway, risk-risk) and whether this is the best approach to answering the questions
• Resources needed and timeline for conducting the assessment

During the Conduct of the Risk Assessment

• A conceptual diagram of the major steps of the risk assessment model
• Method of collecting data, criteria for choosing which data to include in the assessment, how the data are used in the risk assessment model
• Specific data inputs and outputs for the conceptual model
• Use of key assumptions provided by risk managers
• Identification of modeling assumptions
• Criteria used to select the various distributions/models used
• Sensitivity of assumptions and impact on conclusions
• Methods of presenting of the results

During the Risk Analysis Team Review of the Risk Assessment

• Answer(s) to the risk assessment question(s)
• Uncertainty/sensitivity analysis
• Data gaps or research needed to refine the risk assessment
• Transparency of the document

Examples of topics (posed as questions) that the risk analysis team members should discuss are provided in Appendix B. The questions in Appendix B are not meant to be a prescriptive list but rather a list of possible topics designed to encourage a meaningful exchange of information.

To be effective, risk communication must include an exchange of information by all interested parties ensuring that the process and results of any risk analysis are considered transparent and are trusted by outside audiences. For this trust to occur, the information exchange must be encouraged and actively promoted throughout the risk analysis process, particularly during the planning, conduct, review and issuing phases of the risk assessment. A summary of specific risk communication activities conducted within a risk analysis framework is provided in Appendix C. The need for these activities is mentioned in other sections of this document but the focus of Appendix C is the specific activities that are the responsibility of the risk communication team.

Risk Analysis Activities

CFSAN's risk analysis framework includes identification/selection, planning, performance, review, and issuing/publication activities. Responsible parties for each risk analysis activity are shown in Table I-2. The identification and selection process is described in Part II of this document. The activities directly related to the conduct of the risk assessment (planning, performing, reviewing, and issuing) once it is commissioned are described in detail in Part III of this document. The specific goals and end products of the activities that are the responsibility of the risk assessors are described below and summarized in Table I-3.

RA= risk assessment team; RM = risk management team; and

RC = risk communication team.

Because risk assessments are time and resource intensive, the goal of the selection process is to identify a critical regulatory concern that lacks a risk management plan and could benefit from conducting a risk assessment. The process includes the formulation of specific risk assessment questions to be answered and an evaluation of the feasibility of conducting the risk assessment. The selection process uses a decision-based approach to ensure that any risk assessment conducted by CFSAN will meet the Center's priority needs and ensure that adequate resources are available to successfully complete the assessment within the specified timeframe.

During the planning phase, the specific resources needed are allocated, a timeline established, and a charge (questions for the risk assessment to answer) are prepared. Prior to commissioning the assessment, the members for each team should be identified and the teams assembled.

After the risk management team defines the risk assessment questions, the various risk analysis teams are formed and the risk assessment is performed. The risk assessment team collects and assembles data and information, conducts the risk assessment and drafts the technical and interpretive summary documents. Based on the results of the risk assessment the risk management team develops an action plan and the risk communication team develops communication strategies including public health advice messages about the assessment and management action plan.

The risk assessment, management action plan, and communication messages must be reviewed. The review of the risk assessment may result in another iteration of the assessment, a redrafting of the document or both. Reviewing activities are completed when the documents are cleared by the Center and are ready to be released to the public. This step also includes peer review of the risk assessment documents by FDA staff, other agencies, and non-government individuals, as needed. A suitable timeframe should be allotted for the review process. Following the peer review process, the draft risk assessment documents will be issued for public review and comment.

Issuing (or publishing) the risk assessment documents is another activity in the risk analysis process. A roll-out plan is prepared and implemented. A Federal Register notice is drafted and published, a press contact list is compiled, and a public meeting is planned and held to present, discuss, and clarify the findings of the risk assessment. Following the public comment period, the risk assessment may be revised and a final document issued. The final risk assessment document must include a summary of the public comments and how they were addressed in the revised document.

Follow up is also important and included in this framework as part of issuing or publishing the risk assessment. It includes an internal evaluation of the effectiveness of the process and also a determination of future actions required and changes needed.

back to table of contents

Part II. A Decision-Based Approach to Identify and Select Risk Assessments

The first step in CFSAN's risk analysis framework is the identification and selection of a risk assessment. This process uses a decision-based approach and will be conducted in four phases. The purpose is to ensure that the selected risk assessment project will meet CFSAN's regulatory needs, and that sufficient data and adequate resources are available to conduct the assessment.

Goal: Select risk assessment(s) to be conducted by the Center

Product: Statement of risk management issue(s) and risk assessment question(s) to be answered

Activity: Identify and Select All 'Major' Risk Assessments

• Concept generation
• Problem identification
• Data feasibility determination (evaluation and recommendation)
• Disposition (selection)

The following is a description of a proposed process that uses a decision-based approach to identify and select risk assessments. The process is divided into four phases—concept generation, problem identification, data feasibility determination, and disposition. The purpose of a decision-based approach is to ensure that the candidate risk assessments are systematically evaluated based on the Center's regulatory needs and feasibility (resources and data availability. Figure II-1 provides an overview of the activities of each phase of this process.

Figure II-1. Proposed Process for Identifying and Selecting all 'Major' Risk Assessments Conducted by the Center

When is a Risk Assessment Needed?

A 'major' risk assessment requires a substantial commitment of CFSAN resources. Thus, risk assessment is not appropriate when CFSAN's risk managers do not need this level of sophistication to make a decision.

Circumstances that do not warrant a quantitative risk assessment would include:

• a risk that is well described by definitive data,
• a problem that is relatively simple, or
• an issue that is not of regulatory concern.

However, risk assessment is a powerful tool to help the risk managers evaluate and interpret information when:

• the data describing a hazard are incomplete,
• the exposure system is complex, or
• the issue is of high regulatory or stakeholder concern.

Specific examples of when risk assessments are needed in CFSAN may include the following:

• when a chemical fails the safety assessment threshold;
• when the current quantitative standard is not met or not justified;
• when the current standard is inconsistent with other government policies or guidelines;
• when the Center has been petitioned for a regulatory action;
• when a No-Observed Adverse Effect Level (NOAEL) cannot be established for a chemical;
• when a data gap analysis is needed; and
• when the hazard is a serious health issue, emerging pathogen and/or public health concern.

What Type of Risk Assessment is Needed?

The type of risk assessment appropriate for a specific risk management problem depends on the question to be answered and the availability of data. If adequate data are available, a quantitative risk assessment is possible, if fewer data are available qualitative and data gap analysis may be more appropriate.

Safety vs. Risk Assessments. This document is specific to the selection and conduct of risk assessments, not routine safety assessments. A difference is that risk assessment estimates risk and, in some cases, sources of risk and quantitative reductions based on various interventions. A safety assessment, on the other hand, provides a verdict of what is "safe" based on the conventions of the analysis.

Qualitative vs. Quantitative. Risk assessments can be either qualitative or quantitative in their description of the likelihood of adverse health effects, depending on the extent of the knowledge available, the complexity of the problem, and the time available to conduct the assessment. In quantitative assessments, the risk is expressed as a mathematical statement of the chance of illness or death after exposure to a specific hazard, and it represents the cumulative probabilities of certain events happening and the uncertainty associated with those events. Conversely, qualitative risk assessments use verbal descriptors of risk and severity, and often involve the aggregation of expert opinions.

The quantitative risk assessment technique used depends on the available data and on the scope and nature of the questions posed by the risk managers.

Types of Risk Assessments. Four general types of risk assessments likely to be used by the Center include risk ranking, product pathway, risk-risk and geographical. Below is a brief description of these types of assessments.

Risk ranking. Risk ranking assessments compare the relative risk among several hazards or foods. These types of assessment techniques might involve a single pathogen associated with multiple foods, a single food that has multiple pathogens, or multiple pathogens and multiple foods. Risk ranking assessments can help establish regulatory program priorities and identify the critical research needs. The FDA/USDA Listeria monocytogenes risk assessment is an example of a risk ranking assessment. Product pathway analyses. In product pathway assessments, the factors that influence the risk associated with specific food/hazard pairs are examined. Ideally it starts at the farm and ends with consumption. These types of assessment techniques help identify the key factors that affect exposure including the impact of potential mitigation or intervention strategies on the predicted risk. The FDA Vibrio parahaemolyticus risk assessment is an example of a product pathway analysis. Risk-risk. In risk-risk assessments, a trade off of one risk for another is considered, i.e., reducing the risk of one hazard increases the risk of another. An example of this would be a determination of the impact on public health by treating drinking water with a chemical (risk to chlorine exposure) vs. the impact of exposure to pathogenic organisms in water not treated. Geographical. In a geographic risk assessment, the factors that either limit or allow the risk to occur are examined. The risk of introduction of disease agents through food animals or animal products (e.g. intentionally as in bioterrorism or unintentionally) can be examined. For example, the risk of introduction of variant Creutzfeldt-Jacob Disease (vCJD) in humans by the transmission of BSE from cattle through meats and animal product pathways might be examined using a geographical approach.

Who Coordinates the Identification and Selection Process?

CFSAN's Risk Analysis Working Group recommends that the risk assessment coordinating staff within Office of Science (referred to as the coordinating staff in the discussion below) be responsible for coordinating the identification and selection process for all 'major' risk assessments. This approach should be implemented for microbial risk assessments and later expanded to include chemical and other non-microbial hazards.

How Are Candidate Risk Assessments Identified?

Ideas for risk assessments can be obtained from a number of sources including:

• Stakeholders (consumers, public, industry)
• Consumer advocacy groups
• Public health institutions
• Follow-up from a completed risk assessment
• Regulatory/Management staff
• Researchers
• Other government agencies

The CFSAN Risk Analysis Working Group intended that this process would be used to request and evaluate risk assessment ideas on an annual basis so that decisions can be made in time to include the selected risk assessments on the program priority list. However, this process can also be used to evaluate special needs that might arise during a fiscal year.

The coordinating staff will prepare an annual All-Hands Call to be issued electronically to CFSAN staff to request the submission of ideas for risk assessments. The submitted ideas will be shared with the impacted Program Offices for evaluation.

Currently there is no established mechanism for routinely or formally obtaining ideas for risk assessments from the public, other agencies, and shareholders. The CFSAN Risk Analysis Working Group recommends that options for obtaining this input be developed. In the meantime, any ideas that are submitted in an informal manner to the Center should be shared with the impacted Program Office and with the Office of Science's coordinating staff.

Four Phases of the Identification and Selection Process

Below is a detailed description of the four phases of the proposed identification and selection process. Figure II-2 illustrates how the process uses a decision-based approach. During each phase, decisions and recommendations are made and the candidate risk assessments are systematically evaluated.

Phase 1. Concept Generation

Responsible Party. Program Offices with assistance from the coordinating staff.

Activity. Throughout the fiscal year (in time to include on the CFSAN Program Priorities list, if possible), the coordinating staff maintains a list of potential risk assessments.

Information collected includes:

• specific hazards/commodities
• risk management question(s)/regulatory needs
• risk assessment question(s)

With assistance of the coordinating staff, the Program Office, group, or individual that suggests the candidate risk assessment prepares a justification of the scope of the problem. The justification should include sufficient information for a decision to be made on the need for this assessment, such as:

• purpose of the risk assessment
• scope of the problem (such as commodities, disease agent, impact on public health, distribution of risk in the population)
• importance and need for the Center (including current management practices to control the risk)
• how the assessment results would be used by the Center

On an annual basis (and as needed throughout the year), the coordinating staff requests the leadership team to review the list of candidate risk assessments and justifications for these assessments (see Phase 2, described below).

Objective. During Phase 1, the focus should be on the nature of the risks to be evaluated and not specifically a consideration of the data available for conducting the risk assessment, as that is the objective of Phase 3.

Criteria for Identifying Candidate Risk Assessments.

• characteristics and importance of the hazard of concern
• magnitude (presence, prevalence, concentration of hazards) and severity (impact on public health) of the risk
• urgency of the situation
• populations of concern
• other factors associated with specific hazards (such as processing, cooking, cross contamination)

Some example questions to consider when identifying hazards and developing justifications for the candidate risk assessment(s) are provided in Appendix D. Additional research may be needed to answer some of these questions. For example, preliminary epidemiological and medical research could shed light on the extent and severity of specific hazards. Surveys (especially rapid, internet-based surveys) could reveal whether the public considers something to be a health hazard.

Phase 2. Problem Identification

Responsible Party. CFSAN leadership and management teams

Activity. At a regularly scheduled weekly team meeting, the leadership team reviews the justifications for each candidate risk assessment for the purpose of whether the assessment meets CFSAN's needs. The leadership team makes a recommendation for each candidate assessment and provides the recommendation to the management team.

Example recommendations include:

• The candidate risk assessment should be conducted.
• The candidate risk assessment should not be conducted. Either because it is not relevant to CFSAN's mission or it is not needed to make a risk management decision.
• The need for this risk assessment could not be determined based on the information available for the candidate risk assessment. Either request the coordinating staff to provide additional information, or if new data are needed, determine the need for research to be conducted before a decision is made.

The management team reviews the leadership team recommendations and approves of candidate risk assessments to be further evaluated or conducted (see below, description of data feasibility determination, Phase 3) based on technical merit, resource availability, and other factors deemed appropriate.

Objective. Develop justification for conducting risk assessment. The resources needed to conduct the risk assessment should be considered in Phase 2, but a complete assessment of resource needs and available should be addressed in Phase 4.

Criteria. Same as above for Phase 1 plus consideration of risk management/ policy issues.

Phase 3. Data Feasibility Determination (Evaluation and Recommendation)

Responsible Party. Coordinating staff and short-term detailees, as needed.

Activity. Under the direction of the coordinating staff , short term detailees review published and unpublished literature to determine the availability of data needed to conduct a candidate risk assessment that will address the risk management question(s) and answer the corresponding risk assessment question(s). [In most cases, the detailees will be from the program office or group that requested the risk assessment.]

Using the results of the review of data availability, the coordinating staff prepares a recommendation concerning further actions.

Recommended actions may include:

• conduct a quantitative risk assessment
• conduct a qualitative risk assessment
• modify risk management and/or assessment question(s)
• more research needed [Short-term research projects to fill gaps in the information needed to determine feasibility could be commissioned, if necessary.]

The recommendation is provided to the management team. In Phase 4, the management team makes a decision for each candidate assessment.

Objective. Determine appropriate action, specifically whether the stated risk management questions can be answered through a risk assessment; if data are available; or if more research needs to be conducted.

Criteria for Data Feasibility Evaluation. The evaluation of the feasibility of conducting a risk assessment must consider the following:

• availability of data,
• the uncertainty created by lack of data, and
• other legitimate factors specific to individual hazards.

Examples of the type of information collected during a feasibility evaluation are provided in Appendix E. Developing a conceptual model (including overall structure, data inputs, and flow of calculations) may be helpful in determining the types of data and information needed.

Phase 4. Disposition (Selection)

Responsible Party. Management team

Activity. Each fiscal year (in time to include on the priority list if possible), select risk assessments to be conducted based on technical merit, data feasibility, resource availability, and other factors deemed appropriate by the Center.

Objective. Using the results of the recommendations from the feasibility determination (Phase 3) as an aid, the management team selects specific risk assessments meriting further action.

Further actions might include:

• request a specific risk assessment
• select a hazard for which additional research is needed
• determine whether a risk assessment should be reconsidered at a later date (i.e., when additional research is available or changes in priorities permit)
• conduct a more comprehensive or follow up risk assessment
• other appropriate actions

Example Criteria for Selecting Risk Assessments.

• technical merit
• Center's regulatory needs
• timeframe for making a risk management decision
• resource needed/available
• expertise needed
• data availability

Figure II-2. A proposed decision-based approach to identify and select 'major' risk assessments

back to table of contents

Part III. Conducting Risk Assessment within the Risk Analysis Framework

Once a risk assessment is selected as described in Part II of this document, the risk assessment is commissioned. Conducting a risk assessment within CFSAN's risk analysis framework includes the following steps--planning, performing, reviewing, and issuing. These steps, including activities and responsible parties for each, are described in detail in this part of the document.

Steps: Conducting the Risk Assessment

• Step 1: Planning the risk assessment
• Step 2: Performing the risk assessment
• Step 3: Reviewing the risk assessment document(s)
• Step 4: Issuing the risk assessment

This part of the framework document focuses on the activities directly related to the conduct of the risk assessment. However, it is important to understand the sequence of these activities as they relate to the responsibilities of risk managers and risk communicators. Figure III-1 shows the interrelationships of these risk analysis activities.

During the planning step, the risk assessment, risk management and risk communication teams are formed. The risk management team develops a 'charge' for the assessment, including the key assumptions that define the scope of the project. Once the risk assessment team receives the 'charge' they begin to conduct the risk assessment. During the performance step, there is regular interaction between the risk management and risk assessment teams to further refine the scope of the assessment, inform the risk management team of progress in completing the assessment, and eventually providing the results and a draft report for review. Also, during the performance step, the risk communication team is conducting an ongoing interaction with stakeholders to determine their concerns, perceptions, and information needs. During the review of the draft document, the risk managers may begin to develop an action plan and the risk communicators to initiate development of public health messages. Following the issue or publishing of the assessment, the teams should debrief and evaluate the effectiveness of the process.

Figure III-1. Summary of Risk Analysis Activities and Responsible Parties

Step 1. Planning the Risk Assessment

After a specific risk assessment is selected, the next step is to adequately plan the project. The purpose of planning is to identify project goals, resources, and participants. Planning ensures a sound foundation for the successful conduct of a risk assessment.

Goal: Develop scope and timeframe, allocate resources, and identify participants

Product: Assemble teams, risk assessment 'charge,' statement of resources, and schedule

Activities: Planning the Risk Assessment

• Identify participants and assemble the risk analysis teams
• Agreement of the scope, complexity, focus of the risk assessment
• Develop "charge" for the risk assessment, including key assumptions
• Allocate resources (technical and financial support)
• Develop a timeframe and timeline for initiating and completing the risk assessment
• Determine communication needs

Decisions needed during the planning step include:

• how the assessment will be coordinated within the risk assessment team, among the risk analysis teams, with other offices, agencies, and stakeholders
• how the risk assessment and risk management teams should communicate
• how and when to communicate with stakeholders
• how transparency in the process will be maintained (i.e., documentation)

Identify Participants/ Form Teams

Once a decision has been made to conduct a risk assessment, the participants should be identified; the risk management, assessment, and communication teams formed; and a lead assigned for each team. The team leads, risk assessment project manager, and risk analysis coordinator should meet and plan the risk assessment. Planning includes an agreement on the goals, scope, and timing of the risk assessment as well as resources needed to complete the assessment.

Table III-1 provides lists of the typical members of risk assessment, risk management, and risk communication teams. The risk assessment team should include subject matter leads on consumption, pathogen prevalence, food processing (if appropriate), dose-response, and model. Administrative support staff is also needed to provide administrative/clerical assistance such as filing and technical writing expertise. Pending the complexity of the assessment, a RM liaison might be beneficial. The subject matter leads will report to a technical team leader. The team leader must interact closely with the risk assessment project manager. The risk assessment project manager in turn implements the plans of the risk analysis coordinator.

CFSAN = Center for Food Safety and Applied Nutrition
ORA = Office of Regulatory Affairs
EOS = Executive Operations Staff
OCO = Office of Constituent Operations

The teams should be formed after the team leaders have been appointed, in consultation with the Risk Analysis Coordinator, the Risk Assessment Program Manager, and Program Office/Staff Directors.

Define Scope and Develop Charge

The Program Office or component of CFSAN that requests a risk assessment should meet with the risk assessment coordinating staff to develop a prospectus of what will be involved in the assessment. An estimate of personnel, time, resources, and external expertise is developed during these initial discussions.

The risk management team must develop a formal charge to the risk assessment team. This should include the management question to be addressed and specific questions that the risk assessment is to answer. This charge should be developed, circulated for comment, and published in the Federal Register with the intent and request for additional data to be submitted. The charge should include a statement of purpose, define the risk management problems, and the specific risk assessment questions to be answered. Key assumptions must also be identified in the charge.

Risk Management Problem versus Risk Assessment Question. Risk assessment is one tool that managers can use to solve or address a risk management problem or issue. The questions that risk managers must answer are different from those that risk assessors should be asked. An example risk management problem is:

"How should FDA manage the risk of contracting listeriosis from eating ready-to-eat (RTE) foods served in restaurants?"

The answer to the risk management problem is important for the risk assessor to know but not to answer. Based on the above example risk management problem, the risk management team might pose questions to the risk assessor, such as:

• "What is the exposure to Listeria monocytogenes from consuming RTE foods in restaurants?"
• "What is the likelihood of the general population (or to a sensitive subset of the population) contracting listeriosis from eating RTE foods in restaurants?"
• "How much is the risk reduced if FDA required 'use by dates' on RTE foods?"

What Should be Included in the risk assessment charge? The 'charge' to the risk assessment team is a critical document. It defines the overall scope of the problem and the importance of the issue to CFSAN. It should be as detailed as possible to guide the risk assessment efforts. The charge should at least include the following information:

• The specific risk management problem or question, including the scope (assumptions/objectives)
• The risk assessment question(s)
• The type of risk assessment to be conducted
• CFSAN's goal and how the assessment information and conclusions will be used by risk managers
• Background information on the hazard or human health impact relevant to assessment
• Expertise needed to conduct the assessment
• Resources allocated
• Timeline
• List of members of the risk assessment, management and communication teams with the leader (contact) individual clearly identified.

What are the Key Assumptions? Assumptions are needed when data are lacking. Key assumptions define the scope of the project and must be stated in the 'charge' to the risk assessment team. These assumptions must take into consideration the objectives and scope of the project. The risk assessment team is responsible for reviewing the key assumptions, explaining any limitations of them in the modeling effort and for developing the assumptions related to the model.

Key assumptions or the objectives that define the scope of the risk assessment include the following examples.

• Populations at-risk
• End point of concern (i.e., illness, death) or adverse effects
• Route(s) of exposure

Examples of assumptions related to modeling that risk assessors develop in consideration of the available data include the following. These are refined assumptions and should be arrived at following discussions between the risk management and risk assessment teams.

• Grouping of food into categories
• Surrogate exposure data
• Data limited or inclusive from national, international, or states
• Consumer practices or handling

Allocate Resources

CFSAN Personnel. Once personnel needs are identified, the risk assessment project manager submits the needs in the form of a request to the Program Office(s) with that expertise. The request must include the following:

• Type of expertise needed
• Estimated duration of the project
• Time per week that the individual will be needed
• Peak periods of activity

While the risk assessment project manager might "name request" a specific individual, it should be understood that this is only a suggestion. It is up to the Program Office to assign an appropriately qualified individual. This request to the Program Office should be formal, and involve sign off on the request. This action is to protect the assigned employee. Feedback should be provided to the assigned employee's supervisor regarding performance. Commitment of a person and the time they need to be available must be considered along with other duties, which may need to be modified appropriately. A determination must also be made of whether the Union needs to be involved.

External Expertise. Any required external expertise should be identified during the planning of the risk assessment. This might includes consultants, advisory committees, research personnel, the interagency Risk Assessment Consortium, and others. Additional experts may be identified during the conduct of the risk assessment as new needs arise.

Other Resources. Sufficient resources needed to conduct the risk assessment should be made available through either the requesting Program Office or the Office of Science. An administrative lead should be identified and work on the project to provide and administer resources needed by the risk assessment team. These resources might include file cabinets to store/sort references, work space (such as tables, desks in a dedicated room), computers, software, access to copy machines, and other needs as identified.

Research needed to complete the risk assessment should be identified and conducted, if feasible within the timeframe developed for the completion of the assessment.

Project Budget. A budget for the project must be provided. Specific items will depend on the scope of the project but might include the following:

• Travel for CFSAN staff to collect data/information
• Travel for CFSAN staff to attend meetings to discuss/present the assessment
• Travel to bring experts to CFSAN to discuss data, model, etc.
• Costs associated with holding public meetings
• Printing costs for the draft and final documents
• Costs to photocopy publications and purchase books or other reference materials needed by the team and also for display in the public dockets
• Specialized software such as project management, reference organization, or modeling tools
• Costs associated with peer review of the document
• Specialized training for risk assessment participants

Develop Timeline and Milestones

The identification of project timeline and milestones that are definable and clear must be established. Milestones will be used to judge whether the project is on track, delayed, or ahead of schedule. A sufficient number of milestones should be identified to ensure continual progress of the risk assessment. The risk analysis coordinator (with input from the risk assessment project manager) is responsible for keeping the risk management aware of any problems in meeting milestones. Whenever possible, these should be identified early enough so that corrective actions can occur to either realign the risk assessment and put it back on track or alter the timeframe, as appropriate. The risk management leader and the risk analysis coordinator should discuss the problems and propose solutions. The risk analysis coordinator discusses options with the risk management leader and the risk assessment project manager is responsible for implementing the recommended solutions.

The need for and timing of publications in the Federal Register and public meetings should also be discussed during the planning phase.

The timing of and amount of time needed to review draft and final models and documents should be discussed and agreed to so that the commitment and pathway is clearly understood. The risk assessment project manager in consultation with the risk analysis coordinator should discuss the type of review(s) needed, when the review is needed, and assist with making arrangements for acquisition of experts in a timely manner. During different stages of the risk assessment, different information needs to be reviewed by various parties. Examples of the types of information to be reviewed include:

• Scope of risk assessment
• Data
• Modeling assumptions
• Model
• Results (summary tables)
• Draft technical and summary documents

Determine Communication Needs

Interaction between the risk communication and risk assessment and risk management teams is critical throughout the risk assessment process. The risk communication team will need to identify risk communication needs for each unique audience. An analysis of the awareness and knowledge of the issues for each audience as well as the best method for reaching them is critical in preparing risk communication messages, materials, and determine the appropriate channels of communications. Once audiences have been characterized, the next step is to determine communication strategies for communicating that include both one-way communications (messages, materials) and two-way communication (listening to audience needs, open exchange of information). It is important that the broader risk communication messages meet the needs of the variety of audiences. See Appendix C for additional information about risk communication activities.

Step 2. Perform the Risk Assessment

Following the planning step, the risk assessment is commissioned. When the risk management team provides the 'charge' to the risk assessment team, the performance phase of the project can be initiated.

Goal: Answer the risk management question(s)

Product: Draft report of the results and conclusions of the risk assessment

Activities: Performing the Risk Assessment

• Review and refine (as needed) the scope of assessment
• Collect data and information
• Develop and validate model
• Review results
• Draft technical and interpretive summary documents

Risk assessment is a process used to evaluate the likelihood of adverse human health effects occurring after exposure to a pathogenic microorganism. Or more simply stated, it describes what we know and how certain we are of what we know. The risk assessment is also a product—a mathematical model, predications based on simulations, and a report that documents the process.

The generally accepted risk assessment paradigm for food contaminates includes separating the assessment activities into four components: hazard identification, exposure assessment, hazard characterization or dose-response, and risk characterization. This process allows organization, characterization, assessment of uncertainties, and identification of data gaps. See Appendix A for a detailed description of the components of risk assessment.

Figure III-2 provides the overall process of conducting a risk assessment and the various activities. The iterative nature of the risk assessment is illustrated by the feedback loops. The various activities needed to perform the risk assessment are described in greater detail below.

Figure III-2. Activities in Performing a Risk Assessment

Clarify scope of the risk assessment

During the planning step, approval for team members to participate on the risk assessment team would have been obtained from program offices. The conduct of the risk assessment begins when the risk management formally issues a charge (the questions for the assessment to answer and key assumptions to use).

The risk assessment project manager and risk analysis coordinator will hold a kick off meeting with the risk assessment team members to discuss administrative and technical issues concerning the project. At this meeting the risk assessment team members should discuss any concerns or questions about the scope of the project and conduct of the assessment. The risk assessment project manager and the risk analysis coordinator will discuss these concerns. Any potential roadblocks to completing the assessment will be forwarded by the risk analysis coordinator to the risk management team and any other appropriate parties such as the science advisor. Most importantly, each team member needs to understand his or her duties and responsibilities for the conduct of the assessment.

Example agenda items for the kick off meeting might include:

• Comments/ concerns about the scope of the project and key assumptions as presented in the "charge"
• Timeline/timeframe/milestones for the project
• Resources allocated and any additional needs not previously identified
• How and when to communicate with the risk management and communication teams
• Identify tasks and assign responsible parties
• Development of conceptual model, if not done during the data feasibility evaluation (see Part II of this document)
• Procedure for including documents in the repository for the project, including all references cited in the report and decisions made during the conduct of the assessment

Consultation with Risk Managers and Risk Communicators During the discovery and summarizing of data and assembling the model, it is likely that issues will arise that are appropriate for risk managers to decide. These issues need to be separated from technical and scientific issues that are appropriate for the risk assessment team to decide.

The risk management and risk communication teams will be briefed on the assessment at regular intervals during the conduct of the project. Risk management team approval/concurrence will be obtained on the assumptions used when data are lacking, types of data used in the model, additional research required, model structure and logic, and other milestones as agreed to during the planning step.

It is the risk assessor's responsibility to explain the quantitative results of the risk assessment in a way understandable to the risk management and risk communication team member; they should not be expected to translate the results. In addition, it is the risk assessors who need to describe their confidence in their conclusions, and the uncertainties inherent in the risk assessment, and the impact of assumptions made by the RM on the results. There should be an iterative dialog between risk assessors and risk communicators to ensure that the risk assessment reports are sufficiently clear and will answer questions and concerns raised by stakeholders.

The risk assessment team is responsible for:

• maintaining the scientific credibility of the assessment including transparency and quality, and consistency
• informing the risk management team of the impact of adding, removing, or altering data and assumptions
• information risk managers of any deficiencies in data needs or resources
• describing the uncertainty in the results
• refining the key assumptions provided by the risk management team for use in modeling
• assembling the data collected and incorporating these data into appropriate models chosen by the risk assessment team
• explaining the quantitative results of the risk assessment in a way understandable to the risk management team
• describing their confidence in their conclusions, taking into consideration the uncertainties and key assumptions incorporated in the model

Milestones. While risk managers, risk communicators, and risk assessors must confer through the process of conducting the risk assessment, the achievement of specified milestone events should trigger an interaction point. The risk management team should "check off" on what the risk assessors have completed for each milestone, thereby avoiding time-consuming revisions later in the process. These milestones will also be used to track progress of the risk assessment and determine whether the project is on schedule. Specific milestones should be selected during the planning process. Examples of milestone events include:

• Receiving 'charge' from risk managers
• Assembly of the risk assessment team
• Agreement of key objectives and assumptions
• Publishing the FR notice announcing the risk assessment
• Assembly of data and assumptions to be used in modeling
• Development of model
• Results from model simulations
• Results of sensitivity analysis
• Draft report
• Federal Register Notice of assessment availability
• Receipt and review of public comments

Documentation Procedures

The risk assessment project manager is responsible to setup and maintain (with assistance from a clerical assistant) a repository for all documents generated during the conduct of the risk assessment. Documents related to risk management and risk communication should be separately maintained by the respective team leads. The files from the risk management and risk communication teams may be transferred to the risk assessment project managers for archiving when the project is completed.

Examples of risk assessment files to be included in the archived records:

• Record of correspondence
• Record of decisions
• Federal Register notices
• All references cited in the risk assessment
• Contents of the public docket for the assessment (display items and submissions)

Collect/Assemble Data and Information

Under the leadership of the technical team leader and with assistance from the project manager, the team will collect data and information needed for the assessment. This will include the information and data obtained during the data feasibility evaluation conducted as part of the Identification and Selection process (see Part II of this document). A conceptual model may have been developed during the data feasibility evaluation. If available, the conceptual model will be used to guide the data collection. If not available, the risk assessment team should develop the conceptual model before extensive data collection efforts are initiated. Key assumptions provided by the risk management team will be refined for use in modeling, data will be assembled and the model constructed.

As data are collected and assembled for use in the assessment and as inputs for the model, the team must clearly identify the method used to collect data and the criteria used to determine the suitability of the data for use in the model.

Complete reference citations as well as a complete, legible copy of each reference must be given to the risk assessment project manager. These references will be filed in the risk assessment repository and included in the public dockets with the risk assessment.

Some of the information and data might be made available from research carried out or commissioned after the start of the assessment. The risk assessment team should decide what additional research they would like performed and make recommendations to the risk managers.

Develop and Validate Model

The process of developing the risk assessment model should be initiated in stages. Immediately after compilation and collation of all available data it will not be immediately evident what are the most relevant or key parameters of the process or how the relationships between parameters should be modeled. Consequently, the process of modeling is best viewed as an iterative process itself and the elements of the models as dynamic.

Initially, it can be of some advantage to develop a prototype of the risk assessment as quickly as possible in order to consider and evaluate as quickly as possible what types of modeling assumptions are likely to be the most influential. For example, in a process pathway model the distribution of a pathogen prior to the process may be very well characterized but the effect of the process on the pathogen levels may be relatively unknown. In such an event, concentration of effort on delineating plausible model structures for the effect of the process is likely to be more productive and facilitate interaction with RM in regard to plausibility of assumptions. Although uncertainty analysis of components of the model better characterized by available data should be considered, this may not be of paramount concern during the initial stages of model construction. The modeling process must be transparent in order to identify information and data gaps. The structure and flow of the risk assessment must be determined.

This includes specifying:

• the individual data
• how the separate data sets relate to each other (i.e., growth rates are multiplied by storage times to give the amount of growth)
• what variations and uncertainties are assigned to a data set
• what branching may exist in the data flow
• how much detail (number of components) should be in each section of the risk assessment

Decisions must also be made regarding which data to include and how to organize the data in a manner suitable to modeling.

Assumptions. Examples of modeling assumptions include decisions on the type of distribution to use for various aspects of the model. Other assumptions include use of surrogate data for data gaps. The risk management team should be briefed on the assumptions and their likely impact on the results.

Data and model verification/audit. Before running the model, the data inputs will be audited/verified and if possible, the model programming will be audited. The risk management team will have oversight of the assessment process and should review and evaluate the data and model at regular intervals during the conduct of the risk assessment.

CFSAN's Quality Assurance unit might be helpful in assisting in the verification/audit of the data used in the assessment and model. Experts in mathematics and modeling techniques should perform the audit/verification of the model. These experts needed to audit the model may be available from within CFSAN, within FDA, from other federal agencies, from academia, or through contracts with private consultants.

Data quality. The development of data quality criteria must be consistent with the Office of Management and Budget (OMB) guidelines for ensuring and maximizing the quality, including objectivity, utility, and integrity of information disseminated by Federal Agencies (see http://www.whitehouse.gov/omb/fedreg/reproducible.html).

The Joint Institute of Food Safety and Applied Nutrition (JIFSAN) has initiated a project to define data quality criteria for data used in risk assessments that are posted on the JIFSAN Food Risk Clearinghouse web site. The following are a few exerpts from a public meeting held in December 2000, on data quality that was co-sponsored by the interagency Risk Assessment Consortium and JIFSAN.

• Observations in a database should be 'model-free' to allow for data to be used in ways that the original investigator may not have intended.
• Data are rarely collected with risk analysis in mind, so many types of data must be considered for inclusion.
• The purpose and stage of the risk assessment process determines the data that will be used.
• "Good" data are complete, relevant and valid; complete data are objective, relevant data are case-specific, and validation is time-dependent.
  • Complete data includes such things as the source of the data and the related study information (sample size, species studied, seasonality, sensitivity, specificity and precision, and data collection method).
  • Relevant data include age of data, region or country of origin, purpose of study and species involved. Relevant data is difficult to judge and may be case-specific.
  • Valid data is that which agrees with others in terms of comparison of methods and development of tests.
• Data that can be eliminated from the risk assessment depends on the stage of the assessment and the purpose of the assessment. In the early stages of risk assessment, small data sets or those with qualitative values may be useful whereas the later stages of risk assessment it may be more reasonable to include only those data that have been determined to have high quality standards.
• Non-validated data can be included in certain circumstances, such as unique studies that are being reported for the first time.

Additional information concerning on data quality and other related issues can be obtained from the JIFSAN Clearinghouse web site (www.foodriskclearinghouse.umd.edu/Dec_5_2000dataquality.html).

Model Validation. Validation is a process by which a simulation model is evaluated for its accuracy in representing a system. All models are, by their nature, incomplete representations of the system they are intended to model, but, in spite of this limitation, some models are useful. Model validation is the process by which the model is evaluated on its usefulness. One of the more direct ways microbiological risk assessments are validated is to compare model predictions of illness with epidemiological data. Another way of validating is to compare model predictions with survey data (or other data independent of the data used in the model construction) at intermediate steps to the final prediction.

Model validation is not something that is done at the end of the risk assessment; it involves multiple steps:

• creation and validation of a conceptual model
• conversion of the conceptual model into a simulation program
• verification of proper translation of model into a computer program
• running of the simulation program
• validation of results of the simulation program
• communication of the results to the risk managers

Sensitivity/Uncertainty Analysis. If a risk assessment is complex, it may be difficult to ascertain which model components (and supporting data) are the most influential in terms of likely impact on the predictions made. A technique known as sensitivity analysis can be used to identify a part(s) of the model that is the most influential. In the broad sense, sensitivity analysis should encompass a systematic evaluation of the model, underlying and alternative model assumptions, and influence of input model parameters on the output distribution(s). Available Monte Carlo simulation programs, such as @Risk, commonly provide the option to perform sensitivity analysis by producing graphs or rank correlation statistics between input parameters and output parameter(s). These correlation statistics are useful with respect to evaluating the relative impact of each input distribution on the output distribution. However, these statistics are generated after the specification of a preferred model and therefore do not necessarily reflect the potential effect of alternative assumptions. Nevertheless, provided underlying model assumptions are considered reasonable, correlation statistics produced by a sensitivity analysis allows those input distributions where uncertainty has the greatest impact on the outcome to be identified. This, in turn will point to those parts of the risk assessment where additional research may be expected to have the most impact.

Where the distribution in the data may be assigned to variation and uncertainty, a two dimensional sensitivity analysis may provide additional insight. The parameters that are the most important for the central values may not be the most important for the tails of a population distribution. This is especially likely to be true if probability trees are used to represent model uncertainty in frequency distributions- all the models may give similar results for central values, but diverge widely at the tails.

Another approach to understanding the impact of assumptions and data on the results is to run "what if" scenarios. For very complex models, this is a simpler way of determining the uncertainty. For each scenario, specific assumptions or data are changed and the results are compared to the baseline to determine what if any impact the change had on the results.

Evaluating the Value of More Data. Research proposals are often justified on the basis of an expectation that they will reduce uncertainty, sample variability, or both. Until the research is complete, it is not known what results will be achieved, and although the expectation is for uncertainty to be reduced, this may be not realized. It is always possible that additional research will reduce or increase the uncertainty. Nonetheless, additional research may be useful, and we have identified three general purposes for additional research.

• Measurement of values used directly in risk assessment (e.g. in empirical distribution functions). For instance, if a risk assessment makes use of a population distribution for body weight, the risk assessment may be improved by measuring the body weights of more individuals, thus defining variability in a population.
• Collection of data that will allow more accurate estimates of model parameters. Parameters are not measured directly - they gain their meaning through their place in a model or theory. Therefore, the influence of additional data will depend on the way in which the data are used to judge the theories (e.g. the goodness-of-fit and parsimony criteria). If the models are only supported by data directly (there are no coherence arguments in support of the theory), then the problem is collecting more observations. If enough observations become available, then the theory may be discarded altogether in favor of an empirical distribution. However, if coherence arguments are used in support of the theory, then strengthening the theories that support the parameter estimates might be more important.
• Collection of data that will allow discrimination among models. The value of additional data in discriminating among models will also depend on the logic used to identify preferable models. The most useful data for this purpose may differ from that used for parameter estimation. Observations made under conditions where the candidate theories diverge in their predictions are likely to have the most impact. Of course, producing data that favors a particular model or hypothesis will only reduce the uncertainty of a particular risk assessment if model uncertainty is represented in that assessment.

Initiating research is in itself a decision. Whether or not a particular avenue is pursued will also depend on the results that might reasonably be expected and the time, effort, and expense required to obtain them.

Prepare the Draft Document(s)

Review Results. The results of the risk assessment modeling must be reviewed for:

• Reasonableness
• General trends
• Whether the risk assessment question(s) posed by the risk management team was answered
• Need for sensitivity/uncertainty analysis

Technical report vs. Interpretive Summary. Documents should be available in several mediums. Printed copies should be available though the CFSAN Outreach and Information Center, electronic copies should be available on the CFSAN web site, and/or CD-ROM.

The purpose of the technical document is to provide all information needed to reproduce the assessment. This document will be lengthy and technical but must be written in plain language. For the most part, the data used in the assessment should be placed in appendices.

The purpose of the interpretive summary is to explain how the assessment was conducted, the results and conclusions, and importance of the conclusions in a way that non-scientists can understand. The interpretive summary is much shorter in length than the technical document, only about 20 to 25 pages (maximum).

Transparency and Clarity. The risk assessment document(s) should be transparent and clear. Transparency includes describing the approach, assumptions and their impact on the conclusions, use of data/distributions/surrogate data, assumptions and impact on the results, data gaps and future research needs, the uncertainty in the results. Clarity can be achieved by avoiding jargon, using plain English, defining technical terms, and including well designed tables and graphs.

Style. While it is more important that the chosen style be consistently used within the document, some guidelines follow:

• Use Microsoft Word
• Use an appropriate and consistent font size such as Times New Roman, 12 point
• Use 1 or 1 1/2 lines spacing.
• Left margin justify, do not justify right margin.
• Use the Harvard system for references. For example cite last name and date of publication, for example Levitt et al., 2002. Do not cite references from a numbered list.
• As a general guideline, tables and figures should contain enough information to "stand alone"

Format (Technical report). The specific sections to be included in the technical risk assessment document will vary pending the specific needs of the document.

Below is a guideline to the overall organization of a technical document for a major risk assessment.

• Title page
• Preamble [for a draft document this might include a list of questions for which CFSAN is requesting input; for a final document this might include a summary of the public comments submitted]
• Contributors
• Acknowledgements
• Executive Summary
• Abbreviations/acronyms
• Glossary
• Introduction
• Hazard Identification
• Exposure Assessment
• Hazard Characterization [Dose Response Assessment]
• Risk Characterization
• Research Needs
• Interpretation and Conclusions
• References
• Appendices (include data, description of model, ancillary information)

Note: The overall structure of the interpretive summary document should generally follow that of the technical document.

Content (Technical Report). The technical document should include sufficient detail to allow the results of the assessment to be independently verified.

Examples of the types of information included:

• Purpose of the assessment including the risk management problem and risk assessment question(s) to be answered
• List of key assumptions and modeling assumptions and description of the impact of these assumptions on the results
• Tabular or graphic depiction of distributions
• Justification for selected distributions
• Identification of data source and strengths and weaknesses of the data used
• Description of the models
• Sensitivity analysis

Step 3. Reviewing the Risk Assessment Documents

A thorough review process is needed to prepare document(s) for release to the public. The review step includes peer review and agency approval or clearance. Review may occur at various points during the conduct of the assessment and the results of a review could necessitate that the model be re-run and document revised.

Goal: Critical review, approval, and agency clearance of the assessment

Product: Risk assessment document, ready for release to the public

Activities: Reviewing the Risk Assessment Documents

• Risk analysis teams review
• Peer review
• Stakeholder input
• Clearance/approval

Risk analysis team review. The risk assessment team will actively seek comments and consult with the risk management and risk communication teams throughout the conduct of the risk assessment. Specific milestones that identify critical points when the risk assessment team should brief the risk management team will be selected during the planning step. By the time the risk management and risk communication teams review the draft document, it is expected that they are thoroughly knowledgeable about the methodology used and general conclusions.

Peer Review. Currently, CFSAN does not have a formal peer review process policy. Such a policy should be established. For example, the International Life Sciences Institute (ILSI) had developed an independent Model Peer Review Center of Excellence to evaluate toxicity assessments and associated proposed toxicity values for chemicals identified as contaminants of concern at Superfund hazardous waste sites. This process could be adapted to meet CFSAN needs.

In order to evaluate data quality, the criteria used for data selection must be specified in the risk assessment document. Additionally, the risk assessment teams should solicit critical reviews of the methods and models from subject matter experts within and outside of the Center. In order to ensure that we have used the best information available, CFSAN should also make a draft of the risk assessment available to the public for review.

Government and Non-government Experts. Whenever possible, subject matter experts in the government and special government employees (SGE) should be involved in the peer review. SGEs are held to confidentiality agreements that allow draft documents to be reviewed before they are released to the public. This is important because the iterative nature of risk assessments often means that subsequently requested changes to data or model might change the results and conclusions of the document. Circulating preliminary documents to the public before they have undergone extensive Center review and approval causes confusion and should be avoided.

RAC. The interagency Risk Assessment Consortium [1] (RAC) is an important resource for technical advice on the use of data and modeling techniques. As appropriate, the RAC members should be involved in reviewing and providing critical feedback on the draft risk assessments.

Advisory Committee. The advice and opinions of advisory groups such as the National Advisory Committee on Microbiological Criteria for Foods [2] (NACMCF) should be solicited. An initial meeting should occur early in the conduct of the risk assessment. Additional meetings to discuss scientific issues such as the approach, assumptions, and model structure should also be held, as needed. An additional resource is the National Academy of Sciences.

Stakeholder input. Stakeholder input is a critical component of the process and necessary to meet CFSAN's requirement for a transparent process. Workshops and/or public meetings may be held to provide clarity and obtain input from stakeholders regarding both the scope of the assessment, assumptions made, and results. Obtaining stakeholder input is essential to the risk communication process. CFSAN will also provide feedback both formally and informally as to how stakeholder concerns and comments have been addressed. Stakeholders should be encouraged to submit comments to the public docket established for each risk assessment project. Informal meetings and technical presentations by risk assessors may also be made at the request of specific groups, industries, or trade associations. CFSAN should attempt to honor such requests whenever possible. At times it may be more efficient to have an informal meeting at CFSAN for specific identified stakeholders.

Means of informing and listening to stakeholders include:

• Workshops
• Public meetings
• Comments submitted to the public docket
• Informal meetings
• Technical presentations, as requested
• Federal register notices
• Documents on the CFSAN web site

Clearance/Approval

CFSAN's process for clearing and approving of documents must be followed. This clearance/approval process ensures that materials released by CFSAN are scientifically and technically accurate. Form FDA 3451 (1/00) should be used for the purpose of tracking and documenting actions and clearances of responsible offices. A determination must also be made on whether the risk assessment documents require departmental review and FDA's commissioner and/or the DHHS secretary's signature. Collaborative projects with other government agencies will also involve consideration of the clearance process of those agencies.

The risk analysis coordinator should consult with the Regulations Coordination Staff to ensure that the Federal Register Notices are developed and reviewed using appropriate, standard operating procedures.

Step 4. Issuing the Risk Assessment Documents

In this step of the risk analysis framework, the risk assessment document(s) are issued. Following peer review and agency clearance, CFSAN will seek public opinion of the documents. A final (or revised) assessment will include a discussion of the public comments received and how they were addressed.

Goal: Develop a roll-out plan for issuing document(s)

Product: Publish the risk assessment technical and interpretive summary document(s)

Note: Some of the activities implemented during this step may have been initiated and discussed during the planning step; however, the focus at this stage should be implementing specific activities related to releasing the assessment to the public.

Activities: Issuing the Risk Assessment Documents

• Develop and implement roll-out plan
• Address public comments submitted to docket
• Finalize and issue Report
• Follow up and debriefing

Develop and Implement Roll-out Plan

In developing a roll-out plan, a number of issues need to be discussed and decided to determine how to best implement a plan. A roll-out plan cannot be prescriptive, it must be tailored to the specifics of the individual project and as such will be determined on a case-by-case basis. It is recommended that a one to two page analysis of the issues be developed to assist in implementing the roll-out plan. The risk communication team is essential to developing and implementing an efficient roll-out plan.

Some of the issues to be discussed include:

• Audience. Who will be impacted by the assessment? Who needs to be informed - within and outside of CFSAN? Who is the target audience for each document? What tone should be conveyed by the messages? What groups will be interested (press/consumers)?
• Means. How should the audience be informed so there is no misinterpretation of the results? Who needs to be involved within and outside of CFSAN? What vehicles should be used?
• When/where. Timing of issuing documents and holding meetings? What immediate actions (if any) are needed? By whom?

Whenever possible, the plan should contain elements of proactive communication and outreach instead of being reactive. However, we recognize that this may not be possible in all situations.

Who Develops and Implements the Roll-out Plan? The risk communication team will be a major contributor to the development and implementation of key elements of the roll-out plan. The risk assessment project manager and coordinator should assist as needed. However, within CFSAN there are many resources and staff experienced in both communication and outreach activities that should be included. For example Executive Operations Staff, FDA's Office of Public Affairs, Regulations and Policy Staff, Food Safety Staff, Management Systems, and also staff located in individual program offices.

Example elements of the Roll-Out Plan

• A press contact list
• Talk points/ press announcements/briefing materials/Q and A's
• Schedule and hold public meeting(s)
• Plan and hold stakeholder briefing(s)
• Plan and hold other briefings or presentations, as needed
• Develop and publish the Federal Register Notice; Availability of document(s)
• Make and distribute printed copies of document(s) to stakeholders and public
• Prepare and post risk assessment document(s) on CFSAN's web page

[Materials developed and distributed by the risk communication team are described in detail in Appendix C.]

Public Dockets

Stakeholders must be permitted time to submit comments on the draft document. FDA's documents provided in the Federal Register and comments submitted are on display and available for public review at FDA's public dockets (FDA Dockets Management Branch, 5630 Fishers Lane, Room 1060, Rockville, MD 20852). A discussion of the comments submitted and how CFSAN addressed those comments in the risk assessment are presented in the final or next version of the document.

Issue Final Report

Following the public comment period, the draft risk assessment is revised and issued as a final (or next version) document. A notice of availability of the risk assessment must be published in the Federal Register.

Where Should the Risk Assessment and related documents be made available?

• CFSAN's Outreach and Education Center (provides printed copies)
• CFSAN's web site (electronic format)
• JIFSAN Food Safety Risk Analysis Clearinghouse (link to CFSAN web site)
• Public meetings (presentation)
• Constitute Updates (brief description, faxed to stakeholders)
• Other appropriate means identified in the roll-out strategy

Follow up or Debriefing

Following the completion of the risk assessment, the risk analysis teams should meet to evaluate the overall risk assessment process. Such an evaluation should focus on determining whether or not the procedures set forth in the this document were followed and, whether following the procedures set forth in this document facilitated and improved the overall process and resulting risk assessment document. To the extent that the procedures were not followed, or modifications made, the group should try and identify the reasons and determine whether modifications to these procedures should be made. The success of risk communication throughout and at the end of the process should also be evaluated (as described in Appendix C).

In particular, the group should look at the following:

• Was the risk assessment 'charge' from the risk managers/program office sufficiently clear? Did the team take appropriate steps to resolve any concerns over the 'charge' in a timely manner such that the assessment could be modified or a decision to proceed/not proceed be made?
• Were sufficient resources identified and provided? Were there any gaps identified that suggest the need for further capacity building?
• Were key assumptions from the risk managers clearly provided? Did risk assessors/managers and communicators sufficiently discuss and understand the significance of such assumptions in a timely manner?
• Were data gaps or other risk assessment assumptions identified by risk assessors clearly communicated to the other team members?
• Were uncertainties clearly identified along with their resultant impact on the results? Was a sensitivity analysis performed?
• Do risk assessors believe the results are scientifically sound and that the assumptions used followed best risk assessment practice? Were any conflicts between risk assessors and managers identified and if so, how were they resolved?
• Were sufficient resources devoted to the conduct of the assessment including the assignment of appropriate staff that could adhere to schedule requirements?
• Do risk managers feel they got an assessment that answered their question(s)? If not, why not and what could have been done differently?
• At what point was the risk communication team brought into the process? Were stakeholder concerns identified early on in the process? Did these concerns result in any changes to the charge of the assessment/assumptions made, etc? Did the team feel it had sufficient information to facilitate the communication of results?
• Were appropriate and sufficient project milestones identified? Did they facilitate the identification of problems and corrective actions?
• Were any problems identified in a timely manner? Was there a clear process for resolving problems? What other steps (if necessary) would be needed to better resolve issues raised or problems identified?
• Did sufficient communication between all team members occur? Was communication ongoing? Did team leaders sufficiently communicate? Did the appropriate level of information filter down to all team members?
• Did people respond to the identified deadlines for providing comments? Were comments and changes made in the latter stage that had been identified earlier but not addressed? Did the review and release process significantly alter the assessment or result in the need for further work? Why?

In addition, after the risk assessment is completed, risk managers may ask for several types of additional research. For example, they may want epidemiological and statistical research to see how well the predictions of the risk assessment are corroborated by experience. As new information or data become available, CFSAN must determine whether the risk assessment should be revised. As new science and technology related to the hazard come into use, risk managers may want to see how those developments affect the conclusions of the risk assessment. If the risk assessment leads to a proposed regulation, risk managers will want research on the cost and effectiveness of potential risk mitigations.

back to table of contents

Part IV: Conclusions, Recommendations, and Next Steps

Conclusions

The CFSAN Risk Analysis Working Group concluded that risk analysis is a powerful tool that should be used to enhance the scientific basis of regulatory decisions. It should be conducted within CFSAN through the efforts of risk assessment, risk management, and risk communication teams. Risk assessment should be conducted in an iterative manner that allows a refinement of the risk assessment question(s), key assumptions, and data used in the model. The exchange of information (communication) within and between the risk analysis teams, with other agencies, and stakeholders (including industry, consumer groups, and other interested parties) should be encouraged by active participation in the process and collaborations, when appropriate.

The working group considered processes and procedures to address the following three tasks:

• Identify boundaries and responsibilities of key participants in the risk analysis process.
• Develop a process for identifying and selecting risk assessments conducted within the Center.
• Establish procedures for the conduct of risk assessments within a risk analysis framework.

This document provides a proposed process for initiating and conducting 'major' risk assessments within a risk analysis framework that is both decision-based and systematic. An overview of risk analysis concepts and identification of boundaries and responsibilities of key CFSAN participants was provided in Part I of this document.

Before a risk assessment can be initiated, an idea for a risk assessment must be identified and then selected based on Center needs, resource availability, and a data feasibility evaluation. The selection process is divided into four phases—concept generation, problem identification, data feasibility determination, and disposition. The purpose of this decision-based approach is to ensure that the candidate risk assessments are systematically evaluated based on the Center's regulatory needs and feasibility (resources and data availability. Details of the identification and selection process were provided in Part II of this document. The conduct of the risk assessment includes four steps—planning, performing, reviewing, and issuing. The activities associated with these steps were described in Part III of this document. A summary of the overall process for initiating and conducting these risk assessments is provided in Figure IV-1.

However, having proposed a formalized procedure for the conduct of risk assessments is only the beginning. Once a process/procedure is agreed to, it must be clearly communicated to Center staff and other stakeholders. Next, the process must be followed. To the extent that it is followed we can ask ourselves the question-"did it work?"

More specifically, questions to be addressed include:

• Did the process address the issues and concerns identified by the CFSAN Risk Analysis Working Group?
• Were there places where the proposed process did not work or where modifications were made?
• How could the process be modified or improved to facilitate and improve future risk assessments?

Answering these questions may or may not require the services of a neutral facilitator. But, at a minimum, these questions should be systematically explored.

Figure IV-1. Summary: Process for initiating and conducting risk assessments

Recommendations and Next Steps

The working group recommended the following actions to support and promote the use of a risk analysis framework for 'major' risk assessments. Next steps for the first two of the six recommendations are described below.

Recommendation 1. Adopt a decision-based approach to identify and select 'major' risk assessments conducted by CFSAN.

The working group recommended an identification and selection process that is conducted in four phases: concept generation, problem identification, data feasibility determination, and disposition. This approach was described in detail in Part II of this document. To test and implement this proposed approach, the working group suggests the following next steps.

• A mechanism should be developed to request and receive ideas for risk assessments from the general public, industry, consumer groups, other Agencies, and other interested parties.
• The proposed identification and selection process should first be tested with a microbial risk assessment and later implemented for all 'major' risk assessments including chemical.
• After testing this process with a microbial risk assessment, a debriefing meeting should be held. The purpose of the debriefing is to determine whether the proposed identification and selection process was in fact followed and, if so, whether it facilitated and improved the overall decision-making process. The CFSAN Risk Analysis Working Group should conduct the debriefing with the coordinating staff and CFSAN's leadership team. The leadership team should decide whether a neutral facilitator should lead the debriefing.
• Following the debriefing, a final procedure should be described as a standard operating procedure (SOP) and used by the Center for identification and selection of all 'major' risk assessments.
• After the SOP is final, it must be clearly communicated to Center staff, other agencies, and stakeholders.

Recommendation 2. Establish a procedure for the conduct of risk assessment within a risk analysis framework.

The working group recommended that CFSAN should conduct risk assessment within a risk analysis framework. Specific actions were grouped as four steps--planning, performing, reviewing and issuing. The activities associated with each of these steps are described in detail in Part III of this document. To test and implement this proposed process, the working group suggests the following next steps.

• After testing this process with a 'major' risk assessment, a debriefing meeting should be held. The purpose of the debriefing is to determine whether the proposed four step process for conducting risk assessments was in fact followed and, if so, whether it facilitated and improved the overall decision-making process. The CFSAN Risk Analysis Working Group should conduct the debriefing with the leadership team. The leadership team should decide whether a neutral facilitator should lead the debriefing.
• Following the debriefing, a final procedure should be described as a standard operating procedure (SOP) and used by the Center for conducting all 'major' risk assessments. The SOP might include checklists or forms, if appropriate.
• After the SOP is final, it must be clearly communicated to Center staff, other agencies, and stakeholders.

The working group also made the following recommendations to enhance CFSAN's use of a risk analysis framework for conducting risk assessment. Specific procedures for implementing these recommendations should be developed but were considered outside the scope of this document. Senior management should identify and charge appropriate CFSAN staff to develop proposed approaches or processes for the following recommended actions:

Recommendation 3. Develop criteria to evaluate the quality of data used for risk assessments and specify what information is needed to scientifically evaluate the usefulness of a study or data set used for risk assessment.

Recommendation 4. Develop guidelines to evaluate risk assessments and supporting data developed by stakeholders and submitted to the Center.

Recommendation 5. Formalize a peer review process that will encourage critical review and evaluation of CFSAN's risk assessments by government and non-government experts in a manner that improves the science and acceptance of complex risk assessments.

Recommendation 6. Build capacity to conduct complex risk assessments by providing training opportunities for current staff, hiring new staff or using contractors (as needed), and acquiring additional resources such as computers, software, and dedicated workspace.

The recommendations in this document are based on our experience in conducting risk assessments within a risk analysis framework. We believe that this document represents a huge leap forward for CFSAN and its efforts to formalize the use of risk analysis in a regulatory environment. As additional experience and knowledge are gained, the processes described here should be amended, as needed.

back to table of contents

Appendix A: Components of a Risk Assessment

The generally accepted paradigm for microbial, as well as chemical food contaminates, includes separating the assessment into four components: hazard identification, exposure assessment, dose-response assessment, and risk characterization. This process allows organization, characterization, definition of uncertainties, and identification of data gaps.

• Hazard identification. Gathering information about the pathogen, its presence in foods, and the adverse outcome (illness or death) associated with consumption of contaminated foods.
• Exposure assessment. Estimates the levels of the pathogen consumed. This includes the probability that the pathogen will be present in the commodity, the frequency of various levels of the pathogen in the food consumed, and the impact of food handling, processing, and storage conditions on the overall potential exposure.
• Dose-response assessment. Estimates the relationship between the exposure level (dose) and frequency of illness or other adverse effect (response). The severity of the health effect must also be considered. This often includes the challenge of attempting to extrapolate data acquired in animal models to humans.
• Risk characterization. Estimates the likelihood of the adverse outcome from exposure to the pathogen. The exposure and dose-response assessments are integrated to mathematically express the probability of the effect on public health as well as provide the uncertainty associated with this estimate. An important part of this step is determining the degree of uncertainty in relation to the results and distinguishing this from the variation that is inherent in any biological system.

back to table of contents

Appendix B: Example Questions: Communication Between the Risk Managers and Risk Assessors

The questions below are not a prescriptive list but rather a list of possible questions that risk managers and risk assessors might discuss to clarify the scope of the risk assessment and to promote understanding of the results. This list of questions is intended to assist the risk manager and assessor in their deliberations and therefore no implications should be drawn regarding the rank order or the chronology in which the questions are presented. Finally, many of the questions might be posed by the risk assessment team leader to the risk assessment (RA) team.

Before Commissioning the Risk Assessment

Once the risk management (RM) team has provided the RA team with background information, and discussed the problem at hand, the questions to ask the risk assessment team prior to beginning the risk assessment include the following.

1. 1a. Do you have any questions about the risk management problem or the subsequent risk assessment question posed by the risk management team?
2. 1b. What type of risk assessment is needed? Would a threshold or safety assessment answer the risk assessment question posed by the risk managers?
3. 1c. In consideration of the risk management problem, are there other questions that are important for the risk assessors to answer?
4. 1d. Are there substitute risks that need to be evaluated because they will be impacted by any potential management decisions?
5. 1e. Are any additional resources needed to adequately to address the problem?
6. 1f. How should the risk management team communicate with the risk assessment during the project?
7. 1g. How will the CFSAN risk assessment team work with our risk assessment partners (federal, academic, and industry), if appropriate?

During the Risk Assessment

General Data Questions

1. 2a. Were studies that showed negative results or a lack of effect for the compound or food used in the assessment? If not, explain the reason for excluding those data.
2. 2b. Were any data considered as potentially relevant to the assessment but not integrated into the calculations and conclusions? If so, why were those data excluded?
3. 2c. Were qualitative data used? If so, how was it used it in the analysis?
4. 2d. Were data collected outside of the United States used in the model? If so, does the United States import the relevant commodity or is there some other reason those data are relevant to the United States?
5. 2e. Are the data representative of the subpopulation of interest? If not, how were the data weighted? Was the organization that collected the data consulted on proper weighting procedures?
6. 2f. Were Bayesian approaches used in any aspects of the data computations? If so, how were the prior distributions obtained and weighted?
7. 2g. What methods and assumptions were used to estimate the shape, spread, and location (e.g., mean or median) of the exposure or risk distributions? What are the most sensitive assumptions or data elements for the percentiles relevant to the risk management questions?
8. 2h. How is the risk assessment team ensuring that all confidences (legal, proprietary) are being honored?
9. 2i. Have any specific risk management options been eliminated because of the types of use of data or assumptions in the assessment?

Models

1. 3a. What criteria were used to select the statistical model(s)? What models or types of distributions did you not use and why?
2. 3b. Is there anything that can happen in the model that cannot happen in reality or the converse? (For example, unaccounted correlation between body weight and consumption rates or age that cause increased incidence at the tails.)
3. 3c. Is the model flexible enough to be modified if new data become available?

Assumptions

1. 4a. Which assumptions are the most influential on the results (that affect decisions)?
2. 4b. Is there any specific modeling assumption(s) that the RM team should review?
3. 4c. Which assumptions are science-based as opposed to policy-based?

Hazard

1. 5a. Was the severity of the adverse effect and size of the exposed population adequately and properly described?
2. 5b. Is the adverse effect reversible or irreversible?
3. 5c. Is the hazard an acute one or chronic in nature?
4. 5d. Does the adverse effect result from prenatal or early postnatal exposure of is it an effect seen latter in life?
5. 5e. Is the primary evidence for the adverse effect based soley on laboratory animal experiments? Is there some correspondence for such an effect in humans?
6. 5f. Are the adverse effects noted with only high levels of exposure or do they occur at levels more consistent with normal human exposure?
7. 5g. Is there a mechanistic explanation for the adverse effects?
8. 5h. Are there toxicokinetic/dynamic similarities/dissimilarities between laboratory animal models and humans?
9. 5i. Does the adverse effect occur normally in the at risk population and is it a frequent or infrequent effect?
10. What other risk factors are normally associated with the adverse effect?
11. 5j. If human studies are being used is the effect one considered to be a frank clinical one or one that is more subtle and subclinical in nature?
12. 5k. Is there a spectrum or continuum of adverse effects?

Exposure

1. 6a. If relevant, how were food groupings chosen?
2. 6b. Are the contamination data from studies that use random sampling? If not, what adjustments were used to avoid bias?
3. 6c. Were the food consumption data adjusted for geographical, socioeconomic and other factors?
4. 6d. Does the risk assessment include food consumption data from more than one source? If so, what is the rationale for using data from more than one source or combining data from different data sets?
5. 6e. Are there other, important sources of exposure to this pathogen that should be considered?
6. 6f. Were outbreak data and/or surveillance used in the assessment?
7. 6g. Is the exposure assessment derived based on food consumption and residue surveys or are they based on studies of biomarkers of exposure in human studies?
8. 6h. Will the estimates of exposure be done in deterministically or probabilistically?
9. 6i. Are the exposure data adequate to describe a distribution or will they have to be "bootstrapped" into a simulation?

Dose-response

1. 7a. Was lab animal data used as the basis for the dose-response presentation? If so, how was the animal-to-human extrapolation addressed?
2. 7b. Was human data used for the dose-response assessment? How many studies were used? If more than one was a 'meta' analysis performed or was the data pooled?
3. 7c. Were the human studies clinical studies or environmental epidemiology studies? What type of study was used - case-control, retrospective, prospective?
4. 7d. Is the dose route of administration used in the dose-response assessment similar to the normal route of human exposure?
5. 7e. Is the dose-response assessment within the observable range or does it involve low dose extrapolation?
6. 7f. Is the dose range narrow or broad in scope?

Completion of Draft Risk Assessment

1. 8a. What is (are) the answer(s) to the question(s) the risk management team asked?
2. 8b. Was a sensitivity test conducted? Was it broad enough so that all major assumptions and data uncertainties were tested to evaluate how alternate assumptions or data would affect the results?
3. 8c. Are sufficient data and information provided in the report so that another risk assessor could replicate the results?
4. 8d. Which parts of the risk assessment will be most controversial?
5. 8e. Have the major sources of uncertainty been identified? Is it possible to reduce the uncertainty?
6. 8f. Explain which subpopulations are at the highest risk. How many are in each category and why they are in those categories (i.e., high levels of exposure or high sensitivity)?
7. 8g. Does the risk assessment report include an explanation of the difference between uncertainty and variability?
8. 8h. What is the impact of this risk (or risks) relative to other similar risks?
9. 8i. Explain why the risk(s) were characterized in a particular way, e.g., risk per year, per lifetime, per eating occasion.
10. 8j. What is the relationship in this risk assessment of morbidity to mortality? How many in each category and how severe is the morbidity?

back to table of contents

Appendix C: Risk Communication Activities

Below is a summary of specific risk communication activities conducted within a risk analysis framework. The need for these activities was mentioned in other sections of this document and the focus of the discussion below is on specific risk communication activities.

Risk Communication Activities

• Assemble risk communication team
• Identify audiences and risk communications needs
• Determine risk communications strategies
• Distribute risk communication information and education materials
• Evaluate success of risk communications throughout and at the end of the risk analysis process

Assemble Risk Communication Team

During the planning step for the risk analysis project, the risk communication team will be formed by the appointed risk communication team leader(s) in consultation with the Risk Analysis Coordinator and Program Office/Staff directors.

Identify Audiences and Risk Communications Needs

Activity. Identify audiences for risk communication, such as the general public, scientists, the media, consumer and industry representatives, public health professionals, and regulators. Audiences may also include consumers, especially those consumers at-risk for foodborne illness, such as the elderly, pregnant women, young children, and people with weakened immune systems.

The audiences' needs for risk communication are as varied as they are. Careful analysis of the awareness and knowledge of the issues for each audience as well as the best method for reaching them is critical in preparing risk communications messages, materials, and determining the appropriate channels of communications.

Responsibility. The Risk Communications Team and the EOS Advisory Committee representative in consultation with the CFSAN Director and Deputy Director, the Risk Analysis Coordinator, and the Risk Assessment, Risk Management, and Risk Communication Team Leaders. Informal meetings with stakeholders, including meetings with professional colleagues and/or "sister" food safety agency representatives, should be held as well at the discretion of those mentioned above.

Determine Risk Communications Strategies

Activity. Once audiences have been characterized, the next step is to determine the most appropriate communication strategies (messages, materials and channels) for communicating that include both two-way and one-way communication.

The development of risk communication messages for information and education materials should be based on the results of the risk assessment and formal and informal information exchanges with various audiences. Common "talking points" or "themes" should be developed about the risk analysis process. An open exchange of information should insure that the broader risk communication messages are meeting the communication needs of the variety of audiences.

For example, messages to industry about the nature of the risks identified by a risk assessment and the methods of managing those risks may have a different emphasis than the message to consumers about safe handling practices necessary to avoid a specific risk.

Channels for communication may include:

• Public meetings
• Formal and informal working meetings
• Web pages
• Press releases
• Other CFSAN communication channels such as the Constituent Update

Responsibility. The Risk Communications Team in consultation with the Risk Assessment and Risk Management Teams, the EOS Advisory Committee, and stakeholders. Other activities, in addition to the identification of talking points, and responsible parties are noted in Table C-1.

Distribute Risk Communications Materials

The different types of information and educational materials and responsible parties are provided in Table C-2.

Evaluate Success of Risk Communications Throughout and at the End of the Risk Analysis Process

Activity. Determine formal and informal means of evaluating effectiveness of risk communications with the various audiences during and at the end of the risk analysis process. The evaluations should consider whether all appropriate target audiences were identified and the extent of the target audiences' awareness, knowledge, and if appropriate, behavior change as a result of risk communication.

Responsibility. The Consumer Studies Branch, Division of Market Studies, OSAS should undertake consumer evaluation, in consultation with the FSS education staff through questions on the national consumer food safety survey.

back to table of contents

Appendix D: Criteria for Identifying Candidate Risk Assessments

The following are example questions to consider in compiling a list of candidate microbial risk assessments. The questions address the nature of the risk in the context of identifying the risk (or a particular hazard) of concern to a defined population group from consumption of defined product(s).

1. Is this a significant public health problem in terms of incidence?
2. Is this a significant public health problem in terms of severity of illness, morbidity, and mortality?
3. How geographically widespread is the hazard?
4. Is this hazard well recognized or a new or latent problem? Is the situation urgent or not?
5. Is the risk--worsening, stable, improving, or unknown?
6. What is the relative importance of this hazard in terms of the intrinsic properties (i.e., virulence, pathogenicity, host specificity)?
7. Is there a significant potential for human exposure to the hazard due to either high consumption or high contamination levels of specific commodities?
8. Are some people more susceptible to infection, such as the elderly, infants, and immunocompromised persons?
9. Does the public perceive this as a health risk?
10. Are there regulations, programs, or control measures that may be used to control the hazard?
11. Is there a need for a risk assessment because of a problem in trade, or economic, regulatory, or social concern?

back to table of contents

Appendix E: Criteria for Evaluating Technical Feasibility

The evaluation of the technical feasibility of conducting a risk assessment must consider the availability of data, the uncertainty created by lack of data, or the impact of the lack of specific data on the risk estimates and on the feasibility for conducting the assessment. Since the data are never perfect and rarely collected specifically for risk assessment, many types of data must be considered for inclusion into risk assessment. The criteria may depend on the risk management and/or risk assessment questions that are being addressed. The following are examples of criteria to include in an evaluation of data for the purpose of determining the feasibility of conducting a risk assessment.

1. Data availability. Are the data available and relevant to conduct a risk assessment? Specific information to obtain include:
   • Source and purpose of the data or study
   • Data collection (date, country/region of origin, time frame or seasonality, method of data collection, sample size, sampling scheme, etc)
2. Data quality. What is the quality of the available data? Specific information to obtain include:
   • Methodology (testing methods, sensitivity, specificity of test(s), precision, species of animal used, etc)
   • Dimension of uncertainties (sampling errors, measurement errors, etc.)
3. Confidence. Do experts generally agree with the methodology and/or results? Specific information to obtain includes:
   • Validity with regard to findings of other researchers
   • Publications using these data
   • Peer review
   • Evaluation of data
4. Other.
   • Data on changes in production practices (including process innovations)
   • Data on lack of compliance with standards

back to table of contents