The New Digital Frontier: Navigating the FDA's Credibility Modeling Guidance for Medical Devices

The landscape of medical device development is shifting from the traditional laboratory bench to the high-performance computing cluster. Computational Modeling and Simulation (CM&S)—often referred to as in silico methods—is no longer a futuristic concept but a well-established and increasing presence in regulatory submissions. As these digital twins become more central to demonstrating device safety and effectiveness, the fundamental question for regulators and manufacturers alike is: How do we trust the prediction?

On November 17, 2023, the FDA released a landmark guidance document: Assessing the Credibility of Computational Modeling and Simulation in Medical Device Submissions. This guidance provides a risk-informed framework to build that trust.

What is Model "Credibility"?

In the regulatory context, credibility is defined as the trust, established through the collection of evidence, in the predictive capability of a computational model for a specific context of use (COU).

It is important to note the scope of this guidance. It applies specifically to first principles-based models (physics-based or mechanistic), such as those involving fluid dynamics, solid mechanics, or electromagnetics. It does not apply to statistical or data-driven models, such as those rooted in machine learning or artificial intelligence, though hybrid models may fall partially under its umbrella.

Why Move In Silico?

The push for modeling isn't just about speed; it's about depth. CM&S can reveal information that traditional in vivo (animal/human) or in vitro (bench) assessments simply cannot see, such as unexpected adverse events that occur frequently enough in a virtual population to be concerning, but are invisible in a small study sample. Furthermore, the FDA strongly supports the "3Rs" principles—replacing, reducing, and refining animal use in testing—and modeling is a primary vehicle for this ethical shift.

--------------------------------------------------------------------------------

The 9-Step Generalized Framework

The FDA recommends a systematic process for developing and assessing model credibility. This framework is designed to be consistent with the ASME V&V 40 standard but offers a more generalized approach for various types of evidence.

1. The Question of Interest

The process begins not with the model, but with the clinical or engineering decision that needs to be made. You must describe the specific question, decision, or concern being addressed (e.g., "Is the device resistant to fatigue fracture under radial loading?").

2. Context of Use (COU)

The COU is a statement defining the model’s specific role and scope. It answers: What will be modeled, and how will the outputs be used to answer the Question of Interest?. Crucially, the COU should state whether the model is the sole source of evidence or if it will be used alongside bench or animal data.

3. Model Risk Assessment

This is the heart of the framework. Model risk is the possibility that the simulation results could lead to an incorrect decision that results in an adverse outcome. The level of credibility evidence required is commensurate with this risk. Risk is determined by two factors:

• Decision Consequence: The significance of an adverse outcome if the decision is wrong.

• Model Influence: The contribution of the model relative to other evidence in making that decision.

  
  

4. Identify Credibility Evidence

Once risk is understood, you must identify what evidence you will collect. The FDA breaks this down into eight distinct categories, ranging from code verification to emergent behavior.

5. Credibility Factors and Goals

You must define credibility factors—the elements of the process used to establish trust. For each factor, you establish a gradation (a ladder of increasing rigor) and select a credibility goal that matches your assessed model risk.

6. Prospective Adequacy Assessment

Before running simulations, you ask: "If I meet these goals, will the evidence be sufficient for the COU?". This is a prime opportunity to use the Q-submission process to get FDA feedback on your plan before investing heavily in execution.

7. Generate Evidence

This is the execution phase—running the studies, analyzing data, and quantifying uncertainties.

8. Post-Study Adequacy Assessment

After the data is in, you must determine if the goals were actually met and if the totality of evidence supports using the model for the COU. Unlike accuracy, which is quantifiable, adequacy is a qualitative judgment using engineering and clinical expertise.

9. Reporting

The final step is the creation of a Credibility Assessment Report to be included in the regulatory submission.

--------------------------------------------------------------------------------

Defining Model Risk: The 3x3 Matrix

The guidance emphasizes that model risk is not intrinsic to the model itself, but rather to how the model is used.

Decision Consequence should be assessed using standard risk management procedures like ISO 14971. It considers potential patient harm, accounting for both severity and probability.

Model Influence depends on the "weight" given to the model. If a device takes action based solely on a simulation result, influence is at the highest level. If simulation results are just one factor provided to a clinician to inform a choice, influence may be lower—though manufacturers must still account for "reasonably foreseeable misuse," such as a clinician over-relying on the model.

A scheme (like a 3x3 or 5x5 grid) is used to combine these factors into an overall risk level. Higher risk demands more rigorous validation.

--------------------------------------------------------------------------------

The Toolbox: 8 Categories of Credibility Evidence

The guidance recognizes that not all evidence comes from traditional bench testing. It outlines eight categories to characterize your evidence:

1. Code Verification: Confirming that the numerical algorithms are correctly implemented and free of bugs.

2. Model Calibration: Tuning parameters so the model matches a specific dataset. Note: The FDA considers this "weak" evidence because it is not an independent test of the model.

3. Bench Test Validation: Comparing model results with well-controlled laboratory/in vitro data.

4. In Vivo Validation: Comparing results with clinical or animal data on a subject-level basis.

5. Population-Based Validation: Comparing mean/standard deviation results from a "virtual cohort" against clinical datasets.

6. Emergent Model Behavior: Showing that a finalized model reproduces known phenomena that weren't explicitly programmed into the equations (e.g., a blood flow model correctly predicting the onset of turbulence).

7. Model Plausibility: A rationale justifying the choice of governing equations and assumptions. This is often the starting point but is considered a weak form of evidence.

8. Calculation Verification/UQ using COU Simulations: Performing mesh convergence or uncertainty quantification on the final simulations used to answer the Question of Interest.

--------------------------------------------------------------------------------

Adequacy: The Totality of the Case

A critical distinction in the guidance is between applicability and adequacy.

• Applicability is the relevance of your validation activities to the COU.

• Adequacy is the final determination of whether the evidence is sufficient, considering the model risk.

In an adequacy assessment, you must explicitly state any limitations of the model and provide a rationale for why those limitations do not reduce confidence in the final decision. For example, if a model predicts energy absorption in an implant is well within safety thresholds and the uncertainty is small, that proximity to the threshold can be used to justify the model’s adequacy.

--------------------------------------------------------------------------------

Interacting with the FDA

The FDA recognizes that this framework involves significant judgment. Consequently, they recommend a "two-report" strategy for submissions:

1. The Modeling Study Report: Describing the model, parameters, and results.

2. The Credibility Assessment Report: A self-contained document providing the evidence and rationale for why the results can be trusted for the COU.

For novel or high-risk applications, the Q-submission pathway is highly encouraged. This allows manufacturers to present their Question of Interest, COU, and Risk Assessment to the FDA early in the process, ensuring the planned "ladder" of credibility factors is high enough to satisfy regulatory expectations before the real work begins.

Conclusion: Trusting the Digital Twin

As we move toward a future of in silico clinical trials and virtual patient cohorts, the ability to systematically demonstrate model credibility is paramount. This guidance moves the industry away from a "checklist" mentality and toward a robust, risk-based engineering argument. By following this framework, manufacturers can not only streamline their regulatory paths but also gain deeper insights into device performance, ultimately leading to safer and more effective care for the patients at the other end of the simulation.