Critically Appraising Clinical Literature and Weighing Evidence for Your CER

Critically Appraising Clinical Literature and Weighing Evidence for Your CER

Once your literature search is done, a lot of teams think the hard work is behind them. The search found papers; now you write up what they say. That's where things usually go wrong. Finding literature and critically appraising it are two completely different activities, and Notified Bodies have become quite good at spotting CERs where the second step was skipped or done poorly. A list of citations with brief summaries is not a clinical evidence base — it's a bibliography. The difference matters.

What critical appraisal actually involves

Critical appraisal means evaluating each source on its methodological merits before deciding how much weight it should carry in your analysis. You are asking: is this study designed in a way that makes its findings trustworthy? Is the patient population it studied comparable to yours? Are the outcomes it measured the ones that are clinically relevant for your device? A study can be well-published and still be largely irrelevant to your specific CER — and including it as if it were full-weight evidence is a mistake reviewers will catch.

The appraisal covers several dimensions for each source. Study design is the first one — a randomised controlled trial carries more inherent weight than a case series, all else being equal, because the design controls for confounding more effectively. But study design alone doesn't determine weight. A large, well-conducted registry study with your exact device population can outweigh a small RCT on a slightly different device variant. Risk of bias within the study matters just as much as study type: a poorly executed RCT with significant selection bias and high dropout rates may be less useful than a well-documented observational study.

The hierarchy of evidence and how to apply it

There is no single fixed hierarchy of evidence that MDR mandates you to use, but MDCG 2020-5 is clear that appraisal must consider the quality and applicability of each data source. In practice, a workable hierarchy for device CERs runs roughly from systematic reviews and meta-analyses at the top, through RCTs, prospective cohort studies, retrospective studies, case series, and expert opinion at the bottom. The important nuance is that this hierarchy describes the inherent reliability of the study design in the absence of other information — it is a starting point, not a rigid ranking.

What this means in practice: if you have a systematic review of your device technology but it was published eight years ago and your device incorporates a materially different design, it carries less weight than a more recent prospective study that used your current device generation. Recency, relevance of the study population, and the specific outcomes reported all modify the weight you assign. Document these modifying factors explicitly in your appraisal — "this systematic review was down-weighted due to its inclusion of earlier device generations not comparable to the subject device" is exactly the kind of reasoning reviewers want to see.

Applicability and the comparable patient population

One of the trickiest appraisal judgements is whether the population studied in a paper actually represents your intended patient population. A study on your device category in a general adult population has limited applicability if your intended purpose is a paediatric indication. A study conducted in a high-resource surgical centre has limited applicability if your device is intended for community or home use. These population differences don't automatically exclude a study from your evidence base, but they do constrain how much weight you can give it and what conclusions you can draw from it.

This is the part that catches a lot of teams off guard when Notified Bodies push back. The question isn't just "did this study use our device?" — it's "did this study generate data that is relevant to the specific claims and patient population in our CER?" You need to articulate that linkage for each included study. If a study is included despite limited population comparability, the appraisal section should acknowledge the limitation and explain why the data is still useful — perhaps because it's the best available evidence on a particular safety endpoint, or because the physiological mechanism is identical regardless of the specific population.

Handling unfavourable evidence

This is where many CERs develop their most significant weaknesses, and it's one of the first things an experienced reviewer will look for. If your literature search turned up studies with unfavourable findings — higher complication rates than your device claims, failure modes that appear in certain populations, comparator devices that outperformed yours on specific endpoints — those findings need to be addressed, not buried.

Addressing unfavourable evidence does not mean your benefit-risk conclusion has to be negative. It means you engage with the finding: you appraise the study quality, you assess whether the finding is applicable to your device and population, you explain what weight it should carry in the overall analysis, and you describe what it means for residual risk. A CER that ignores or glosses over unfavourable evidence is a significant audit finding. A CER that engages with it rigorously, weighs it fairly, and reaches a well-reasoned conclusion despite it is a strong document.

Building a coherent evidence base

After appraising individual sources, the CER needs to synthesise them into an overall picture. This is where the analysis section of the CER does its most important work. You're not just stacking up studies — you're building a coherent argument about what the evidence, taken as a whole, tells you about your device's safety and performance.

Synthesis means identifying where multiple sources converge on the same conclusion (and explaining why that convergence strengthens the finding) and where sources conflict (and explaining how you resolved the conflict). It means explaining what the strongest evidence tells you and what the weaker evidence adds at the margins. It means being explicit about remaining uncertainties — areas where the evidence base is thin, where study populations don't fully match yours, or where longer-term data doesn't yet exist. Those remaining uncertainties should map directly to your PMCF objectives: if you can't fully answer a clinical question from existing literature, your PMCF plan should be designed to close that gap.

Documenting appraisal in the CER

The appraisal needs to be documented at a level of detail that allows a reviewer to follow your reasoning. For each included study, a summary table or structured appraisal entry that covers study type, population, sample size, key findings, risk of bias assessment, and the weight assigned in the overall analysis is the minimum. For particularly important studies — the ones your safety or performance conclusions rely on most heavily — a more detailed narrative appraisal is appropriate.

One thing that helps: use a consistent appraisal framework across all included sources. You don't have to use a formal tool like GRADE or CASP, but whatever approach you use should be applied consistently. Inconsistency in appraisal depth — thorough for favourable studies, perfunctory for unfavourable ones — is a red flag that reviewers will notice immediately.