Searching systematically: literature searching in a systematic way

According to the Centre for Evidence Based Practice in Oxford UK critical appraisal is the systematic evaluation of clinical research papers in order to establish whether a study:

1. Addresses a clearly focused question?

2. Uses valid methods to address this question?

3. Finds valid results which are important?

4. Finds results which are applicable to my patient or population?

If the study in question does not adequately answer all of these questions, there is no point continuing with it.

    The CASP checklists are critical appraisal tools designed to be used when reading research.

Checklists  are available for:

• Systematic Reviews
• Randomised Controlled Trials
• Cohort Studies
• Case Control Studies
• Economic Evaluations
• Diagnostic Studies
• Qualitative studies
• Clinical Prediction Rule

These are free to download and can be used by anyone under the Creative Commons License.

The Centre for Evidence Based Medicine provides worksheets to assist with the critical appraisal of various publication types:

• Systematic Review
• Diagnosis
• Prognosis
• Therapy / RCT
• Qualitative Studies

Each worksheet is a downloadable pdf document which follows a standard way of appraising evidence based on the publication type. Translations of the worksheets are available for a variety of languages. 

AMSTAR is a checklist for systematic reviews.  

It was  developed to evaluate systematic reviews of randomised trials.

SIGN Guidelines: methodical assessment of studies checklists

Includes checklists and notes for:

• Systematic Reviews and Meta Analysis
• RCTs
• Cohort Studies
• Case Control Studies
• Economic Studies
• Diagnostic Studies

BMJ Best Practice Critical Appraisal Tools

This site includes checklists for different study types including. 

• RCTs
• Diagnostic test studies
• Systematic Reviews.....

The Catalogue of Bias   - produced by the Centre for Evidence-Based Medicine. 

This is a very useful collection of Biases which includes full descriptions, citation and references.  List is being continually updated by the CEBM.

Examples of bias included in the collection are:

Adherence Bias:  "A systematic distortion in outcome data that arises when participants who adhere to a study protocol or intervention differ from those who do not adhere, when that difference relates to the outcome of interest".

Confounding Bias:  "A distortion that modifies an association between an exposure and an outcome because a factor is independently associated with the exposure and the outcome."  - ie an association is suggested where none exists or masks a true association

Hawthorne effect: "When individuals modify an aspect of their behaviour in response to their awareness of being observed".

Language Bias:   "Publication of research findings in a particular language".

Publication Bias: "When the likelihood of a study being published is affected by the findings of the study".

Chapter 7: Considering bias and conflicts of interest among the included studies

"Review authors should seek to minimize bias. We draw a distinction between two places in which bias should be considered. The first is in the results of the individual studies included in a systematic review. The second is in the result of the meta-analysis (or other synthesis) of findings from the included studies".

 Boutron I, Page MJ, Higgins JPT, Altman DG, Lundh A, Hróbjartsson A. Chapter 7: Considering bias and conflicts of interest among the included studies [last updated August 2022]. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.5. Cochrane, 2024. Available from www.training.cochrane.org/handbook.