Projects per year
Abstract
Objective: To review the findings of studies that have evaluated the design and/or usability of key risk of bias (RoB) tools for the assessment of RoB in primary studies, as categorised by the LATITUDES Network (a searchable library of RoB tools for evidence synthesis): PROBAST, RoB2, ROBINS-I, QUADAS-2, QUADAS-C, QUAPAS, ROBINS-E, and the COSMIN RoB checklist.
Study design and setting: Systematic review of methodological studies. We conducted a forward citation search from the primary report of each tool, to identify primary studies aiming to evaluate the design and/or usability of the tool. Two reviewers assessed studies for inclusion. We extracted tool features into Microsoft Word and used NVivo for document analysis, comprising a mix of deductive and inductive approaches. We summarised findings within each tool and explored common findings across tools.
Results: We identified 13 tool evaluations meeting our inclusion criteria: PROBAST (3); RoB2 (3); ROBINS-I (4); QUADAS-2 (3). We identified no evaluations for the other tools. Evaluations varied in clinical topic area, methodology, approach to bias assessment and tool user background. Some had limitations affecting generalisability. We identified common findings across tools for 6/14 themes: 1) challenging items (e.g. RoB2/ROBINS-I “deviations from intended interventions” domain), 2) overall RoB judgement (concerns with overall risk calculation in PROBAST/ROBINS-I), 3) tool usability (concerns about complexity), 4) time to complete tool (varying demands on time e.g. depending on number of outcomes assessed), 5) user agreement (varied across tools), and 6) recommendations for future use (e.g. piloting) and development (add intermediate domain answer to QUADAS-2/PROBAST; provide clearer guidance for all tools). Of the other eight themes, seven only had findings for the QUADAS-2 tool, limiting comparison across tools, and one (“re-organisation of questions”) had no findings.
Conclusion: Evaluations of key RoB tools have posited common challenges and recommendations for tool use and development. These findings may be helpful to people using or developing RoB tools. Guidance is necessary to support the design and implementation of future RoB tool evaluations.
Study design and setting: Systematic review of methodological studies. We conducted a forward citation search from the primary report of each tool, to identify primary studies aiming to evaluate the design and/or usability of the tool. Two reviewers assessed studies for inclusion. We extracted tool features into Microsoft Word and used NVivo for document analysis, comprising a mix of deductive and inductive approaches. We summarised findings within each tool and explored common findings across tools.
Results: We identified 13 tool evaluations meeting our inclusion criteria: PROBAST (3); RoB2 (3); ROBINS-I (4); QUADAS-2 (3). We identified no evaluations for the other tools. Evaluations varied in clinical topic area, methodology, approach to bias assessment and tool user background. Some had limitations affecting generalisability. We identified common findings across tools for 6/14 themes: 1) challenging items (e.g. RoB2/ROBINS-I “deviations from intended interventions” domain), 2) overall RoB judgement (concerns with overall risk calculation in PROBAST/ROBINS-I), 3) tool usability (concerns about complexity), 4) time to complete tool (varying demands on time e.g. depending on number of outcomes assessed), 5) user agreement (varied across tools), and 6) recommendations for future use (e.g. piloting) and development (add intermediate domain answer to QUADAS-2/PROBAST; provide clearer guidance for all tools). Of the other eight themes, seven only had findings for the QUADAS-2 tool, limiting comparison across tools, and one (“re-organisation of questions”) had no findings.
Conclusion: Evaluations of key RoB tools have posited common challenges and recommendations for tool use and development. These findings may be helpful to people using or developing RoB tools. Guidance is necessary to support the design and implementation of future RoB tool evaluations.
| Original language | English |
|---|---|
| Article number | 111370 |
| Journal | Journal of Clinical Epidemiology |
| Volume | 171 |
| Early online date | 24 Apr 2024 |
| DOIs | |
| Publication status | Published - 1 Jul 2024 |
Bibliographical note
Publisher Copyright:© 2024 The Author(s)
Research Groups and Themes
- BrisTAG
Fingerprint
Dive into the research topics of 'Common challenges and suggestions for risk of bias tool development: a systematic review of methodological studies '. Together they form a unique fingerprint.Projects
- 1 Active
-
Bristol TAG
Whiting, P. F. (Co-Principal Investigator), Welton, N. J. (Co-Principal Investigator), Owen-Smith, A. L. (Co-Investigator), Phillippo, D. M. (Co-Investigator), Caldwell, D. (Co-Investigator), Marques, E. M. R. (Co-Investigator), Jones, H. E. (Co-Investigator), Thom, H. H. Z. (Co-Investigator), Pedder, H. (Co-Investigator), Savovic, J. (Co-Investigator), Higgins, J. P. T. (Co-Investigator), Sadek, A. S. (Co-Investigator), Lopez Manzano, C. (Co-Investigator), Cooper, C. (Co-Investigator), Tomlinson, E. (Co-Investigator), Carroll, J. S. (Co-Investigator), Walker, J. G. (Co-Investigator), Ward, M. E. (Co-Investigator) & James, R. M. (Co-Investigator)
1/04/22 → 31/03/27
Project: Research, Parent