Draft search-strategy template for the synthesis series — disposition

§3Disposition and amendments

§3.1Disposition table

Table 1. Every submission with its disposition. Each row links to the submission in full.

§3.2Summary by disposition

Table 2. Counts by disposition, with the meaning of each.

§3.3Resulting amendments

1. Interoperability with published certainty guidance must be preserved in any revision — arising from DRAFT-SEARCH-STRAT/001. The framework states at first use that the Institute rating scale is the four-level scale used in published guidance, and the distinction between absent and conflicting evidence is carried in an accompanying formulation rather than as a fifth level.
2. Protocol amendments are permitted without a requirement to record when they were made — arising from DRAFT-SEARCH-STRAT/002. Every protocol amendment is now dated, is recorded against the stage of the review in progress at the time, and appears in the published protocol history, so that a reader can see which analyses were specified before the results were known.
3. The conditions for upgrading observational evidence are too permissive — arising from DRAFT-SEARCH-STRAT/003. An upgrade for a large effect now requires an explicit statement of the confounding structure that would be needed to produce the observed effect and a reason for regarding it as implausible, and the statement is published with the rating.
4. The framework should say when a rating must not be issued at all — arising from DRAFT-SEARCH-STRAT/004. Where no eligible study exists, no certainty rating is issued, no domain assessment is rendered, and the outcome is recorded as not assessed with the reason stated. The rule is applied retrospectively across the series.
5. Preprints and conference abstracts are excluded categorically — arising from DRAFT-SEARCH-STRAT/007. Preprints and conference abstracts are now screened, listed and reported as a distinct evidence class with their status stated, contribute to the narrative assessment, and are excluded from pooled estimates with the reason recorded rather than excluded at…
6. The assessment protocol is published after the assessment rather than before it — arising from DRAFT-SEARCH-STRAT/008. Protocols are now published at registration, before screening begins, and amendments carry the date on which they were made and the reason.
7. The database set omits sources relevant to the compounds in scope — arising from DRAFT-SEARCH-STRAT/009. The standard database set is extended, the conditional sources are named together with the trigger for searching them, and every review states which sources were searched and which were not, with the reason.
8. It is not clear which provisions bind the assessment committee — arising from DRAFT-SEARCH-STRAT/010. Binding provisions are now stated in a fixed form, are listed together in an annex, and a departure from any of them must be recorded in the document it affects with the reason, while descriptive passages are marked as descriptions of practice.
9. Nothing prevents a conclusion stronger than its certainty rating — arising from DRAFT-SEARCH-STRAT/011. Conclusion wording is now drawn from a fixed set of formulations tied to the certainty rating, the mapping is published in the methodology document, and a document whose conclusion does not match its rating cannot be ratified.
10. Automated assistance in screening should be disclosed and validated — arising from DRAFT-SEARCH-STRAT/012. The methodology now requires that any automated assistance in screening or extraction be disclosed in the review, that its output be verified by a human for every included record, and that its measured performance against human screening be reported.
11. No information-size criterion is applied before an interval is judged — arising from DRAFT-SEARCH-STRAT/013. An information-size check is now applied and its result recorded. It is not applied mechanically, because a criterion computed on an assumed effect size can itself be the more arbitrary judgement, and the domain note states which consideration governed.
12. Indirectness is defined so broadly that any evidence could be downgraded under it — arising from DRAFT-SEARCH-STRAT/014. An indirectness downgrade now requires the assessor to name the specific difference and to state the mechanism by which it would be expected to change the effect, and the statement is published with the rating so that it can be disputed.
13. The risk-of-bias instrument is not named, so a judgement cannot be reproduced — arising from DRAFT-SEARCH-STRAT/015. The instrument and version are now named in the framework and recorded on every assessment, so that a judgement can be checked against the instrument that produced it.
14. The search date is not on the face of the document — arising from DRAFT-SEARCH-STRAT/016. The search date is now printed adjacent to every certainty rating and is carried in the document metadata, so that the interval between the search and the reading is visible without reference to the methods section.
15. The four rating labels are not self-explanatory to a reader who meets them once — arising from DRAFT-SEARCH-STRAT/017. Every rating now carries a one-sentence gloss at the point of use, stating that the rating describes confidence in the estimate and not the size of the effect.
16. The framework does not say what new evidence would change a rating — arising from DRAFT-SEARCH-STRAT/018. Every rating now carries a statement of what would change it, and the search date against which it was made. A trial meeting the stated description triggers reassessment ahead of the ordinary cycle.
17. Non-randomised evidence is admitted for harms and excluded for benefits without a stated basis — arising from DRAFT-SEARCH-STRAT/019. The asymmetry is now stated and justified in the framework: rare and long-latency harms are frequently observable only in non-randomised data, while a benefit observable only in non-randomised data is more often a product of the selection that produced it.

References cited on this page

References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.

1. International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute

Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.