Draft search-strategy template for the synthesis series — submissions
The 19 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
19 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
Interoperability with published certainty guidance must be preserved in any revision
The respondent read search-strategy template for the synthesis series in draft and has confined this submission to a single provision.
The respondent, who has published on the implementation of certainty frameworks across health systems, states that the proposed distinction risks producing a five-level scale that no external assessment can be read against.
Worked examples showing how the distinction would be implemented in three assessment bodies accompanied the submission, together with a request that the final framework state its relationship to published guidance at first use.
The secretariat accepts this submission. Interoperability was the constraint that shaped the revision and the draft did not say so.
The framework states at first use that the Institute rating scale is the four-level scale used in published guidance, and the distinction between absent and conflicting evidence is carried in an accompanying formulation rather than as a fifth level.
Protocol amendments are permitted without a requirement to record when they were made
This submission addresses search-strategy template for the synthesis series from the standpoint of a reader who will encounter its output rather than its text.
The respondent states that the value of a registered protocol lies in the sequence, and that an amendment without a date cannot be shown to have preceded the results.
The respondent proposes that every amendment be dated and that its position relative to screening and analysis be recorded.
The secretariat accepts this submission. An undated amendment removes the property that registration exists to provide.
Every protocol amendment is now dated, is recorded against the stage of the review in progress at the time, and appears in the published protocol history, so that a reader can see which analyses were specified before the results were known.
The conditions for upgrading observational evidence are too permissive
The respondent’s comment on search-strategy template for the synthesis series arises from having applied a comparable framework to the same compounds.
The respondent states that the draft permits an upgrade for a large effect without requiring that confounding of the magnitude needed to produce it be shown to be implausible.
The respondent proposes that upgrading be removed from the framework entirely.
The respondent’s submission overlaps with submission 001 and was prepared without sight of it.
The secretariat accepts this submission in part. The conditions are tightened. Upgrading is retained, because a framework that cannot recognise a strong observational signal will misrate the cases where randomisation is not available.
An upgrade for a large effect now requires an explicit statement of the confounding structure that would be needed to produce the observed effect and a reason for regarding it as implausible, and the statement is published with the rating.
The framework should say when a rating must not be issued at all
This submission concerns the draft of search-strategy template for the synthesis series. The respondent assesses evidence for a national body and the observation arises from applying comparable guidance.
A certainty rating over an empty evidence base is a rating of nothing, and the machinery of domains and downgrades applied to no studies produces a number that looks like an assessment. The framework should forbid this rather than leave it to judgement.
The respondent proposes an explicit rule: where no eligible study exists for an outcome, no rating is issued and the outcome is recorded as unassessed.
This submission is made in the same spirit as submission 003 and on a different aspect of the draft.
The secretariat accepts this submission without qualification. It identifies a defect the Institute had not corrected.
Where no eligible study exists, no certainty rating is issued, no domain assessment is rendered, and the outcome is recorded as not assessed with the reason stated. The rule is applied retrospectively across the series.
A sortable table implies a comparison the underlying data do not support
The draft of search-strategy template for the synthesis series was read by a respondent whose concern is its interoperability with published certainty guidance.
The draft presents a sortable table whose columns are drawn from sources of differing quality. The respondent states that sorting on such a column produces an ordering that looks like a ranking and is not one.
The respondent proposes that sorting be disabled on any column whose values are not commensurable.
The secretariat notes this submission and records that the point is correct in principle.
No amendment arises here because every sortable table in the document set already carries a standing statement above it that the ordering is not a ranking and that the values in each column are commensurable only where the column header says so. The proposal to disable sorting was considered and not adopted, because a reader who cannot sort a table generally sorts it elsewhere and without the statement.
The document set should be published in translation
The respondent submits on search-strategy template for the synthesis series. The point would apply equally to any document in the series.
The respondent notes that the assessments concern compounds supplied internationally and that publishing only in English restricts access to the assessment to readers who work in it.
The respondent proposes machine translation of the document set as an interim measure, with human review of the certainty language.
The secretariat does not accept this submission, and records that the underlying point is sound and that the proposed remedy is the difficulty.
A translation whose certainty language has drifted is a different assessment carrying the Institute's name, and the Institute cannot review translations it does not have the capacity to review. The documents remain in English. The submission is published in full because the access problem it identifies is real and unresolved.
Preprints and conference abstracts are excluded categorically
This submission concerns search-strategy template for the synthesis series and a convention used across the Institute’s output.
The respondent states that a categorical exclusion removes results that are sometimes the only ones available, and that the reason for excluding them, absence of peer review, is a matter of degree rather than a category.
The respondent proposes that they be included with a downgrade.
The secretariat accepts this submission in part. Such sources are identified and reported and may inform an assessment. They do not contribute to a pooled estimate, because the reporting is generally insufficient for risk-of-bias assessment.
Preprints and conference abstracts are now screened, listed and reported as a distinct evidence class with their status stated, contribute to the narrative assessment, and are excluded from pooled estimates with the reason recorded rather than excluded at screening.
The assessment protocol is published after the assessment rather than before it
This is a submission on search-strategy template for the synthesis series.
A protocol published with the result cannot be distinguished from a description of what was done. Its value lies in being fixed before the evidence is seen.
The respondent proposes that protocols be published at registration, with any amendment dated.
The respondent has read submission 001 and asks that this submission be considered with it.
The secretariat accepts this submission.
Protocols are now published at registration, before screening begins, and amendments carry the date on which they were made and the reason.
The database set omits sources relevant to the compounds in scope
The respondent notes that search-strategy template for the synthesis series has to work where the evidence base is very thin as well as where it is deep, and submits with that in view.
The respondent, an information specialist, states that regional databases index trials of several compounds in the Institute's scope that are not indexed in the listed sources.
The respondent proposes that the named regional databases be added to the standard set.
The respondent supports submission 003 so far as it goes and adds the matter set out here.
The secretariat accepts this submission in part. Two of the named sources are added to the standard set. The remainder are added as conditional sources searched where the review question concerns a compound developed or registered in the relevant region.
The standard database set is extended, the conditional sources are named together with the trigger for searching them, and every review states which sources were searched and which were not, with the reason.
It is not clear which provisions bind the assessment committee
This submission concerns search-strategy template for the synthesis series and makes one point.
The respondent states that the document mixes requirements with descriptions of current practice in the same voice, so that a reader cannot tell which departures would be a breach and which would be a change of habit.
The respondent proposes that binding provisions be distinguished typographically and listed.
The secretariat accepts this submission. A rule indistinguishable from a description is not enforceable and does not reassure.
Binding provisions are now stated in a fixed form, are listed together in an annex, and a departure from any of them must be recorded in the document it affects with the reason, while descriptive passages are marked as descriptions of practice.
Nothing prevents a conclusion stronger than its certainty rating
The respondent submits on search-strategy template for the synthesis series. A framework of this kind is judged by whether two competent assessors applying it to the same evidence reach the same rating.
The respondent states that the framework rates certainty and then leaves conclusion wording to the author, so that a low rating and a confident conclusion can coexist in one document.
The respondent proposes that conclusion wording be drawn from a fixed set tied to the rating.
The secretariat accepts this submission. The coupling is the mechanism by which a rating changes what a reader takes away.
Conclusion wording is now drawn from a fixed set of formulations tied to the certainty rating, the mapping is published in the methodology document, and a document whose conclusion does not match its rating cannot be ratified.
Automated assistance in screening should be disclosed and validated
The respondent has read search-strategy template for the synthesis series in draft and makes a single submission.
The respondent asks whether any automated tool is used in screening or data extraction, and states that if one is, its performance should be reported in the same way as a human screener's agreement.
The respondent proposes that automated assistance be prohibited.
The secretariat accepts this submission in part. Disclosure and validation are adopted. A prohibition is not, because the alternative to a validated tool is not a human but a smaller search.
The methodology now requires that any automated assistance in screening or extraction be disclosed in the review, that its output be verified by a human for every included record, and that its measured performance against human screening be reported.
No information-size criterion is applied before an interval is judged
Having read the draft under consultation, which concerns search-strategy template for the synthesis series, the respondent submits as follows.
A narrow interval obtained from a small number of events may still be the product of chance, and the framework as drafted would rate it precise. An information-size criterion is the standard remedy.
The respondent proposes that a criterion be applied before precision is judged.
The respondent read submission 006 after drafting this one and has not altered it, the two points being distinct.
The secretariat accepts the principle and applies it as a check rather than as a rule.
An information-size check is now applied and its result recorded. It is not applied mechanically, because a criterion computed on an assumed effect size can itself be the more arbitrary judgement, and the domain note states which consideration governed.
Indirectness is defined so broadly that any evidence could be downgraded under it
The respondent submits on search-strategy template for the synthesis series.
The respondent states that the definition covers differences in population, intervention, comparator, outcome and setting without any threshold, so that an assessor who wishes to downgrade can always find a ground.
The respondent proposes that the assessment name the specific difference and state why it would be expected to change the effect.
The secretariat accepts this submission. A criterion that can always be satisfied is not a criterion.
An indirectness downgrade now requires the assessor to name the specific difference and to state the mechanism by which it would be expected to change the effect, and the statement is published with the rating so that it can be disputed.
The risk-of-bias instrument is not named, so a judgement cannot be reproduced
This is a submission on search-strategy template for the synthesis series, made from a statistical standpoint.
The framework requires risk of bias to be assessed and does not say against what. Two assessors using different instruments will reach different domain judgements on the same trial, and neither could be said to have applied the framework incorrectly.
The respondent proposes that the instrument and its version be named, and recorded with every assessment.
The secretariat accepts this submission.
The instrument and version are now named in the framework and recorded on every assessment, so that a judgement can be checked against the instrument that produced it.
The search date is not on the face of the document
Having read the draft of search-strategy template for the synthesis series, the respondent puts one point to the committee.
The draft carries a publication date and a review date but not the date on which the evidence was last searched. Those are three different dates and only the third tells a reader how current the assessment is. A document published in one quarter may rest on a search run two quarters earlier, and nothing on the page allows that gap to be measured.
The respondent proposes that the search date be printed adjacent to every certainty rating rather than in the methods section, on the ground that a reader who acts on a rating is unlikely to have read the methods section first.
The secretariat accepts this submission. The distinction between publication, review and search dates is real and the draft did not make it visible where it mattered.
The search date is now printed adjacent to every certainty rating and is carried in the document metadata, so that the interval between the search and the reading is visible without reference to the methods section.
The four rating labels are not self-explanatory to a reader who meets them once
The respondent has read search-strategy template for the synthesis series in draft and makes one submission.
High, moderate, low and very low describe confidence in an estimate and are read as descriptions of the size of an effect. A reader meeting the words once will take a low rating as a small benefit.
The respondent proposes a standing one-sentence gloss printed with every rating, not merely defined in a glossary.
The secretariat accepts this submission.
Every rating now carries a one-sentence gloss at the point of use, stating that the rating describes confidence in the estimate and not the size of the effect.
The framework does not say what new evidence would change a rating
The respondent read search-strategy template for the synthesis series in draft. The point applies to it and to the series generally.
A rating is a judgement about a body of evidence at a date. Without a statement of what would change it, a reader cannot tell whether a newly published trial is material.
The respondent proposes that every rating carry a statement of the finding that would alter it.
The respondent notes submission 013 above and does not repeat the ground it covers.
The secretariat accepts this submission.
Every rating now carries a statement of what would change it, and the search date against which it was made. A trial meeting the stated description triggers reassessment ahead of the ordinary cycle.
Non-randomised evidence is admitted for harms and excluded for benefits without a stated basis
The respondent submits on search-strategy template for the synthesis series, on a matter that is not specific to this draft but is visible in it.
The asymmetry is conventional and defensible, and the framework applies it without stating it, so a reader encountering an observational study in a harms section and not in an efficacy section will read the difference as an oversight.
The respondent proposes that the asymmetry be stated and justified.
This submission is made in the same spirit as submission 008 and on a different aspect of the draft.
The secretariat accepts the requirement to state and declines to change the practice.
The asymmetry is now stated and justified in the framework: rare and long-latency harms are frequently observable only in non-randomised data, while a benefit observable only in non-randomised data is more often a product of the selection that produced it.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.