Draft synthesis: Percentage weight change as a surrogate… — submissions
The 9 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
9 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
Registered trials that never reported are not counted anywhere in the review
The respondent states that the screening flow accounts for records retrieved and excluded but does not record registered trials identified with no posted result, which are neither included nor excluded and simply disappear.
The respondent proposes that they be counted and reported as a category, with the proportion of the registered evidence base they represent.
The secretariat accepts this submission. A review that cannot say how much of the evidence base is unreported cannot say how much weight its own estimate deserves.
Registered trials without posted results are now identified, counted and reported as a distinct category in the screening flow, with the proportion of registered participants they represent stated in the limitations.
Declared interests should appear on the document rather than on a separate page
The draft links to a central conflicts register. The respondent argues that a reader assessing whether to rely on a particular document should not have to leave it to find out who assessed it and what they declared.
The respondent proposes that the interests of every named contributor to a document be printed on that document.
The secretariat notes this submission and records that the draft already provides for it, which the respondent could reasonably have missed because the provision sits in an appendix.
Every document carries the declared interests of its named contributors in its front matter, and the central register exists so that a reader can see a person across all documents rather than one at a time. No amendment arises; the provision has been moved from the appendix into the body of the methodology document so that it is findable.
The choice of effect measure is not justified and changes the appearance of the result
The draft reports relative effects for benefits and absolute effects for harms. The respondent states that the combination flatters the intervention and that the choice should be justified or made uniform.
The respondent proposes that both relative and absolute effects be reported for every outcome.
The secretariat accepts this submission in part. Both measures are reported for every outcome where the baseline risk needed for the absolute effect can be stated. Where it cannot, the relative effect is reported alone with the reason.
Every outcome now reports the relative effect and, where an assumed baseline risk can be stated and sourced, the corresponding absolute effect, with the baseline risk and its source given in the same row.
Subgroup findings are reported that were not registered in the protocol
The draft reports differences between subgroups that do not appear in the registered protocol. The respondent states that unregistered subgroup analysis is hypothesis-generating and that the draft presents it in the same form as the pre-specified results.
The respondent proposes that unregistered analyses be removed.
The secretariat accepts this submission in part. The analyses are retained and relabelled rather than removed, because removing an analysis that was conducted leaves no record that it was.
Every subgroup analysis is now labelled as pre-specified or post hoc against the registered protocol, post hoc analyses are reported in a separate subsection without a certainty rating, and the protocol version against which the labelling was made is stated.
Intention-to-treat and efficacy-estimand results are combined without distinction
Several contributing trials report both a treatment-policy result and an efficacy result that censors at discontinuation. The draft draws from whichever is reported first in each paper.
The respondent, a trial statistician, states that the two answer different questions, that the difference is substantial where discontinuation is common, and that a synthesis should choose one and say which.
The secretariat accepts this submission. Mixing estimands within a single pooled estimate is a defect of the synthesis rather than of the trials.
The treatment-policy estimand is used throughout as the primary analysis, the efficacy estimand is reported as a secondary analysis where available, and the estimand used is stated in every row of the summary of findings.
The conclusion is stated more strongly than the certainty rating supports
The draft rates the anchor outcome at low certainty and then states in the conclusion that the intervention is effective. The respondent states that the two sentences cannot both be true as written.
The respondent proposes that conclusion language be tied mechanically to the certainty rating, so that a low rating cannot produce an unqualified claim.
The secretariat accepts this submission. The mismatch is the failure mode the certainty framework exists to prevent.
Conclusion wording is now drawn from a fixed set of formulations tied to the certainty rating, so that a low certainty rating produces a statement that the evidence may suggest an effect and that the estimate is likely to change with further research.
Point estimates are given without an interval
Several estimates in the draft appear as single figures. The respondent states that a point estimate without an interval invites a precision the underlying data do not support, and that the effect is worst where the estimate is drawn from a small contributing set.
The respondent proposes that no point estimate appear anywhere in the document set without its interval, including in summary tables and in the abstract.
The secretariat accepts this submission in part. Intervals are added wherever the source reports one. The proposal is declined for figures the source published without an interval, because the Institute will not compute an interval a source did not report.
Every estimate now carries its interval where the source reported one, and where it did not, the estimate is annotated as reported without an interval rather than left to appear as a precise figure.
Results at materially different follow-up durations are pooled
Contributing trials report the outcome at durations ranging across more than a year. The draft pools them into a single estimate. The respondent states that a mean at one duration and a mean at another are not estimates of the same quantity when the effect is still changing.
The respondent proposes that estimates be reported by duration band.
The secretariat accepts this submission. Pooling across durations assumes a plateau the contributing trials do not demonstrate.
Estimates are now reported by duration band, with the number of contributing trials and participants in each band stated, and no estimate is pooled across bands.
The document should state what a reader ought to do
The draft assesses evidence and stops. The respondent, a practising clinician, states that a reader arriving at the document with a decision to make is left to convert an assessment into an action without help, and proposes that each document close with a recommendation.
The respondent argues that other evidence bodies issue recommendations and that declining to do so transfers the difficult part of the work to the reader.
The secretariat does not accept this submission, and records that the point is a reasonable one rather than a misunderstanding.
The Institute assesses evidence and does not issue recommendations, because a recommendation embeds values and a resource context that the Institute does not hold and cannot state. That constitutional limit is published on the methodology page and is not varied by consultation. The submission remains published in full.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.