Draft synthesis: Weight-reducing agents for obstructive sleep… — submissions
The 16 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
16 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
The timepoint at which each outcome was extracted was chosen after the data were seen
This is a submission on the draft review of In adults with obesity and moderate to severe obstructive sleep apnoea, what is the effect of a pharmacological weight-reducing agent on the…, made from a statistical standpoint.
The methods state that outcomes were extracted at the latest available timepoint. Several contributing trials report the same outcome at three timepoints, and taking the latest is a choice that can be made in the knowledge of what each shows.
The respondent proposes that the extraction timepoint be prespecified in the protocol, and that where it was not, the review report the estimate at every reported timepoint so that the choice is visible.
The secretariat accepts this submission. An extraction rule applied after the data are visible is a degree of freedom and should be recorded as one.
The extraction timepoint is now prespecified in the protocol for new reviews. Where a review predates the requirement, estimates are reported at every timepoint the contributing trials report.
Point estimates are given without an interval
The respondent’s comment on the draft review of In adults with obesity and moderate to severe obstructive sleep apnoea, what is the effect of a pharmacological weight-reducing agent on the… arises from comparing the included set against the respondent’s own knowledge of the field.
Several estimates in the draft appear as single figures. The respondent states that a point estimate without an interval invites a precision the underlying data do not support, and that the effect is worst where the estimate is drawn from a small contributing set.
The respondent proposes that no point estimate appear anywhere in the document set without its interval, including in summary tables and in the abstract.
This submission is made by a respondent with a different professional interest in the outcome from the one behind submission 001.
The secretariat accepts this submission in part. Intervals are added wherever the source reports one. The proposal is declined for figures the source published without an interval, because the Institute will not compute an interval a source did not report.
Every estimate now carries its interval where the source reported one, and where it did not, the estimate is annotated as reported without an interval rather than left to appear as a precise figure.
Results at materially different follow-up durations are pooled
This submission concerns In adults with obesity and moderate to severe obstructive sleep apnoea, what is the effect of a pharmacological weight-reducing agent on the… and makes one point.
Contributing trials report the outcome at durations ranging across more than a year. The draft pools them into a single estimate. The respondent states that a mean at one duration and a mean at another are not estimates of the same quantity when the effect is still changing.
The respondent proposes that estimates be reported by duration band.
The secretariat accepts this submission. Pooling across durations assumes a plateau the contributing trials do not demonstrate.
Estimates are now reported by duration band, with the number of contributing trials and participants in each band stated, and no estimate is pooled across bands.
Doses differing several-fold are pooled without examining dose-response
The respondent has read In adults with obesity and moderate to severe obstructive sleep apnoea, what is the effect of a pharmacological weight-reducing agent on the… in draft and makes a single submission.
Contributing trials administer doses that differ by a factor of several. The draft pools them and reports one estimate. The respondent states that where a dose-response relationship exists the pooled figure corresponds to no dose that anyone receives.
The respondent proposes that estimates be reported by dose and that dose-response be examined where the data allow.
The secretariat accepts this submission. A pooled estimate across doses is an estimate for an average dose that no protocol specifies.
Estimates are now reported by dose group, a dose-response examination is reported where three or more dose levels contribute, and the pooled across-dose estimate is removed rather than retained alongside.
Studies using different outcome definitions are pooled into one estimate
The respondent notes that the review question — In adults with obesity and moderate to severe obstructive sleep apnoea, what is the effect of a pharmacological weight-reducing agent on the… — is answerable only if the contributing trials measured the same thing, and submits with that in view.
The contributing studies define the outcome in at least two ways, and the draft pools them. The respondent states that the resulting estimate is an average across definitions rather than an estimate of any one quantity.
The respondent proposes that studies be pooled only within a definition, and that the definitions be reported separately with their own certainty ratings.
The respondent has read submission 001 above and makes this submission independently of it.
The secretariat accepts this submission. Pooling across definitions produces a figure with no referent, and the draft did it without saying so.
Studies are now pooled only within an outcome definition, each definition is reported with its own estimate and certainty rating, and the summary of findings states which definition each row concerns.
The search was limited by publication date and the limit is not justified
The respondent has read the draft synthesis on In adults with obesity and moderate to severe obstructive sleep apnoea, what is the effect of a pharmacological weight-reducing agent on the… and makes one submission.
The search strategy carries a start date some years before the review. No reason is given. If the limit reflects the date of first synthesis of the compound it is defensible and should be said; if it reflects convenience it is a source of bias.
The respondent proposes that every date limit carry its justification in the search-strategy section.
Submission 004 raises an adjacent matter. The respondent regards the two as separable and addresses only this one.
The secretariat accepts the requirement to justify and declines to remove limits that are justified.
Every date limit in the series now carries its justification in the search-strategy section. Limits that could not be justified on re-examination have been removed and the affected searches re-run.
The axis of the forest plot exaggerates a small difference
The draft review of In adults with obesity and moderate to severe obstructive sleep apnoea, what is the effect of a pharmacological weight-reducing agent on the… was read against its registered protocol.
The plot is drawn on an axis spanning a narrow range, so an effect of no clinical consequence occupies most of the width of the figure. A reader who takes the visual impression rather than the number will overstate the finding.
The respondent proposes that a minimally important difference be marked on every forest plot where one has been established for the outcome.
The secretariat accepts the proposal where a minimally important difference exists and declines to invent one where it does not.
Forest plots now carry a marked minimally important difference wherever a published one exists for the outcome, with its source cited. Where none exists the figure says so beneath the axis.
An indirect comparison is presented without an assessment of transitivity
The respondent read In adults with obesity and moderate to severe obstructive sleep apnoea, what is the effect of a pharmacological weight-reducing agent on the… in draft. The point applies to it and to the series generally.
The draft compares two interventions through a common comparator. The respondent states that the trials contributing to each side differ in background therapy and in baseline severity, and that the resulting estimate assumes a similarity the draft does not test.
The respondent proposes that transitivity be assessed explicitly and reported before any indirect estimate is presented.
The secretariat accepts this submission. An indirect estimate presented without a transitivity assessment invites a reader to treat an assumption as a result.
Transitivity is now assessed against a stated list of effect modifiers and reported before any indirect estimate, and where the assessment fails the contrast is reported as unestimable with the reason rather than estimated with a caveat.
The comparator was given at a dose below the one in general use
This submission addresses the draft synthesis on In adults with obesity and moderate to severe obstructive sleep apnoea, what is the effect of a pharmacological weight-reducing agent on the… from the standpoint of a reader who will use the summary and not the appendices.
Two contributing trials compare the intervention against an active comparator titrated to a dose lower than that reached in ordinary practice. Pooling them with trials using a full comparator dose produces an estimate that flatters the intervention.
The respondent proposes that comparator dose be tabulated in the included-studies table and that a sensitivity analysis restricted to full-dose comparators be reported.
The secretariat accepts this submission. Comparator dose is a condition of an effect estimate and was not being recorded.
Comparator dose is now a column in the included-studies table, and a sensitivity analysis restricted to comparators at the dose in general use is reported wherever the trials permit it.
One trial appears to be included twice under two publications
The respondent submits on In adults with obesity and moderate to severe obstructive sleep apnoea, what is the effect of a pharmacological weight-reducing agent on the…. The point would apply equally to any document in the series.
Two included records report overlapping participant numbers, the same enrolment window and the same sponsor. If they are the same trial reported twice, its weight in the pooled estimate is doubled.
The respondent proposes that the review record, for each included study, the trial identifier rather than the publication, and cluster publications under it.
The secretariat accepts this submission. Counting publications rather than trials is the commonest way a review overstates its evidence base.
Included studies are now recorded against a trial identifier, with publications listed beneath it, and the included-studies table reports the number of trials and the number of reports separately.
Risk-of-bias judgements are reported as an overall rating without the domains
This submission concerns the draft review of In adults with obesity and moderate to severe obstructive sleep apnoea, what is the effect of a pharmacological weight-reducing agent on the…. The respondent has conducted reviews on adjacent questions and the observation arises from that experience.
The draft reports a single overall risk-of-bias judgement per study. The respondent states that the overall judgement conceals which domain drove it, and that a reader assessing whether the concern applies to their question needs the domain.
The respondent proposes a domain-level table with the supporting quotation for each judgement.
The secretariat accepts this submission. The overall judgement is a summary of the domains and publishing only the summary makes it uncheckable.
Risk of bias is now reported at domain level for every included study, with the text on which each judgement was based quoted and referenced, and the overall judgement derived from the domains rather than asserted alongside them.
References should carry a persistent identifier for every cited source
The respondent read the draft review of In adults with obesity and moderate to severe obstructive sleep apnoea, what is the effect of a pharmacological weight-reducing agent on the… and has confined this submission to a single matter.
Several references in the draft carry a journal, a year and a volume but no persistent identifier. The respondent states that retrieval of such a reference is materially slower and that identifiers should be supplied throughout.
The respondent asks in the alternative that where an identifier exists but is not carried, the omission be explained rather than left as a gap the reader must interpret.
The secretariat accepts the second limb of this submission and declines the first. Identifiers are supplied wherever the Institute holds one. Where the Institute does not hold an identifier it will not supply one, because a reconstructed identifier that resolves to the wrong record is a worse defect than an absent one.
Every reference without a persistent identifier now carries an explicit statement that the identifier is not held by the Institute, so that its absence is a recorded fact rather than an apparent oversight.
A superseded version should remain reachable from the version that replaced it
Having read the draft synthesis on In adults with obesity and moderate to severe obstructive sleep apnoea, what is the effect of a pharmacological weight-reducing agent on the…, the respondent puts one point to the committee.
The respondent states that the draft supersedes an earlier document and that a reader who cited the earlier version has no way to reach it from the new one, which makes it impossible to see what changed.
The respondent asks that every version carry a link both to what it supersedes and to what supersedes it.
The respondent has read submission 008 and asks that this submission be considered with it.
The secretariat notes this submission. The corrections and versioning policy already requires bidirectional version links and every superseded document is retained at its own address.
No amendment arises. The requirement is stated in the corrections and versioning policy and the amendment log of this document links to the version it replaced. The respondent is correct that the link was absent from the draft page furnished for consultation, which was a defect of the consultation copy and not of the policy.
The search strategy as published cannot be re-run
The respondent submits on the draft synthesis addressing In adults with obesity and moderate to severe obstructive sleep apnoea, what is the effect of a pharmacological weight-reducing agent on the…. This submission concerns the methods rather than the conclusion, on the view that a conclusion is only as good as the search that produced it.
The strategy is described in prose rather than reproduced as executed. The respondent, an information specialist, attempted to reconstruct it and obtained a different yield, which may reflect the reconstruction or the original.
The respondent proposes that the strategy be published line by line as run in each database, with the interface, the date and the number of records retrieved at each line.
The respondent has read submission 001 and puts a further matter to the secretariat.
The secretariat accepts this submission. A strategy that cannot be re-run cannot be checked, and a review whose search cannot be checked rests on an assertion.
The full line-by-line strategy for each database is now published with the interface, the run date and the yield at each line, and the total retrieved is reconciled against the screening flow.
The document should not describe uses outside the approved indication
Having read the draft under consultation, which concerns In adults with obesity and moderate to severe obstructive sleep apnoea, what is the effect of a pharmacological weight-reducing agent on the…, the respondent submits as follows.
The submission is made on behalf of a marketing-authorisation holder. It states that the draft describes uses of the compound that fall outside the approved indication, that such uses are not supported by the sponsor, and that describing them may be read as legitimising them.
The sponsor asks that the sections concerned be removed, or in the alternative that they carry a prominent statement that the sponsor does not support such use.
The secretariat does not accept this submission. The Institute records what is supplied and what is used, because a reader encountering a compound outside a regulated route is the reader most in need of an assessment of the evidence for it.
The sections remain. Every such section already states the regulatory status of the use described and states that the evidence for it is assessed separately from the evidence for the approved indication. The submission is published in full and identified as an industry submission.
Registered trials that never reported are not counted anywhere in the review
The respondent submits on In adults with obesity and moderate to severe obstructive sleep apnoea, what is the effect of a pharmacological weight-reducing agent on the….
The respondent states that the screening flow accounts for records retrieved and excluded but does not record registered trials identified with no posted result, which are neither included nor excluded and simply disappear.
The respondent proposes that they be counted and reported as a category, with the proportion of the registered evidence base they represent.
The secretariat accepts this submission. A review that cannot say how much of the evidence base is unreported cannot say how much weight its own estimate deserves.
Registered trials without posted results are now identified, counted and reported as a distinct category in the screening flow, with the proportion of registered participants they represent stated in the limitations.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.