Draft revision of the Institute’s certainty assessment… — submissions
The 15 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
15 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
It is not clear which provisions bind the assessment committee
The respondent states that the document mixes requirements with descriptions of current practice in the same voice, so that a reader cannot tell which departures would be a breach and which would be a change of habit.
The respondent proposes that binding provisions be distinguished typographically and listed.
The secretariat accepts this submission. A rule indistinguishable from a description is not enforceable and does not reassure.
Binding provisions are now stated in a fixed form, are listed together in an annex, and a departure from any of them must be recorded in the document it affects with the reason, while descriptive passages are marked as descriptions of practice.
The database set omits sources relevant to the compounds in scope
The respondent, an information specialist, states that regional databases index trials of several compounds in the Institute's scope that are not indexed in the listed sources.
The respondent proposes that the named regional databases be added to the standard set.
The secretariat accepts this submission in part. Two of the named sources are added to the standard set. The remainder are added as conditional sources searched where the review question concerns a compound developed or registered in the relevant region.
The standard database set is extended, the conditional sources are named together with the trigger for searching them, and every review states which sources were searched and which were not, with the reason.
The search date is not on the face of the document
The draft carries a publication date and a review date but not the date on which the evidence was last searched. Those are three different dates and only the third tells a reader how current the assessment is. A document published in one quarter may rest on a search run two quarters earlier, and nothing on the page allows that gap to be measured.
The respondent proposes that the search date be printed adjacent to every certainty rating rather than in the methods section, on the ground that a reader who acts on a rating is unlikely to have read the methods section first.
The secretariat accepts this submission. The distinction between publication, review and search dates is real and the draft did not make it visible where it mattered.
The search date is now printed adjacent to every certainty rating and is carried in the document metadata, so that the interval between the search and the reading is visible without reference to the methods section.
The document should state what a reader ought to do
The draft assesses evidence and stops. The respondent, a practising clinician, states that a reader arriving at the document with a decision to make is left to convert an assessment into an action without help, and proposes that each document close with a recommendation.
The respondent argues that other evidence bodies issue recommendations and that declining to do so transfers the difficult part of the work to the reader.
The secretariat does not accept this submission, and records that the point is a reasonable one rather than a misunderstanding.
The Institute assesses evidence and does not issue recommendations, because a recommendation embeds values and a resource context that the Institute does not hold and cannot state. That constitutional limit is published on the methodology page and is not varied by consultation. The submission remains published in full.
Automated assistance in screening should be disclosed and validated
The respondent asks whether any automated tool is used in screening or data extraction, and states that if one is, its performance should be reported in the same way as a human screener's agreement.
The respondent proposes that automated assistance be prohibited.
The secretariat accepts this submission in part. Disclosure and validation are adopted. A prohibition is not, because the alternative to a validated tool is not a human but a smaller search.
The methodology now requires that any automated assistance in screening or extraction be disclosed in the review, that its output be verified by a human for every included record, and that its measured performance against human screening be reported.
Quantitative claims are reproduced without the method that produced them
Several figures in the draft are quoted from sources that determined them by different methods. A figure obtained by one determination and a figure obtained by another are not comparable, and the draft places them in the same sentence without distinguishing them.
The respondent, an analytical chemist, proposes that every quantitative claim carry the method that produced it at the point of use rather than in the reference.
The secretariat accepts this submission. Placing two figures side by side is an implicit claim that they are the same kind of quantity, and in the cases identified they were not.
Every quantitative claim now carries the determination that produced it at the point of use, and figures obtained by non-comparable methods are no longer presented in the same row or sentence.
Registry records without posted results are neither screened nor counted
The respondent states that the screening procedure operates on bibliographic records and that a registered trial without a publication falls outside it entirely.
The respondent proposes that registry records be screened as records in their own right and reported as a separate category.
The secretariat accepts this submission. A procedure that cannot see an unreported trial cannot measure how much of the evidence base is unreported.
Registry records are now screened as records in their own right, and those meeting the eligibility criteria without a posted result are reported as unreported studies with their registered outcome measures and their completion dates.
The document should describe how an assessment becomes a decision
The respondent, employed by a health-technology assessment body, states that the methodology stops at the certainty rating and that the step from a rating to a decision is where most disagreement actually occurs.
The respondent proposes an evidence-to-decision framework.
The secretariat notes this submission and records the point as correct about the field and not about this document.
No amendment arises. The Institute assesses evidence and does not make decisions, because a decision requires values and a resource context the Institute does not hold. The boundary is stated on the methodology page and the submission has prompted it to be stated at the head of the document rather than in the section on scope.
The document set should be published in translation
The respondent notes that the assessments concern compounds supplied internationally and that publishing only in English restricts access to the assessment to readers who work in it.
The respondent proposes machine translation of the document set as an interim measure, with human review of the certainty language.
The secretariat does not accept this submission, and records that the underlying point is sound and that the proposed remedy is the difficulty.
A translation whose certainty language has drifted is a different assessment carrying the Institute's name, and the Institute cannot review translations it does not have the capacity to review. The documents remain in English. The submission is published in full because the access problem it identifies is real and unresolved.
The treatment of sponsor-conducted evidence is unworkable where all the evidence is sponsor-conducted
The respondent states that the draft directs assessors to consider sponsor conduct as a risk-of-bias consideration, and that for several compounds every contributing trial shares one sponsor, so the direction produces a uniform downgrade that distinguishes nothing.
The respondent proposes that sponsor concentration be reported as a property of the evidence base rather than used as a downgrade.
The secretariat accepts this submission in part. Sponsor concentration becomes a reported property of the evidence base. It continues to inform the risk-of-bias assessment where a specific feature of a trial, rather than the identity of its sponsor, supports a judgement.
Sponsor concentration is now reported with every certainty rating as the number of independent sponsors contributing evidence, and a downgrade may not be made on sponsor identity alone but must name the feature of the trial on which it rests.
Declared interests should appear on the document rather than on a separate page
The draft links to a central conflicts register. The respondent argues that a reader assessing whether to rely on a particular document should not have to leave it to find out who assessed it and what they declared.
The respondent proposes that the interests of every named contributor to a document be printed on that document.
The secretariat notes this submission and records that the draft already provides for it, which the respondent could reasonably have missed because the provision sits in an appendix.
Every document carries the declared interests of its named contributors in its front matter, and the central register exists so that a reader can see a person across all documents rather than one at a time. No amendment arises; the provision has been moved from the appendix into the body of the methodology document so that it is findable.
A superseded version should remain reachable from the version that replaced it
The respondent states that the draft supersedes an earlier document and that a reader who cited the earlier version has no way to reach it from the new one, which makes it impossible to see what changed.
The respondent asks that every version carry a link both to what it supersedes and to what supersedes it.
The secretariat notes this submission. The corrections and versioning policy already requires bidirectional version links and every superseded document is retained at its own address.
No amendment arises. The requirement is stated in the corrections and versioning policy and the amendment log of this document links to the version it replaced. The respondent is correct that the link was absent from the draft page furnished for consultation, which was a defect of the consultation copy and not of the policy.
Indirectness is defined so broadly that any evidence could be downgraded under it
The respondent states that the definition covers differences in population, intervention, comparator, outcome and setting without any threshold, so that an assessor who wishes to downgrade can always find a ground.
The respondent proposes that the assessment name the specific difference and state why it would be expected to change the effect.
The secretariat accepts this submission. A criterion that can always be satisfied is not a criterion.
An indirectness downgrade now requires the assessor to name the specific difference and to state the mechanism by which it would be expected to change the effect, and the statement is published with the rating so that it can be disputed.
The rating conflates absence of evidence with conflicting evidence
The framework applies a single very low certainty rating to two situations a reader will act on differently. In the first, several studies exist and disagree. In the second, no study exists at all. The same badge in both places gives the same signal for two different states of knowledge.
The respondent proposes that the rating be accompanied by a standing formulation identifying which of the two applies, identical wherever it appears so that it can be recognised at a glance.
The secretariat accepts this submission in full. The distinction is real, it is decision-relevant, and the draft did not make it.
A standing formulation has been adopted and is applied to every low and very low certainty rating in the document set. The rating scale itself is unchanged, so that Institute ratings remain readable against published assessments using the same scale.
The conditions for upgrading observational evidence are too permissive
The respondent states that the draft permits an upgrade for a large effect without requiring that confounding of the magnitude needed to produce it be shown to be implausible.
The respondent proposes that upgrading be removed from the framework entirely.
The secretariat accepts this submission in part. The conditions are tightened. Upgrading is retained, because a framework that cannot recognise a strong observational signal will misrate the cases where randomisation is not available.
An upgrade for a large effect now requires an explicit statement of the confounding structure that would be needed to produce the observed effect and a reason for regarding it as implausible, and the statement is published with the rating.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.