Draft indication set and anchor outcomes — submissions
The 15 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
15 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
Quantitative claims are reproduced without the method that produced them
Several figures in the draft are quoted from sources that determined them by different methods. A figure obtained by one determination and a figure obtained by another are not comparable, and the draft places them in the same sentence without distinguishing them.
The respondent, an analytical chemist, proposes that every quantitative claim carry the method that produced it at the point of use rather than in the reference.
The secretariat accepts this submission. Placing two figures side by side is an implicit claim that they are the same kind of quantity, and in the cases identified they were not.
Every quantitative claim now carries the determination that produced it at the point of use, and figures obtained by non-comparable methods are no longer presented in the same row or sentence.
A fixed section structure will force empty sections
The respondent states that a common structure across the series will produce documents with sections that have nothing to report, and that an empty section is read as an oversight.
The respondent proposes that sections be omitted where they do not apply.
The secretariat accepts this submission in part. Sections are retained and their emptiness is made explicit, because an omitted section is indistinguishable from a section that was never considered.
A section with nothing to report now states that fact and the reason in a fixed form, rather than being omitted or left blank, so that a reader can tell the difference between an absence of evidence and an absence of effort.
The default ordering of a comparison table carries an implication
The respondent states that whatever column a table is sorted on by default will be read as the column that matters, and that the draft sorts on a score.
The respondent proposes an alphabetical default.
The secretariat accepts this submission. A default is a claim whether or not it is intended as one.
The default ordering is now alphabetical, a standing statement above every comparison table records that the ordering is not a ranking, and the sort state is reset on reload so that a shared link does not carry an ordering the sender chose.
A printed copy loses the information that makes it checkable
The respondent states that printing a document produces pages without the identifier, the version or the retrieval date, so that a printed copy cannot be tied back to what it was printed from.
The respondent proposes that these be printed in a fixed position on every page.
The secretariat accepts this submission. A printed assessment circulates for longer than the browser tab it came from.
The print stylesheet now carries the identifier, version, publication date and retrieval date on every page, expands link destinations into the text, and prints the full reference list rather than a link to it.
The document should not describe uses outside the approved indication
The submission is made on behalf of a marketing-authorisation holder. It states that the draft describes uses of the compound that fall outside the approved indication, that such uses are not supported by the sponsor, and that describing them may be read as legitimising them.
The sponsor asks that the sections concerned be removed, or in the alternative that they carry a prominent statement that the sponsor does not support such use.
The secretariat does not accept this submission. The Institute records what is supplied and what is used, because a reader encountering a compound outside a regulated route is the reader most in need of an assessment of the evidence for it.
The sections remain. Every such section already states the regulatory status of the use described and states that the evidence for it is assessed separately from the evidence for the approved indication. The submission is published in full and identified as an industry submission.
A field the Institute could not verify should be shown rather than omitted
The respondent states that omitting an unverifiable field leaves a reader unable to distinguish between a fact that does not exist, a fact the Institute did not seek and a fact it sought and could not confirm.
The respondent proposes that unverified fields be displayed with their status.
The secretariat accepts this submission. Three different situations were being rendered identically as blank space.
Every field now carries one of four states, being verified, claimed and unverified, sought and not established, or not applicable, and the state is displayed rather than inferred from an empty cell.
Two factual descriptions of the sponsor's programme are inaccurate
The submission is made on behalf of the marketing-authorisation holder and is confined to two matters of fact. The draft describes a trial as terminated where the sponsor closed it at a pre-specified interim analysis, and gives a dose in a unit that does not match the approved labelling.
Supporting documentation, comprising the published trial report and the current summary of product characteristics, accompanied the submission. No view is expressed on the certainty ratings, which the sponsor considers a matter for the assessment committee.
The secretariat accepts this submission. Both points are matters of fact, both were checkable against documents the Institute holds, and both were wrong in the draft.
The trial status now reads as completed at a pre-specified interim analysis, with the analysis identified, and the dose is stated in the unit used in the approved labelling. The submission is identified as an industry submission on its face in accordance with the conflicts policy.
A superseded version should remain reachable from the version that replaced it
The respondent states that the draft supersedes an earlier document and that a reader who cited the earlier version has no way to reach it from the new one, which makes it impossible to see what changed.
The respondent asks that every version carry a link both to what it supersedes and to what supersedes it.
The secretariat notes this submission. The corrections and versioning policy already requires bidirectional version links and every superseded document is retained at its own address.
No amendment arises. The requirement is stated in the corrections and versioning policy and the amendment log of this document links to the version it replaced. The respondent is correct that the link was absent from the draft page furnished for consultation, which was a defect of the consultation copy and not of the policy.
The series scope is drawn widely enough to absorb documents that belong elsewhere
The respondent states that the scope as drafted would admit documents that are neither assessments nor standards, and that a series which admits everything ceases to tell a reader anything about what a document is.
The respondent proposes a narrower scope with an exclusion list.
The secretariat accepts this submission in part. An exclusion list is added. The scope itself is not narrowed further, because the excluded classes are better handled by naming them than by tightening a boundary that would then be argued over.
The series scope now carries an explicit exclusion list naming the document classes that do not belong to it and stating where each belongs instead, so that a placement decision is checkable rather than editorial.
Point estimates are given without an interval
Several estimates in the draft appear as single figures. The respondent states that a point estimate without an interval invites a precision the underlying data do not support, and that the effect is worst where the estimate is drawn from a small contributing set.
The respondent proposes that no point estimate appear anywhere in the document set without its interval, including in summary tables and in the abstract.
The secretariat accepts this submission in part. Intervals are added wherever the source reports one. The proposal is declined for figures the source published without an interval, because the Institute will not compute an interval a source did not report.
Every estimate now carries its interval where the source reported one, and where it did not, the estimate is annotated as reported without an interval rather than left to appear as a precise figure.
A withdrawn document should not simply disappear
The respondent states that a reader arriving from an external citation at a withdrawn document needs to know that it existed, that it was withdrawn and why, and that a redirect to an index gives none of that.
The respondent proposes a notice at the original address.
The secretariat accepts this submission. A silent removal makes the citation that led there unresolvable.
A withdrawn document is now replaced at its own address by a notice recording its identifier, its title, the date and ground of withdrawal, and a link to whatever supersedes it, and the withdrawal is recorded in the corrections log.
The same concept is given three different names in one document
The draft refers to the same quantity as a response rate, a responder proportion and a categorical outcome in different sections. The respondent, who works in health-technology assessment, states that a reader cannot tell whether the three refer to one thing or to three.
The respondent proposes that the glossary term be used at every occurrence and that the glossary entry be linked at first use in each section rather than only at first use in the document.
The secretariat accepts this submission. The variation was stylistic and its cost to the reader exceeds any benefit.
A single term is now used throughout for each defined concept, and the glossary entry is linked at first use within each section rather than once per document.
Declared interests should appear on the document rather than on a separate page
The draft links to a central conflicts register. The respondent argues that a reader assessing whether to rely on a particular document should not have to leave it to find out who assessed it and what they declared.
The respondent proposes that the interests of every named contributor to a document be printed on that document.
The secretariat notes this submission and records that the draft already provides for it, which the respondent could reasonably have missed because the provision sits in an appendix.
Every document carries the declared interests of its named contributors in its front matter, and the central register exists so that a reader can see a person across all documents rather than one at a time. No amendment arises; the provision has been moved from the appendix into the body of the methodology document so that it is findable.
A sortable table implies a comparison the underlying data do not support
The draft presents a sortable table whose columns are drawn from sources of differing quality. The respondent states that sorting on such a column produces an ordering that looks like a ranking and is not one.
The respondent proposes that sorting be disabled on any column whose values are not commensurable.
The secretariat notes this submission and records that the point is correct in principle.
No amendment arises here because every sortable table in the document set already carries a standing statement above it that the ordering is not a ranking and that the values in each column are commensurable only where the column header says so. The proposal to disable sorting was considered and not adopted, because a reader who cannot sort a table generally sorts it elsewhere and without the statement.
The document is unreadable without specialist training
The respondent, a trustee of a patient organisation, states that the draft is written for a reader who already understands certainty grading, and that the people most affected by the subject matter will not reach the assessment at all.
The respondent proposes a plain-language summary at the head of every document, written to the same standard of accuracy as the document itself and not as a promotional abstract.
The secretariat accepts this submission in part. A plain-language summary is added. The proposal that it replace the technical abstract is declined, because the abstract is the part of the document other assessors read and cite.
Every document now opens with a plain-language summary of not more than 150 words, placed above the technical abstract and carrying the same certainty language, so that the two cannot diverge.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.