Draft standard CEI-MS-04: Determination of water content in… — submissions
The 16 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
16 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
Absence of evidence is presented in a form a reader will take as negative evidence
Where the Institute has identified no study, the draft states that no evidence was found. In several places that sentence sits immediately after a paragraph describing an adverse outcome, and the juxtaposition invites the reading that the compound was studied and found wanting.
The respondent proposes a standing formulation, used identically wherever the situation arises, distinguishing an outcome that was studied and not demonstrated from an outcome that has not been studied at all.
The secretariat accepts this submission. The two states are different, they support different decisions, and the draft rendered them in language a reader could not reliably separate.
A standing formulation has been adopted and is applied wherever an evidence gap is recorded, distinguishing an outcome assessed and not demonstrated from an outcome not assessed. The formulation is identical at every occurrence so that it can be recognised at a glance.
References should carry a persistent identifier for every cited source
Several references in the draft carry a journal, a year and a volume but no persistent identifier. The respondent, who works in a library setting, states that retrieval of such a reference is materially slower and that identifiers should be supplied throughout.
The respondent asks in the alternative that where an identifier exists but is not carried, the omission be explained rather than left as a gap the reader must interpret.
The secretariat accepts the second limb of this submission and declines the first. Identifiers are supplied wherever the Institute holds one. Where the Institute does not hold an identifier it will not supply one, because a reconstructed identifier that resolves to the wrong record is a worse defect than an absent one.
Every reference without a persistent identifier now carries an explicit statement that the identifier is not held by the Institute, so that its absence is a recorded fact rather than an apparent oversight.
A system suitability criterion is unachievable on instrumentation in general use
The respondent, employed by a contract analytical laboratory, states that one of the suitability criteria in the draft can be met only on current-generation instrumentation, and that laboratories operating validated older systems would be unable to report a conformant determination.
Data from three instruments of different generations, showing the criterion met on one and missed on two, accompanied the submission.
The secretariat accepts this submission in part. The criterion is relaxed to the level the supplied data support for the general case, and retained at the stricter level for the specific discrimination that motivated it.
The suitability criterion is now stated at two levels, a general level achievable on instrumentation in common use and a stricter level required only where the standard is applied to the discrimination named in the scope, with the reason for the difference stated.
A determination does not record the version of the standard it was made under
The respondent states that the standard will be revised and that results reported under different versions will circulate together, indistinguishable from one another.
The respondent proposes that the standard identifier and version be carried on the face of every determination.
The secretariat accepts this submission. A conformance claim against an unspecified version is not a conformance claim.
Every determination reported under this standard now carries the standard identifier and version on its face, and a claim of conformance that does not identify the version is recorded by the Institute as not conformant.
The document is unreadable without specialist training
The respondent, a trustee of a patient organisation, states that the draft is written for a reader who already understands certainty grading, and that the people most affected by the subject matter will not reach the assessment at all.
The respondent proposes a plain-language summary at the head of every document, written to the same standard of accuracy as the document itself and not as a promotional abstract.
The secretariat accepts this submission in part. A plain-language summary is added. The proposal that it replace the technical abstract is declined, because the abstract is the part of the document other assessors read and cite.
Every document now opens with a plain-language summary of not more than 150 words, placed above the technical abstract and carrying the same certainty language, so that the two cannot diverge.
Results are reported without an estimate of measurement uncertainty
The respondent, an analytical chemist, states that a figure reported to one decimal place implies an uncertainty the standard nowhere estimates, and that two determinations differing within the uncertainty will be read as a difference.
The respondent proposes that an uncertainty budget be required for every determination.
The secretariat accepts this submission in part. A repeatability figure and a stated reporting precision are required. A full uncertainty budget is not, because the standard is written to be usable by laboratories that do not maintain one and the requirement would exclude them without improving the reading.
Every determination now reports the repeatability observed in the run, the reporting precision, and a statement that two results differing by less than the stated repeatability are not distinguishable by this method.
The same concept is given three different names in one document
The draft refers to the same quantity as a response rate, a responder proportion and a categorical outcome in different sections. The respondent, who works in health-technology assessment, states that a reader cannot tell whether the three refer to one thing or to three.
The respondent proposes that the glossary term be used at every occurrence and that the glossary entry be linked at first use in each section rather than only at first use in the document.
The secretariat accepts this submission. The variation was stylistic and its cost to the reader exceeds any benefit.
A single term is now used throughout for each defined concept, and the glossary entry is linked at first use within each section rather than once per document.
The standard assumes a reference standard of known assigned content
The respondent states that for many compounds in scope no reference standard with an assigned content exists, and that the standard as drafted has nothing to say to a laboratory in that position.
The respondent proposes that the standard state what can be determined without one and how such a determination should be labelled.
The secretariat accepts this submission. A standard applicable only where a reference standard exists is silent in the cases most in need of it.
The standard now states which determinations remain available without a reference standard of assigned content, requires that such determinations be labelled as relative rather than absolute, and requires the assumption used in place of an assigned content to be stated.
The standard does not say what to do with a result outside the expected range
The respondent states that the draft describes how to obtain a result and stops, and that the handling of an unexpected result is where the integrity of a determination is decided.
The respondent proposes that repetition be permitted only under stated conditions and that every determination performed be reported, including those superseded.
The secretariat accepts this submission. A standard silent on repetition permits the selective reporting it exists to prevent.
The standard now requires that every determination performed on a sample be reported, that a repeat be permitted only where an assignable cause is documented, and that the assignable cause be stated on the face of the report.
A single determination is permitted where duplicates would be inexpensive
The respondent states that the draft permits a result to be reported from one injection, and that the marginal cost of a duplicate is small relative to the value of detecting a preparation error.
The respondent proposes that duplicate determinations be mandatory.
The secretariat accepts this submission in part. Duplicate preparation and injection are required, and a criterion for agreement between them is stated. The proposal to require full duplicate sample preparation on separate days is declined as disproportionate.
The standard now requires duplicate preparation and injection with a stated maximum difference between the two, and requires that a difference exceeding it be investigated before any figure is reported.
The search date is not on the face of the document
The draft carries a publication date and a review date but not the date on which the evidence was last searched. Those are three different dates and only the third tells a reader how current the assessment is. A document published in one quarter may rest on a search run two quarters earlier, and nothing on the page allows that gap to be measured.
The respondent proposes that the search date be printed adjacent to every certainty rating rather than in the methods section, on the ground that a reader who acts on a rating is unlikely to have read the methods section first.
The secretariat accepts this submission. The distinction between publication, review and search dates is real and the draft did not make it visible where it mattered.
The search date is now printed adjacent to every certainty rating and is carried in the document metadata, so that the interval between the search and the reading is visible without reference to the methods section.
The matrix interferes with the determination and no control is required
The respondent states that the matrices in scope can suppress or enhance the response, and that without an inhibition and enhancement control a result cannot be attributed to the analyte.
Data showing recovery outside acceptable limits in an uncontrolled matrix accompanied the submission.
The secretariat accepts this submission. A determination on an interfering matrix without a validated control establishes nothing, and the draft permitted one.
An inhibition and enhancement control is now required for every determination under this standard, with stated recovery limits, and a determination reported without it is recorded as not conformant rather than as a low result.
The standard would be applied more consistently with a worked example
The respondent states that the requirement most often misapplied in practice is the one the draft states most briefly, and that a worked calculation would settle it.
The respondent proposes a worked example showing the same data treated correctly and incorrectly, with both results stated.
The secretariat accepts this submission. An example showing only the correct treatment does not show the reader what they are doing wrong.
A worked example is added showing the same primary data treated two ways, with both resulting figures stated and the difference between them quantified, so that the consequence of the misapplication is visible rather than described.
A field classified as critical is not one a reading turns on
The respondent examines the fields classified as critical and states that two of them are administrative rather than interpretive, so that a certificate could be rendered uninterpretable by an omission that changes no reading.
The respondent proposes that a field be classified critical only where its absence changes what the figure means.
The secretariat accepts this submission in part. One field is reclassified as recommended on the test the respondent proposes. The second is retained as critical because its absence prevents the determination being attributed to a batch.
The classification test is now stated in the standard: a field is critical where its absence changes the meaning of a reported figure or prevents attribution to a batch. One field has been reclassified under that test and the reasoning recorded.
The limitations section states what the method cannot do but not what a reader will wrongly conclude
The limitations section is accurate and abstract. The respondent states that the misreadings this standard should prevent are specific and predictable, and that naming them would be more use than a general statement of scope.
The respondent proposes that the limitations section list the specific inferences a conformant result does not support.
The secretariat accepts this submission. A limitation stated abstractly is a limitation a reader will not apply to the case in front of them.
The limitations section now lists the specific conclusions a conformant determination does not support, each stated as a sentence a reader could otherwise have written, and the list is placed before the references rather than in an annex.
A conformance claim is not defined and will be made loosely
The respondent states that the draft nowhere defines what it means for a determination to conform, and that in the absence of a definition the phrase will be applied to determinations that meet some of the requirements.
The respondent proposes an explicit definition and a requirement that partial conformance be described as such.
The secretariat accepts this submission. An undefined conformance claim is a marketing phrase.
Conformance is now defined as meeting every requirement classified as critical together with the stated system suitability criteria, partial conformance must be described by naming the requirements not met, and a claim of conformance made without the sampling statement is not conformant.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.