Draft synthesis: Completeness of certificates of analysis in… — submissions
The 8 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
8 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
Risk-of-bias judgements are reported as an overall rating without the domains
The respondent notes that the review question — What proportion of certificates of analysis supplied with research peptides contain the fields required to interpret the reported purity figure? — is answerable only if the contributing determinations measured the same quantity, and submits with that in view.
The draft reports a single overall risk-of-bias judgement per study. The respondent states that the overall judgement conceals which domain drove it, and that a reader assessing whether the concern applies to their question needs the domain.
The respondent proposes a domain-level table with the supporting quotation for each judgement.
The secretariat accepts this submission. The overall judgement is a summary of the domains and publishing only the summary makes it uncheckable.
Risk of bias is now reported at domain level for every included study, with the text on which each judgement was based quoted and referenced, and the overall judgement derived from the domains rather than asserted alongside them.
Contributing surveys anonymise their sources to different degrees
The respondent’s comment on the draft review of What proportion of certificates of analysis supplied with research peptides contain the fields required to interpret the reported purity figure? arises from comparing the pooled figures against the reports they were taken from.
Some contributing surveys name the sources they bought from, some describe them by category, and some do not describe them at all.
The respondent states that a synthesis drawing on all three cannot report by source, and that a reader may take the category descriptions as comparable when they were not defined in common.
The respondent proposes that the level of source description be recorded for each survey and that no analysis be reported by source.
The secretariat notes this submission and records that the practice the respondent describes is a property of the contributing literature rather than of the draft.
No analysis by source is reported. The level of source description is now recorded survey by survey, and the review states that it draws no conclusion about any individual source and that its findings are properties of the sampled literature.
What the contributing methods could not have detected is not stated
The respondent read the draft review of What proportion of certificates of analysis supplied with research peptides contain the fields required to interpret the reported purity figure? and has confined this submission to a single matter of measurement.
The draft reports what the contributing surveys found and does not report what their methods were incapable of finding.
The respondent states that a reverse-phase method is blind to aggregation, that an ultraviolet detector does not see a species without a chromophore, and that a reader takes an unqualified negative as an absence.
The respondent proposes that every reported absence carry the detection scope of the method that produced it.
The secretariat accepts the disclosure and declines to reclassify the findings, which were correctly reported for the methods used.
Every reported absence now carries the detection scope of the method behind it, and the review states in its abstract that an absence in this evidence base is an absence within a stated scope rather than an absence in the material.
The document should not describe uses outside the approved indication
The respondent has read the draft synthesis on What proportion of certificates of analysis supplied with research peptides contain the fields required to interpret the reported purity figure? and makes one submission about the analytical evidence it pools.
The submission is made on behalf of a marketing-authorisation holder. It states that the draft describes uses of the compound that fall outside the approved indication, that such uses are not supported by the sponsor, and that describing them may be read as legitimising them.
The sponsor asks that the sections concerned be removed, or in the alternative that they carry a prominent statement that the sponsor does not support such use.
The secretariat does not accept this submission. The Institute records what is supplied and what is used, because a reader encountering a compound outside a regulated route is the reader most in need of an assessment of the evidence for it.
The sections remain. Every such section already states the regulatory status of the use described and states that the evidence for it is assessed separately from the evidence for the approved indication. The submission is published in full and identified as an industry submission.
Whether the analyst knew the source of each sample is not reported
This is a submission on the draft review of What proportion of certificates of analysis supplied with research peptides contain the fields required to interpret the reported purity figure?, made from the standpoint of a laboratory that would have to reproduce the determinations.
The contributing surveys do not state whether the laboratory knew which source each sample came from.
The respondent states that integration decisions and the treatment of borderline results involve judgement, and that knowledge of the source is capable of influencing that judgement without any intention that it should.
The respondent proposes that blinding be recorded where it is reported and requested from contributing authors where it is not.
The respondent’s submission overlaps with submission 001 and was prepared without sight of it.
The secretariat notes this submission and has recorded blinding where the contributing authors reported it, which is a minority of the set.
No amendment to the estimates arises. Blinding is now recorded survey by survey, the absence of the information is stated rather than assumed either way, and the point is carried into the limitations.
Time between manufacture and analysis is not accounted for
The draft review of What proportion of certificates of analysis supplied with research peptides contain the fields required to interpret the reported purity figure? was read against the certificates and reports it draws on.
The contributing surveys analysed material at unrecorded intervals after it was manufactured, and the draft treats every result as describing the material as supplied.
The respondent states that several of the attributes reported change with time and temperature, so an unrecorded interval is an uncontrolled variable running through the whole evidence base.
The respondent proposes that the interval be recorded where it can be established and stated as unknown where it cannot.
This submission is made in the same spirit as submission 005 and on a different aspect of the draft.
The secretariat accepts this submission. An attribute that changes with time cannot be summarised without recording when it was measured.
The interval between the date on the certificate and the date of analysis is now recorded for every result where both dates are held, is stated as not established where they are not, and results with an unknown interval are reported separately from those with a known one.
Whether the contributing determinations were made on calibrated systems is not recorded
The respondent submits on the draft review of What proportion of certificates of analysis supplied with research peptides contain the fields required to interpret the reported purity figure?. This submission concerns how the contributing results were obtained rather than what they show.
The draft reports results without recording whether the systems that produced them were within calibration and whether system suitability was demonstrated at the time.
The respondent states that a determination made on a system that did not meet suitability is not a result, and that the review cannot presently distinguish those from the rest.
The respondent proposes that suitability reporting be recorded for every contributing determination.
The secretariat accepts the recording and declines to exclude results on an absence of reporting, which would confuse an unreported check with a failed one.
Suitability reporting is now recorded for every contributing determination as reported, not reported, or not applicable, and a sensitivity analysis restricted to results with reported suitability is presented beside the main estimate.
The same concept is given three different names in one document
Having read the draft synthesis on What proportion of certificates of analysis supplied with research peptides contain the fields required to interpret the reported purity figure?, the respondent puts one point about the underlying measurements to the committee.
The draft refers to the same quantity as a response rate, a responder proportion and a categorical outcome in different sections. The respondent, who works in health-technology assessment, states that a reader cannot tell whether the three refer to one thing or to three.
The respondent proposes that the glossary term be used at every occurrence and that the glossary entry be linked at first use in each section rather than only at first use in the document.
The secretariat accepts this submission. The variation was stylistic and its cost to the reader exceeds any benefit.
A single term is now used throughout for each defined concept, and the glossary entry is linked at first use within each section rather than once per document.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.