Glossary category

Evidence assessment

Terms describing how the Institute judges and reports the strength of evidence.

Terms

29 documents

Table 1. Terms in the evidence assessment category.

Term	Definition
Absence of evidence	A state in which no study adequate to answer a question has been identified. The Institute records an absence of evidence as distinct from evidence of absence: the first says nothing about whether an effect exists, the second is a finding that it does not.
Amendment log	The record attached to every Institute synthesis listing each change made after protocol registration, when it was made, and why. A change that is not in the log did not happen.
Certainty of evidenceQuality of evidence · GRADE rating	The Institute’s judgement of the extent to which an effect estimate is adequate to support a conclusion. Reported at one of four levels: high, moderate, low or very low. A certainty rating describes confidence in an estimate and is never a recommendation, an endorsement or a statement that a compound should or should not be used.
Downgrading	The reduction of a certainty rating from its starting level because of a concern in one of five domains: risk of bias, inconsistency, indirectness, imprecision and publication bias. Each downgrade is recorded against its domain with a stated reason.
Evidence of absence	A finding, from a study adequately powered to detect an effect of the size of interest, that no such effect was observed. A negative result from an adequately powered trial is evidence of absence; a negative result from an underpowered one is not.
Evidence synthesisSystematic review	A document answering a structured question by identifying, appraising and combining the studies that bear on it, according to a protocol registered before the search was executed.
High certainty	The Institute is confident that the true effect lies close to the estimate. Further research is very unlikely to change confidence in the estimate. A high rating is not a claim of proof and does not extend beyond the population, comparator and outcome that produced the estimate.
Imprecision	A confidence interval wide enough that its ends would support different decisions, or an event count too small to constrain the estimate. Assessed against a decision threshold rather than against statistical significance.
InconsistencyHeterogeneity	Unexplained variation in effect estimates across contributing studies, beyond what chance would produce. Consistency of direction with variation in magnitude is treated differently from variation in direction.
Indirect comparison	An estimate of the difference between two interventions that have not been compared directly, derived through their comparisons with a common comparator. Weaker than a direct comparison and dependent on transitivity.
Indirectness	A mismatch between the population, intervention, comparator or outcome of the contributing studies and those of the question being answered. Also applied where an outcome that would answer the question was not measured at all.
Interaction testTest for subgroup differences	A formal test of whether a treatment effect differs across subgroups. A difference in statistical significance between two subgroups is not an interaction test and the Institute does not report one as though it were.
Low certainty	Confidence in the effect estimate is limited and the true effect may be substantially different. The Institute states in each case whether the limitation arises from conflicting evidence or from thin evidence, because the two carry different implications.
Moderate certainty	The true effect is likely to be close to the estimate, but there is a real possibility that it is substantially different. Further research is likely to have an important impact on confidence in the estimate.
Negative result	A result in which the observed effect does not differ from the comparator by more than chance would produce. Informative where the study was adequately powered for the effect of interest and uninformative where it was not, and the Institute states which applies.
Network meta-analysisMixed treatment comparison	A synthesis combining direct and indirect comparisons across a connected set of interventions. The Institute publishes network estimates with its transitivity assessment attached and does not publish ranking probabilities.
Overview of reviewsUmbrella review	A review whose unit of inclusion is a published systematic review rather than a primary study. Overlap between the included reviews is assessed and reported, because a primary study counted in several reviews is not several pieces of evidence.
PICO	The four elements that define a review question: population, intervention, comparator and outcome. The Institute registers a PICO frame before searching and records any subsequent change as a post-hoc amendment.
Protocol registration	The recording of a review question, eligibility criteria and analysis plan before the search is executed. Registration is what makes a subsequent change identifiable as post hoc rather than planned.
Publication biasSmall-study effects	The risk that the studies available to a review are not a representative sample of the studies conducted, most often because studies with unfavourable results were not published. Suspected where the evidence base is small, recent and wholly sponsor-generated.
Risk of biasStudy limitations	The extent to which features of a study’s design or conduct could produce an effect estimate that systematically departs from the truth. Assessed at the level of the individual study and carried into the certainty rating for the outcome.
Scoping review	A structured map of the evidence available for a question, published where the Institute judges that a quantitative synthesis is not yet supportable. A scoping review states what would have to exist before an intervention review could be attempted.
Selective outcome reporting	The reporting of a subset of measured outcomes chosen after the results were known. Detected by comparing the published report with the registered protocol, which the Institute does for every included study where a protocol is retrievable.
Sensitivity analysis	A repetition of an analysis under a different defensible assumption, conducted to establish whether a conclusion depends on that assumption. Reported whether or not the conclusion changes.
Subgroup analysis	An analysis restricted to a subset of participants. Hypothesis-generating unless the subgroup and the interaction test were pre-specified, and the Institute records which of the two applies in every case.
Summary of findingsSoF table	A structured table presenting, for each pre-specified outcome, the number of contributing participants and studies, the absolute and relative effect, the certainty rating and the reason for any downgrade. The Institute treats it as the principal output of a synthesis.
Transitivity	The assumption, required for indirect comparison, that participants in the contributing trials could in principle have been randomised to any of the compared interventions. Assessed explicitly and reported before any network estimate is presented.
Upgrading	The increase of a certainty rating above its starting level, applicable in principle to observational evidence showing a large effect, a dose–response gradient or a plausible confounder that would reduce the observed effect. The Institute has applied an upgrade rarely and records the reasoning in full each time.
Very low certainty	The Institute has very little confidence in the effect estimate. In this document set a very low rating most often reflects an absence of controlled human evidence rather than the presence of conflicting evidence, and every entry says which applies.