Measurement quality

Measurement quality determines whether an observed score represents what a study intends to assess. Interpreting a measurement requires its precision, reliability, accuracy and validity, the meaning of its scale and the unit being counted.

Precision, reliability and validity

A precise measurement has little relative error or dispersion and returns consistent findings under the same or similar conditions. ¹ In an unbiased system, more observations can increase precision. ¹

Repeatability means similar results when the same participant is assessed with the same measure by the same rater under identical or very similar conditions, within a short period in which the measured variable is unlikely to change. ¹ Reproducibility means little dispersion when the same subject is assessed under changing conditions. ¹

Reliability is the consistency of an assessment method, including intrarater, inter-rater and test–retest reliability. ^{1, 2} Accuracy is closeness to a true or reference value. ¹ Reliability does not establish accuracy: a speedometer that consistently shows 50 miles per hour at an actual speed of 60 miles per hour is reliable but inaccurate. ² Validity concerns whether a measurement assesses what it purports to assess. ¹

Measurement reporting

The Methods section should describe assessments so that readers can understand and ideally replicate the investigation. ^{3, 4} Relevant psychometric properties and the populations in which an instrument has been validated should be reported. ³

Descriptions of assessments should clarify:

1. Whether an increase or decrease indicates improved or worsened function.
2. Whether normative values correspond to no disease or impairment.
3. Whether threshold values represent minimum clinically important differences.

⁴

For patient-reported outcomes and many other assessments, the meaning of positive or negative values also needs to be explained. ³ A survey report should state how it was fielded, whether the instrument was validated or translated, including back-translation where used, its response rate and whether nonrespondents differed from respondents. ⁵

Comparability after a change

Pandemic-related methodological advice in March 2021 called for outcome validation when clinic vital-sign measurements were replaced by measurements at home using another device. ^{6, 7} The issue was whether the measurements made by the two devices were comparable. ⁷

The unit of observation

A report should make its unit of observation clear: an edited manuscript described 25 eyes without saying how many patients they represented. ^{8, 9} The protocol specifies the primary outcome of interest and its unit of observation, but reports may still fail to say how patients responded. ^{8, 9}

Scale meaning and response options

Linearity and thresholds

A difference of a given size may have different meanings at different positions on a nonlinear scale. ^{10, 11, 8} A ten-point pain scale may be nonlinear: a reduction from 9 to 8 may be greater than a reduction from 4 to 3. ^{10, 11} Writers need not know in advance whether a scale is linear, but should ask the authors. ^{10, 11}

The Beck Depression Inventory, which ranges from 0 to 63, illustrates three questions about interpreting a score difference: ¹⁰

1. Does a three-point difference at one end of the scale mean the same as a three-point difference at the other?
2. What is the smallest clinically meaningful difference?
3. Are there threshold scores with different meanings?

¹⁰

Where a change falls on the scale and whether it crosses a severity threshold affect its interpretation. ^{11, 8} Mean baseline values are useful alongside post-treatment means: when baseline means are near normal, even an effective treatment may have little room to reduce them. ^{10, 11}

Question wording and response behaviour

Unequal psychological distances between response categories can make parametric statistics inappropriate. ¹² For example, in “terrible, horrible, awful, fair, slightly good, all right, reasonably good”, the distance between “awful” and “fair” differs from that between “fair” and “slightly good”. ¹²

Avoiding extreme responses creates central-tendency bias, while agreeing to satisfy the investigator creates acquiescence bias. ¹² Respondents may try to appear strong or impaired, socially acceptable or deliberately less socially acceptable, producing faking-good, faking-bad, social-desirability or norm-defiance biases. ¹² Values that differ across cultures, within or between countries, can also bias responses. ¹²

Respondents may prefer positive to negative options, so weaker negative modifiers such as “not quite” may be preferable to stronger ones such as “totally”. ¹² “How concerned are you about…” presupposes concern and may overstate it; a preliminary “Are you concerned…” question can avoid that assumption. ¹² Omitting “undecided” or “no opinion” can overestimate the proportion holding an opinion and draw undecided respondents towards middle categories. ¹²

An unbalanced range such as “good, average, poor, very poor, awful” may elicit more negative answers than “excellent, very good, good, average, poor”. ¹³ Respondents may favour options on the left; it may be necessary to repeat a scale with “strongly agree” and “strongly disagree” at opposite ends and then reverse their positions. ¹³ Direction-of-comparison bias favours the entity used to define the comparison: asking whether golf is more exciting than tennis may favour golf, with the pattern reversing when the names are reversed. ¹³

Designing a cross-country questionnaire

The Global Diabetes Survey’s 2012 proposal illustrates questionnaire development for comparing national care. ¹⁴ Its objectives were:

1. Assess national diabetes-care quality annually in each participating country.
2. Identify gaps and barriers, combining comparisons within and between countries to inform planning, management and organisation of national diabetes plans.
3. Analyse annual changes in care quality using follow-up survey data.

¹⁵

The design required:

1. Standardised questions adequately representing care quality in different countries.
2. Answers from stakeholders representing all relevant areas and focus groups in diabetes care.

¹⁶

The proposed development sequence was to review existing diabetes guidelines and national plans, weigh the relevance of identified items, select questions, translate them and develop an online format. ¹⁷ The questionnaire was intended to assess structure, process and outcome quality and to be answerable by different stakeholders; registered volunteers were conducting a Delphi-like optimisation. ¹⁷

The proposal sought representative recruitment from diabetes-related stakeholder groups. ¹⁸ Named groups included:

• Patients or patient organisations.
• General practitioners.
• Diabetes specialists.
• Diabetes educators.
• Diabetes nurses.
• Other diabetes professions.
• Scientists and scientific organisations.
• Diabetes prevention experts.
• Politicians and policy makers.
• Health insurance and payer representatives.
• Industry participants.
• Pharmacists.
• Indirectly affected persons.
• Other interest groups.

^{14, 17}

The recruitment target was one person per group per 250,000 patients with diabetes in each country. ¹⁷

Reliability and agreement study reports

The 2011 Guidelines for Reporting Reliability and Agreement Studies (GRRAS) recommend the following: ¹⁹

1. Identify in the title or abstract that inter-rater or intrarater reliability or agreement was investigated.
2. Explicitly name and describe the diagnostic or measurement device of interest.
3. Specify the subject population of interest.
4. Specify the rater population of interest, where applicable.
5. Describe existing knowledge of reliability and agreement and give the study rationale, where applicable.
6. Explain the sample-size choice and state planned numbers of raters, subjects or objects, and replicate observations.
7. Describe the sampling method.
8. Describe the measurement or rating process, including the interval between measurements, available clinical information and blinding.
9. State whether measurements or ratings were independent.
10. Describe the statistical analysis.
11. State actual numbers of included raters and subjects or objects and completed replicate observations.
12. Describe rater and subject characteristics, such as training and experience.
13. Report reliability and agreement estimates with measures of statistical uncertainty.
14. Discuss the results’ practical relevance.
15. Provide detailed results if possible, for example online.

¹⁹