Qualitative study reports
The report of a qualitative study shows its reader the path from the research question to the interpretation: why the question suits a qualitative method, who collected the material, from whom and how, how the analysis was developed and checked, and which story the report tells.
COREQ and RATS
COREQ, the consolidated criteria for reporting qualitative research, is a 32-item checklist for interviews and focus groups, published by Tong, Sainsbury and Craig in 2007. RATS is a set of review guidelines for qualitative research, named for relevance, appropriateness, transparency and soundness, from J. P. Clark's chapter on how to peer review a qualitative manuscript in the second edition of Peer Review in Health Sciences.
Research question and choice of method
Under RATS, the research question should be stated explicitly, justified and linked to the existing knowledge base of empirical research, theory and policy, and RATS asks whether it is interesting and relevant to clinical practice, public health or policy. The study design should be described and justified, including why a particular method was chosen, and RATS asks whether qualitative methodology is the best approach for the study aims. RATS matches each method to what it suits: interviews to experience, perceptions, behavior, practice and process; focus groups to group dynamics, convenience and nonsensitive topics; ethnography to culture, organizational behavior and interaction; and textual analysis to documents, art, representations and conversations.
COREQ asks what methodological orientation was stated to underpin the study, such as grounded theory, discourse analysis, ethnography, phenomenology or content analysis.
Research team and reflexivity
COREQ's items on the research team and reflexivity ask which authors conducted the interviews or focus groups, and what their credentials were, such as PhD or MD, their occupation at the time of the study, their gender, and their experience and training. They also ask whether a relationship with participants was established before the study began, what participants knew about the researcher, such as personal goals and reasons for doing the research, and what characteristics were reported about the interviewer or facilitator, such as bias, assumptions, reasons and interests in the research topic.
RATS asks whether the researchers are appropriate and how they might bias the conduct of the study and its results, for good and bad. In the manuscript, RATS looks for whether the researchers occupy dual roles as clinician and researcher, whether the ethics of this are discussed, and whether the researchers critically examine their own influence on the formulation of the research question, data collection and interpretation.
Participants and data collection
COREQ asks how participants were selected, such as purposively, by convenience, consecutively or by snowball, how they were approached, such as face to face, by telephone, by mail or by email, how many took part, and how many refused or dropped out and why. It asks where the data were collected, such as at home, in a clinic or at a workplace, whether anyone besides participants and researchers was present, and what the important characteristics of the sample were, such as demographic data and date. It also asks whether the authors provided questions, prompts or guides and pilot tested them, whether repeat interviews were carried out and how many, whether audio or visual recording was used, whether field notes were made during or after the interview or focus group, how long it lasted, and whether data saturation was discussed.
RATS asks whether the participants selected are the most appropriate to give access to the kind of knowledge the study seeks and whether the sampling strategy is appropriate. The criteria for selecting the sample should be justified and explained: theoretical sampling is based on preconceived or emergent theory, purposive sampling seeks diversity of opinion, and volunteer sampling serves feasibility and hard-to-reach groups. RATS asks whether recruitment was conducted using appropriate methods and whether there could be selection bias, and the manuscript should give details of how recruitment was conducted and by whom, and of who chose not to participate and why. Data collection should be systematic and comprehensive, with the methods outlined and examples given, such as interview questions, and the study group and setting clearly described. Why and when data collection stopped should be justified and described, and RATS asks whether stopping then was reasonable.
Ethics
Under RATS, the manuscript should detail the informed-consent process explicitly and clearly, discuss anonymity and confidentiality, and cite ethics approval.
Analysis and interpretation
COREQ asks whether transcripts were returned to participants for comment or correction, how many coders coded the data, whether the authors described the coding tree, whether themes were identified in advance or derived from the data, what software, if any, managed the data, and whether participants gave feedback on the findings. It asks whether participant quotations were presented to illustrate the themes and findings, each identified, for example by participant number, whether the data presented and the findings were consistent, whether major themes were clearly presented, and whether diverse cases or minor themes were described.
RATS asks whether the type of analysis suits the type of study: thematic analysis is exploratory, descriptive and hypothesis generating, framework analysis is given policy as an example, and constant comparison or grounded theory is theory generating and analytical. The analytic approach should be described in depth and justified. Indicators of its quality are a description of how themes were derived from the data, inductively or deductively, evidence that alternative explanations were sought, and analysis and presentation of negative or deviant cases.
RATS asks whether the interpretations are clearly presented and adequately supported by the evidence, and whether quotations are used and are appropriate and effective. The manuscript should include a description of the basis on which quotations were chosen, semiquantification when appropriate, and richly detailed illumination of context, meaning or both. The method of checking the trustworthiness and reliability of the data and interpretations should be described and justified, such as an audit trail, triangulation or member checking, along with whether an independent analyst reviewed the data and contested the themes and how disagreements were resolved.
Discussion and presentation
Under RATS, findings should be presented with reference to the existing theoretical and empirical literature, showing how they contribute to it, and strengths and limitations should be explicitly described and discussed. RATS asks whether the manuscript is well written and accessible, and looks for evidence of following guidelines on format and word count, detail of methods or additional quotations in an appendix, and writing for a health sciences audience.
The story the report tells
Writing well is essential, especially for qualitative researchers, yet their writing may offend with turgid prose and seemingly endless lists of unlinked codes and categories.
Writing up qualitative research poses two early challenges. The first is how to tell the story, by identifying the style most suitable to the research, its purpose and its audience. Unlike the IMRAD format that dominates medicine and the health sciences, one size does not fit all in qualitative research writing. The second is which of the many stories a data set supports to tell, by determining its central point or story line and the argument to be made. The writer has to move from retelling participants' stories by summarising the data to transforming the data through analysis and interpretation.
Data are there to think with and about, but the generation of ideas can never depend on data alone. Data are instead used selectively to exemplify and illustrate the story being told. A good story includes an argument that runs through the text like a red thread and holds it together, and it involves the reader along the way and results in a compelling message. Good stories come in many forms, but they all have a central story line and aim to engage the reader.
Red flags
RATS lists red flags: common features of ill-conceived or poorly executed qualitative studies that are a cause for concern and must be viewed critically. They might be fatal flaws, or they may result from a lack of detail or clarity.
- Grounded theory: not a simple content analysis but a complex, sociological, theory-generating approach.
- Jargon: descriptions that are trite, pat or jargon filled should be viewed skeptically.
- Overinterpretation: interpretation must be grounded in accounts and semiquantified if possible or appropriate.
- Seems anecdotal or self-evident: may be a superficial analysis, not rooted in a conceptual framework or linked to previous knowledge, and lacking depth.
- Consent process thinly discussed: may not have met ethics requirements.
- Doctor-researcher: consider the ethical implications for patients and the bias in data collection and interpretation.