Tell reviewers what to focus on, by role, by document and by stage
Reviewers need to be told what to read a draft for: that review is not quality control, what each functional role reads for, how the reading changes with the document type and the development stage, and what makes a comment specific, actionable and relevant.
What review is and what it is not
Review is a process conducted at different stages of document development to ensure that a document meets its purpose: that it is complete, coherent, and aligned with the overall strategy of the project and with the expectations of its ultimate audience. For a regulatory document that audience is always the health authority, not a reviewer's own manager.
Review differs from quality control in many aspects. It should not be a check of numbers, spelling and company style guide conventions. Those are the medical writer's responsibility throughout all development stages, and the responsibility of quality control specialists and document managers or administrators at later stages, usually as the company's standard operating procedures define. It is often easier for reviewers to check abbreviation lists, fix spelling errors and propose taste-driven changes to text that is already linguistically correct. What medical writers ultimately want from reviewers is constructive, unambiguous and timely strategic input, primarily on the sections relevant to each reviewer's functional role, considering both the document type, by its place and purpose within the electronic common technical document (eCTD), and the development stage. One experienced writer calls these strategic, function-driven, eCTD placement-driven and staged review.
Review relies on clear communication between the medical writer and the reviewers, and it can be a frustrating process for both when communication fails. Misunderstandings between writers and reviewers can have a dire effect on the quality of a document. Review is a two-way street, and both parties are responsible for making it run as smoothly and efficiently as possible. There is ultimately no "us" and "them" in document development, but a common goal for a team to reach within a clearly defined timeframe: fit-for-purpose documents for submission to health authorities, which will ideally result in a successful marketing authorisation.
Why reviews fall short
Several studies have shown that inefficient reviews are not uncommon across the pharmaceutical industry. When reviewers were asked, over several review rounds, to categorise their own comments on a specific document as rhetorical (content-related or strategic), editorial or stylistic, they assessed them as largely rhetorical. When consultants categorised the same comments against more objective criteria than personal opinion, the vast majority were editorial and stylistic, even on advanced drafts. Most of the reasons for such ineffective review are largely unintentional and can be at least partially corrected with appropriate training.
Reviewers are not primarily reviewers. They were trained as experts in their field, such as statistics, bioanalytics or clinical research, not as communicators, and the reviewer hat is only one of the many they wear in a pharmaceutical company. They can often be quite frustrated that the medical writer and the other reviewers do not understand what they want, since their comment was quite clear to them. Some reviewers have no clear understanding of how their review should change with the document type or the development stage, especially in companies that give insufficient training on the eCTD structure during onboarding. A reviewer will often approach a clinical study report, a summary of clinical safety and a clinical overview with the same strategy: reading page by page, focusing largely on abbreviations and spelling, and making fewer and fewer comments by the time the results and conclusions are reached. The expectations and objectives of review are more often than not unclear to reviewers, although medical writers may think otherwise.
The need for a focused review may seem obvious, but one experienced writer has very often seen reviewers lose that focus. Her examples include statisticians improving wording to personal taste, at times changing the meaning of a sentence that was correct; clinical experts checking the list of abbreviations rather than the clinical interpretation of the data; and regulatory affairs managers adding two spaces after every full stop throughout a document while ignoring the company's style conventions.
What to read for, by functional role
In one experienced writer's scheme, each functional role reads for a different part of the document.
| Role | Focus |
|---|---|
| Clinical expert | Medical background and unmet medical need; clinical interpretation and relevance of results; overall consistency of messaging across sections and across the clinical programme; the benefit-risk assessment |
| Statistician | The description of statistical methods and the statistical interpretation of endpoints |
| Regulatory affairs manager | Adherence to the regulatory guidelines specific to the class of drug and therapeutic area and, where applicable, addressing regulatory feedback received during the clinical development programme, such as on study design or safety |
| Clinical pharmacologist | Pharmacology methods and results |
| Clinical study manager | The description of study conduct |
| Bioanalytical expert | The description of bioanalytical methods |
| Senior management | The alignment of key messaging with overall product and company strategy |
Besides using this table in training, the same writer suggests attaching it as a PDF when starting a review cycle and whenever a document goes to senior management for review, directly or through someone else such as the medical or clinical expert, the regulatory affairs manager or the senior manager's personal assistant. Senior managers are unlikely to attend training, but the table at least tells them what the writer expects. A writer should also talk to a senior manager's personal assistant well in advance to set aside time in the manager's calendar. Such a table can be an effective way to create a common understanding within a document development team of what the review process entails.
What to read for, by document type
In the same scheme, a reviewer reads each document type for different things.
| Document type | Purpose within the eCTD | Focus |
|---|---|---|
| Clinical study report | Reports the methods applied in an individual study and its results | How the study was conducted and how results are reported at individual study level; the most detailed clinical data presentation |
| Summary of clinical safety | Summarises the safety data relevant to a particular submission | How safety is summarised across studies, usually in safety poolings; an intermediate level of detail |
| Clinical overview | A critical appraisal of the data in a submission, providing the clinical context and a benefit-risk assessment that should ultimately support the proposed label | How results are critically evaluated in support of that label; the benefit-risk assessment is essential and should be supported by the rest of the document and aligned with the summary of product characteristics, the US prescribing information and the risk-management plan; the data are not presented again at the level of detail used in the individual reports and summaries, but critically assessed |
In electronic submissions, which most submissions were by 2018, the reference documents are one click away in the eCTD structure.
What to read for, by development stage
In the same scheme, the focus also changes with the development stage.
| Stage | Focus |
|---|---|
| Shell | The methodology is complete and accurately described; the data presentation strategy is defined and fit for this specific document, given the class of drug, therapeutic area and eCTD location; the use of tables against text is agreed within the team and adequate; the inclusion of endpoints is accurate and complete; the key conclusions matching the study's primary and secondary objectives will be supported by the data shells |
| First draft | The interpretation of the data is correct, and the document is complete, with everything needed in and nothing missing |
| Second draft | First-draft comments have been appropriately addressed, and key messages are clear and consistent within the document and across the entire project, not only at the clinical level but also with the preclinical data |
| Final draft | The high-level messaging adequately supports the overall project filing strategy |
| Final version | The document is fit for purpose |
Reviewers should be told, as applicable at each draft stage, whether the document has yet been through quality control. The second or final draft is usually also reviewed by senior management, the principal or coordinating investigator, or both, and other reviewers such as key opinion leaders may also be involved at that stage. At the comments resolution meeting, reviewers from the core clinical team need to be ready to tackle challenging, strategic or conflicting comments from these additional reviewers. Writers should remind core team members of this, and potentially ask them to align with their functional line managers early on, to avoid very late surprises as far as realistically possible.
Specific, actionable, relevant comments
Writers should train reviewers to give clear, specific, constructive, actionable and relevant comments, helping each team member meet their full potential as a reviewer. Writers should let reviewers see how it feels to receive poor comments, show them why their comments are not as clear as they may think, and, after giving them guiding principles, have them think about and practise how to word comments better. As a rule of thumb, comments should be specific and constructive, actionable, and relevant.
- A specific and constructive comment points out what is bad and suggests how to correct it. "This is bad" written on a document's front page does not say what is bad and does not help the team move forward.
- An actionable comment uses an action verb, such as add, write or refer to, rather than an open question. "What about adverse events?" may leave the writer thinking "What about them?", and "Why didn't you describe adverse events?" may draw "Because I didn't want to".
- A relevant comment is one whose timing and relevance the reviewer has considered before asking for a major change that would completely change the structure or that comes too late to address before the target submission; a primary analysis, for example, cannot be redefined after database lock. A nice-to-have comment also differs from a critical one, which the team must consider and discuss.
Some common comments are ineffective, and each can be replaced by a better one.
| Comment | Why it does not help | Better form |
|---|---|---|
| "Can we align this with our recent project X?" | The writer may not have worked on that older submission and may not know whether a sentence, a paragraph or a similar table is meant | If one sentence is wanted as a prototype, the reviewer looks it up and supplies it, as in "Please describe serious infection results as follows: …" |
| "This is bad" | It does not say what is bad — a word, a sentence, a paragraph or the whole document — or how to improve it | An alternative and a reason: "Please reword this sentence to x and y because…" |
| "What about allergic reactions?" | It is neither specific nor actionable, does not state the intended level of detail, and would need a discussion before the writer could address it | The reviewer checks the data, considers their key message, and proposes a specific presentation and wording, such as "Please present allergic reactions in tabular format by treatment group" |
| "Refer to Figure 8 and place in context" | It gives a reference the writer can easily add but no content for the context | Specific wording that clarifies the regulatory or clinical context |
| "Should we use inhibitor or antagonist?" | It is a question that has quite likely been asked before within the same team for the same document | A proposal, such as using one term throughout the document and consistently across the clinical programme |
| "Can we add more information on cardiovascular events?" | It will usually be answered "Yes, if we have the data", leaves open whether serious events, fatal events, a table, text or events leading to discontinuation are meant, and shows that the reviewer did not look at the data and think about what to add with priority | A request for the frequencies of patients with serious cardiac events by treatment group, with the statement to add and the source table |
| "Why aren't you presenting haematocrit values?" | It could be answered "I didn't want to", does not say whether table or text is wanted or what the key message is, and does not say why that parameter matters | A statement of the concern and the proposed wording to add, with its source table |
A reviewer is required to take one step forward and not only identify a problem but also provide a solution. That requires some thinking and often digging into the data, but it is the only way a review is actually complete, whatever a reviewer who emails "I have completed my review" may think. Without a truly complete review, the review process adds little value, uses time and resources, and leaves medical writers feeling they are pulling teeth at comments resolution meetings. Providing good comments takes substantial time during review but avoids a lot of confusion within the team later. Learning to provide them also takes time, but once learnt and applied it pays off for everyone involved in developing the document. Replacing broad questions with specific answers and clear rewording suggestions pays off.
A responsible reviewer's checklist
One experienced writer suggests introducing reviewers, within the writer's organisation or at a client's site, to a tool she calls the Responsible Reviewer's Checklist, which sets out their responsibilities from before the review to after the comments resolution meeting. Its steps repeat at each stage until the document is final.
- Before review, get familiar with the eCTD structure, the standard operating procedures and the relevant regulatory guidelines, and put yourself in the shoes of the ultimate target audience — for a regulatory document the health authority, which needs to make a yes-or-no decision rather than be educated — and ask what it needs to know.
- Get familiar with the source data — protocol, statistical analysis plan, case report form, tables, figures and listings — and do not ask for changes that can no longer be implemented after database lock, such as a change in the primary analysis, an additional sensitivity analysis requested after second-draft review of a clinical study report close to the submission deadline, or adding data not in the database.
- Get familiar with the electronic review tools in use, whether a document management system or a collaborative review tool, and use their capabilities.
- Once timelines are agreed, make yourself available and plan the time ahead in your calendar, with your personal assistant's support; delegate the review if you will be away on vacation, guide your back-up through the basics of the project and the review expectations beforehand, and tell the medical writer about your absence and how to contact your representative.
- During review, focus on three things: input on the sections relevant to your functional role, keeping in mind the document type and the development stage; strategic input on content, in specific, actionable and relevant comments; and categorising your comments as major and critical, minor and not critical, or cosmetic — avoiding wordsmithing to personal taste and avoiding checking abbreviations or numbers in programmed in-text tables.
- At the comments resolution meeting, come prepared: consider the other reviewers' comments, and propose pragmatic solutions that balance the required content against the target submission date, especially for critical comments that may require a change in strategy or an additional analysis, considering whether the comment is truly critical at this particular stage.
- After the meeting, offer support to the medical writer with any remaining challenging comments, if needed.
Briefing and training reviewers
Writers should clarify what they want reviewers to focus on and how comments should be provided, ideally well ahead of the start of the review, at the kick-off meeting. Reviewers who understand what is being asked of them will hopefully return comments that are easy to implement and do not require endless discussion at comments resolution meetings.
One experienced writer finds that teams need to be reminded of what to focus on according to their function, the document type and the development stage. Some reviewers tend to lose track of the ultimate audience of regulatory documents, the health authority, and focus anxiously instead on how their line manager will perceive their review. Bringing reviewers back on track and reminding them what to focus on is part of the medical writer's job.
It is up to medical writers to recognise why reviews fall short and to train their reviewers without sounding patronising. A writer who has worked in several pharmaceutical companies and on the vendor side, and so knows that the medical writer's role is defined differently in different institutions, advises always raising awareness of the role medical writers play in developing documents and in the industry, and clarifying the responsibilities of all parties even where a standard operating procedure already covers them. A reviewer may have worked with medical writers in a different set-up in another company, and so expect very different things of them than writers assume, or may never have heard the term "medical writer", especially if new to their role in the pharmaceutical industry. It may be obvious to writers that, beyond writing, they manage comments, act as mediators in case of conflicting strategic timelines, bring discussions back on track during meetings and make sure that comments are kept. For a reviewer working on a submission for the first time, whether or not they wish to acknowledge it, an experienced medical writer can also be a resource for regulatory guidance, eCTD structure basics and the roles of different team members during document development, beyond what the procedures cover. A kick-off meeting for every writing project is a good starting point to set such things straight.
The principles and tables of review focus can be introduced at the kick-off meeting with each clinical team and built into routine cross-functional training, both during onboarding and as refresher training at least once a year. Laminated hand-outs with key principles and specific examples are useful at the end of such training, for reviewers to keep on their desks and use daily while reviewing. Writers should adhere to the same principles when they mentor junior medical writers and review their documents. It is up to all medical writers to create, enforce and maintain good writing and review practices across the industry.
Nobody is born a reviewer: review is a skill that can be taught and learnt. Medical writers, as communication experts, should take the time to train their teams on both their own and the team's responsibilities in document development, not all of which a standard operating procedure can cover. Everyone involved in developing a document benefits from such training, which ensures smooth collaboration and ultimately saves the pharmaceutical company time and money. Having reviewers adhere to basic, clearly laid out principles aligned with their functional role, the document type and the review stage should improve the efficiency and effectiveness of review throughout clinical development and across organisations.