Guidance for Stand-Alone Paper (SAP) Literature Review

Aims of this page

This page is intended to support trainees who are writing a Stand-Alone Paper (SAP) for their literature review. It will:

1. Describe the steps needed to produce your SAP. Although SAP reviews vary in content and method, there is a common sequence of tasks to be accomplished.
2. Signpost the resources available to support carrying out the SAP.

Sequence of steps

The table below describes the sequence of steps that you will need to take in order to complete your literature review.

At each step, you will need to make methodological decisions, and it is important that the finished SAP literature review be explicit about this process. For example, if at Step 3 ('Setting inclusion and exclusion criteria') you decide to exclude studies published before 1980, you will need to briefly explain why you did this.

Under some circumstances, you may have to deviate from this timetable. For example, the empirical project might require intensive data collection over the summer between the second and third year, meaning that the literature review has to be delayed. If you think you will be unable to submit a good draft in November of the third year, you should discuss this with your supervisor and come up with an alternative timetable.

Step 1 - Question formulation for the SAP

The SAP literature review asks a question (or linked questions) that can be answered by the empirical literature. Choosing a good question is not a trivial task; it takes time and effort. Here are some tips:

• The process of choosing a question is iterative. It involves coming up with some initial ideas and then using informal searches to get a sense of the body of relevant literature, and whether any reviews in the area have already been done. These initial searches will yield information that helps you to clarify your provisional questions. You will then go back to the literature, and this could cause further evolution of your questions.
• Involve your supervisor. They are an invaluable ally and guide in this iterative process of coming up with ideas and then exploring the literature.
• Make sure your question has some clinical relevance.
• The review topic should have a logical link to the empirical paper topic, so that there is a common thread running through the thesis. For example, the link could be a shared clinical phenomenon, population or intervention.

Relevance of previous reviews

It is common to come across already-published reviews that at least partially overlap with your own SAP literature review. This creates a dilemma: should you proceed with your review anyway, or find another question? The key issue is whether your review could potentially contribute substantial new knowledge, beyond that available from the already-published review(s). For example, several important new studies may have been published since the previous review. Or, perhaps your SAP question can be adjusted, to be wider or narrower.

Step 2 - Deciding on a review method

All literature reviews involve synthesising studies in order to answer a question, or a set of questions. There are different ways of doing this, i.e., there are different types of literature review. The type depends on the question, and the nature of the available literature. The main types are:

Narrative synthesis

This is the approach most commonly used for a SAP literature reveiw in DClinPsy thesis. Its defining characteristic is the "systematic review and synthesis of findings from multiple studies that relies primarily on the use of words and text to summarise and explain the findings of the synthesis" (Popay et al., University of Lancaster).

You can find some examples of DClinPsy SAP literature reviews that use narrative syntheses, in the literarure reveiw section of the Research Project Support Moodle.

Meta-analysis

Meta-analysis is a procedure for statistically synthesising findings by aggregating effect sizes across several studies. It can be used to yield a useful summary statistic and also examine moderators, i.e., test whether there are study variables that influence effect sizes. Whilst meta-analysis is often used to synthesise group difference effect sizes from trials, it is actually a versatile technique that can provide a quantitative synthesis of different types of effect size statistics (e.g., odds ratios, correlations).

You can find some examples of DClinPsy SAP meta-analyses in the literature review section of the Research Project Support Moodle.

Getting help with meta-analysis

If you need help with your meta-analysis, the first person to consult is your supervisor. If you need additional support, Vyv Huddy and Amanda Williams are both happy to support trainees with their literature reviews, including meta-analyses, and should be contacted by email in the first instance.

Meta-analysis software

Review Manager (RevMan) is freely downloadable Cochrane Collaboration software developed for systematic review and meta-analysis, and is supported by information on the Cochrane site. It is especially straightforward to use for synthesising the outcomes of clinical trials. Amanda Williams offers support for use of RevMan.

In addition, the 'metan' routine in STATA is a useful tool for meta-analysis, and is available to trainees via the cluster room PCs and the UCL remote desk top. Vyv Huddy offers support for use of STATA for meta-analysis. Another option is the 'metafor' routine in R, which is freely downloadable. The course test library has a book on doing meta-analysis using STATA. To borrow this, please contact the research administrator.

Reviews of Qualitative Research: Meta Synthesis

Qualitative studies can be aggregated using a procedure called 'meta-synthesis', or 'qualitative meta-analysis'. The procedure is essentially a thematic analysis of the themes in each of the individual studies under review. The following reference is a useful introduction to this type of literature review:

Timulak, L. (2009). Meta-analysis of qualitative studies: A tool for reviewing qualitative research findings in psychotherapy. Psychotherapy Research, 19(4-5), 591-600.

You can find an example of a DClinPsy SAP meta synthesis in the literature review section of the Research Project Support Moodle.

Step 3 - Setting inclusion and exclusion criteria

The inclusion and exclusion criteria translate the research question into a set of formal requirements for the papers to be reviewed. They are analogous to the inclusion and exclusion criteria in the empirical study, which explicitly state which participants you intend to recruit. They address the question of exactly which kinds of studies you are looking for to answer your question(s). The search strategy then will dictate how you locate these studies.

For reviews of interventions, and for some other review topics, it is often helpful to use the PICOS framework when setting out the inclusion and exclusion criteria, i.e. to specify the Population, Intervention, Comparator, Outcomes and Setting.

The quantity of studies

Trainees often ask 'how many studies do I have to review for a SAP?' Alas, there is no simple answer to this question. Fundamentally, the SAP literature review is intended to demonstrate your capacity to appraise and synthesise empirical evidence. This is demonstrated not just by the number of papers reviewed, but also on the richness of the information they convey and the quality of your synthesis.

In practice, most UCL DClinPsy SAP literature reviews cover more than 10 papers and fewer than 40. Nevertheless, some successful SAP literature reviews have covered fewer than 10 papers. Such reviews were enriched by various means, such as reviewing multiple outcomes or multiple assessment points and providing a deeper consideration of methodology.

SAP reviews rarely incorporate more than 40 studies and in this case would most often synthesise their findings using quantitative meta-analytic methods. A review can be undermined by trying to cover too many studies. If initial searches identify an excessive number of studies, then the review's scope can be narrowed.

Step 4 - Designing and conducting the search

The SAP literature review must be systematic, using a rigorous and transparent search of the literature in order to identify all studies that meet your inclusion criteria. The systematic search is the cornerstone of a good SAP literature review, as it ensures that a fair and balanced picture of current research is presented.

Conducting a systematic search for a DClinPsy literature review

The Cochrane Handbook offers guidance on systematic search procedures. However, for the DClinPsy, compromises are commonly made: e.g., only peer-reviewed papers, English language, and not emailing authors for missing information. Some of these compromises may need to be reconsidered if you subsequently prepare the paper for publication.

Support from Library Team

It is often useful, though it is not required, to discuss the search strategy with a UCL librarian. Librarians are experts in information retrieval, and the UCL Library Service has an identified Psychology specialist. A member of the UCL library team delivers specific session on carrying out your systematic search. This includes advice on:

· resources for identifying key concepts

· choosing an appropriate database

· defining search terms

· revising the search strategy

· hand searching

· removing duplicate items

· good practice in systematic searching.

A library services module is available on Moodle - WISE - provides further resources on these steps and can be found here (link: http://www.ucl.ac.uk/library/training/guides).

Recording the search results

It may help to store and organise the information gathered by the search on a database. This could include the identified studies, recording duplicate publications of results, reasons for inclusion or exclusion, information that has been extracted and coded (demographics, clinical information, design, measures and main findings).

For your SAP reviews there is no obligation to have independent confirmation of study inclusion, data entry, coding of studies or for other researchers to check of the veracity of the database. This is often carried out in registered systematic reviews, and, if you wish to publish your review then this may be necessary. However, such 'double-checking' places unacceptable demands on trainees, who have to do their review in a limited amount of time.

Flow diagram using PRISMA format

The search should be depicted in a flow diagram, showing the stages from the search to the final set of studies, and indicating decisions and reasons for exclusion at each stage. The PRISMA statement (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) provides a framework that can be used for all kinds of reviews. When you present your SAP literature review in the thesis, it should contain a PRISMA diagram.

Step 5 - Synthesising the literature

Once you have identified the studies that meet your inclusion and exclusion criteria, it is time to embark upon the fundamental task of your SAP review: synthesising these studies in order to address your research question(s).

How you do this will depend upon the type of SAP review: for example, the method of synthesis is different for a meta-analysis compared to a narrative review. Nevertheless, it is always a good idea to create a table describing key features of your studies. This serves to summarise information, and can then give you ideas about themes and trends in the studies.

Quality appraisal / Risk of Bias

All SAP literature reviews involve a critical appraisal of the methodological quality of the included studies, and of how study quality impacts on interpretation of findings. Increasingly, researchers use structured measures, known as 'quality appraisal tools' to rate the quality of each study. There is a course document on the literature review section of the Research Project Support Moodle giving a list of these tools.

There is no requirement for trainees to use a quality appraisal tool in their literature review. Whether or not you use one will depend upon the nature of the literature you are reviewing, the question you are asking and whether a relevant tool exists. If the review concerns treatment efficacy based on findings of randomised controlled trials, a quality appraisal tool will normally be used, usually the Cochrane one called 'risk of bias'. On the other hand, many topics are not well suited to this sort of structured appraisal, or relevant appraisal tools may have not been developed.

The bottom line is that trainees should only use an appraisal tool if it will enhance their review. When thinking about whether to use one, it may be worth considering some of the criticisms that have been levelled at them:

• Lack depth necessary for certain specialist topics
• Items can be difficult to interpret (e.g. "study design evident and appropriate")
• Information required is often missing, and that can be authors' or journal editors' decision, so cannot be interpreted usefully.
• Single overall quality score is arbitrary, scoring serious defects (e.g. statistical flaws or inappropriate outcome measures were used) as the same weight as minor concerns (therapists could not be blinded to which intervention they delivered).

Step 6: Writing the SAP review

The SAP literature review has the format of a scientific paper, with sections entitled Introduction, Methods, Results and Discussion.

One commonly used approach to producing a first draft of the SAP is to initially write the Introduction and Methods sections, and to produce the table (or tables) describing the identified studies. This is then discussed with the supervisor, laying the ground for the Results and Discussion to be produced. It is important to talk to your supervisor early on in the process to agree the steps you will take towards producing a first draft.

Redrafting

All written work is improved via a process of drafts being critiqued. You should expect to receive in depth comments from your supervisor on your first draft; and to undertake substantial revisions when producing a second draft. You may need to go through further iterations to ensure the literature review is at the appropriate standard. Please see the section of the literature review on the 'Writing and presenting your thesis' for more information on the format and structure of this piece of work.

When writing the SAP literature review, be sure to cover research and clinical implications in your conclusions. Say what studies would best resolve uncertainties about your research question, and say what clinicians might decide to do in the light of current evidence. Recognise that clients will also read these, or they may be summarised for patient groups: think about what they want to know and how you can help meet that need, at least in part.

Length of the final draft

As a rule of thumb, reviews should be around 8,000 words, not including tables and references. If your review is longer than this, the chances are that it would be improved by being edited down.

Additional matters on the literature review

Publishing your SAP review

Trainee SAP literature reviews are original pieces of work that generate new understandings of clinically-important topics. They are often worth publishing. We recommend that, once you have the review written, you discuss this with your supervisor. They can help you decide what journal to target, and how the review will need to be modified in order to be considered for publication. You may need to do some work shortening the review, updating your search and getting someone to independently validate some steps of the review process (e.g., searches, data extraction). The PRISMA criteria described above often become a requirement at this stage; this depends on the editorial guidelines of specific journals.

There has been a lot of criticism of systematic reviews recently for concluding that the intervention may or may not be helpful and that more research is needed. That doesn't really say anything new, and is disappointing to clinicians, service users, and service funders and commissioners. It is far more helpful to be specific about what effects interventions have or do not have; how confident you can be in that finding (e.g., would it easily be overturned by the publication of a couple of large negative trials?), and what the 'more research' that is needed should look like. This draws on good features of the studies you reviewed, and tells people what to look for (or potential researchers what to do) that will advance the field.

Prospective registration of SAP reviews

Transparency is a vital aspect of published literature reviews. Review registration has emerged as a means of ensuring transparency by demonstrating that a review took place as planned, and can be replicated easily. This means the protocol cannot be amended at a later stage in response to disappointing results. The International Prospective Register of Systematic Reviews (link: https://www.crd.york.ac.uk/prospero/about.php?about=about) has been developed to enable researchers to register their reviews and publish their protocol. To register a review, researchers complete an online form that prompts for a range of information concerning the protocol. This is a useful process while planning a review, regardless of whether the review is subsequently registered. There is no obligation for trainees to register their review but journals are increasingly asking for information regarding registration. This should be borne in mind if the trainee and supervisor plan to publish the review.

Discovering a similar review at a late stage of the process

Occasionally, as a trainee is coming to the end of producing their SAP literature review, they discover another recently-published review that substantially overlaps. It is important to recognise that under such circumstances, trainees are not expected to start again on another review. As long as the overlapping review was published after the trainee did their searches, examiners will be understanding. If you do discover that an overlapping review has been published, the important thing is to consult your supervisor and the Research Director. They can help you manage the situation.