Advanced Research Methods in Psychology

David Giles

• Publisher: Routledge
• Year: 2013
• ISBN:
  9781134694891
• Edition: 1
• Topic: Psychology
• Subtopic: History & Theory in Psychology
• Index: Psychology

How do you perform a MANOVA? What is grounded theory? How do you draw up a repertory grid? These, and many other questions are addressed in this wide-ranging handbook of methods and analytic techniques which uniquely covers both quantitative and qualitative methods.
Based on a broad survey of undergraduate curricula, the book takes curious readers through all of the methods that are taught on psychology courses worldwide, from advanced ANOVA statistics through regression models to test construction, qualitative research and other more unusual techniques such as Q methodology, meta-analysis and log-linear analysis. Each technique is illustrated by recent examples from the literature. There are also chapters on ethics, significance testing, and writing for publication and research proposals.
Advanced Research Methods in Psychology will provide an invaluable resource for advanced undergraduates, postgraduates and researchers who need a readable, contemporary and eclectic reference of advanced methods currently in use in psychological research.

1   Introduction

For a long time now, the psychology textbook market has been flooded with books on introductory statistics and basic research methods. Most of these books repeat the same information, covering experimental design, descriptive and inferential statistics, and a smattering of other research methods that are likely to be encountered in the first year of a psychology degree. Sometimes lecturers deliver their material straight from these texts; otherwise, students are recommended the textbooks as supplementary reading.

I am sure that, with teaching quality assessment shaking up psychology departments in the last decade, the provision of teaching materials has improved dramatically, but in most cases, supplementary reading is essential for students to grasp some of the more complicated material encountered in lab classes and statistics workshops. However, there comes a point in most psychology undergraduates’ education when the supply of useful supplementary information begins to dry up. This point usually arrives midway through level 2 (in English universities, at any rate), when research methods staff have the freedom to start delivering more specialised material.

The idea behind the present text was to dispense with introductory methods material and start somewhere in the middle, so that it would cover all the further statistical techniques that undergraduates can expect to encounter, along with qualitative approaches and more esoteric methods. The choice of content is based on an exhaustive trawl through the websites of British universities during 1999, when roughly half of the psychology departments posted sufficient information for me to identify the techniques covered.

However, I do not want readers to think of this book as an undergraduate text as such. Because of the variation in material covered in psychology degrees from level 2 onwards, I envisage this book being useful for psychology researchers well past undergraduate level. Indeed, I have already lent drafts of various chapters to colleagues. The increasing popularity of qualitative research in psychology is a case in point. Some qualitative psychologists have queried whether these chapters really deserve to be classified as ‘advanced’ methods; perhaps they are of little use to postdoctoral discourse analysts or grounded theorists. However, many experienced psychologists have received training only in quantitative methods, and freely admit that they are ill equipped to supervise qualitative projects at level 3, or even to teach these methods at levels 1 and 2.

Because I have decided to start in the middle, this book makes a lot of assumptions about the knowledge expected of a reader. I would be very surprised if any reader has failed to complete at least one year of an undergraduate degree; therefore, I am assuming that s/he will be familiar with introductory methods material. Typically, the first year of a degree is spent training students to understand inferential statistics, weigh up the pros and cons of basic research design (mostly experimental), write lucid formal reports, and consider some of the wider issues involved in psychological research (ethics, reliability and validity, and so on). More specifically, I would expect the reader to have covered the following:

•   Mean/median/mode

•   Standard deviation/Normal distribution

•   Variance

•   Statistical error

•   Generalising from sample to population

•   Difference between parametric and nonparametric tests

•   Correlation and simple linear regression

•   t-tests, one-way ANOVA, repeated measures ANOVA, factorial ANOVA

•   Basics of reliability and validity

•   Difference between interval, nominal and ordinal scales

•   Difference between independent and dependent variables

•   Basics of experimental design

•   Conventions of report writing

Some readers may wonder why I have neglected to cover basic ANOVA. The reason is that many departments now expect undergraduates to master ANOVA at level 1, even as far as factorial designs and interpreting interactions. Furthermore, there are very few introductory research methods and statistics texts that do not cover ANOVA, even as far as two-way mixed designs. Inevitably, there is great variation in the depth of this coverage. For instance, there are very few texts offering an adequate and accessible coverage of multiple comparisons in factorial designs. However, I would argue that this is their problem! It would be unwise of me to attempt to patch up other textbooks’ coverage of advanced issues in ANOVA without starting from the basics myself. So this book kicks off with a section that I have termed ‘Beyond ANOVA’, in which I introduce readers to more advanced techniques that build on the basic ANOVA model.

Occasionally, I make reference to the tension between Fisherian approaches to statistics and the Neyman-Pearson tradition. In brief, Fisherian approaches are those that centre around the null hypothesis as the key concept in inferential statistics, as argued by Ronald Fisher in his influential writings before the Second World War (e.g., Fisher 1935). ANOVA is the most important technique in this tradition, having been devised by Fisher as a way of directly testing null hypotheses under rigidly controlled experimental conditions. The Neyman-Pearson tradition (following Jerzy Neyman and Egon Pearson) focuses on correlation and regression as statistical techniques, arguing against the tradition of null hypothesis significance testing and in favour of more exploratory techniques examining the relationships between variables.

It has been argued that research methods training in psychology has evolved by way of an unsatisfactory hybrid of these approaches (Gigerenzer and Murray 1987). In the UK, much criticism has been levelled at A-level teaching of statistics in psychology for treating the null hypothesis as gospel, and asking students to learn by rote confusing and inaccurate interpretations of statistical significance (MacRae 1995). These issues are discussed in more detail in Chapter 21. More generally, there is increasing pressure on editors of psychology journals to supplement null hypothesis significance testing (if not abandon it altogether) with other statistics, notably effect size and confidence intervals, and to force authors to consider the power of their designs (a key concept in the Neyman-Pearson tradition).

I have resisted the temptation to address power concerns in the book as a whole because the jury is still out on how much they matter to psychology. To some researchers, power concerns are simply another restraint imposed by statistical purists demanding unrealistically large sample sizes; to others, they are no alternative to statistical significance, just a means of ensuring a better chance of obtaining a significant value. A priori power calculations are meaningful only where effect sizes can be predicted, and population norms known (as, say, with IQ); does this mean that psychological research can only be conducted on a select handful of measures and – possibly dubious – concepts? At worst, for most studies in psychology (where effect sizes tend to be small), high power requirements threaten to stifle research on interesting topics and populations. Furthermore, just how much more detail is really necessary to drive home a point in the Results section of a paper? The day when all authors are required to report confidence intervals may never arrive. Therefore, in most of the book, I have focused on the kinds of statistics you are likely to see reported in papers published in the 1990s and early 2000s.

The first two parts of the book are differentiated loosely on the Fisherian/ Neyman-Pearson distinction. Part I (Beyond ANOVA) deals with techniques which emerged on the back of the factorial ANOVA model and are primarily aimed at testing null hypotheses, typically differences between means, in an experimental context. In practice, MANOVA and discriminant analysis are used far more often to analyse data collected in survey-type studies than in true experiments. Nevertheless, they are frequently used, even in this context, as a way of testing hypotheses by comparing group means.

In contrast, Part II (Measures of relationship: regression techniques) deals with statistics which have been developed in the Neyman-Pearson tradition, building on the basic linear regression equation to explore relationships between large numbers of variables. In these techniques, significance testing is concerned largely with the ‘fit’ of specific models that are designed to predict a particular variable. For this reason, significance testing plays a less prominent role in these techniques than in those deriving from Fisherian principles. In Chapter 7, I introduce the reader to structural equation modelling (SEM), an advanced set of techniques which became extremely popular in psychology during the 1990s. There is not enough room in this book to cover SEM in full, though, and anyone who seriously considers using it for analysis will need to consult a more specialist text.

In Part III (Questionnaires and scales) the emphasis shifts from statistical analysis towards research design in general, as I describe the steps involved in constructing tests and scales, and techniques for measuring their reliability. There follows a chapter on factor analysis, which – in its exploratory form – shares much in common with other Neyman-Pearson type techniques; indeed, it largely dispenses with significance testing. However, the development of confirmatory factor analysis, along the same lines as SEM, throws a bit of a spanner in the works, since it is heralded by its supporters as a significance testing technique. This section closes with a round-up of some alternative techniques for data reduction such as cluster analysis and multi-dimensional scaling, methods which are popular in research but rarely covered at undergraduate level.

The following section (Part IV) represents something of a departure from traditional methods texts in that it contains a detailed look at qualitative research in psychology. More and more psychology departments throughout the world are incorporating qualitative methods into their research methods teaching at undergraduate and postgraduate level, and qualitative research is increasing rapidly in the social sciences in general. The British government has recently included qualitative methods in their ‘benchmarks’ for psychology degrees. However, there is still a lot of suspicion surrounding the merits of qualitative research, and teachers and researchers in the quantitative tradition sometimes still regard it as a soft option for students who can’t add up. To set the record straight, I have covered two qualitative techniques (grounded theory and discourse analysis) in some depth, and suggest some ways of ensuring that qualitative research meets the rigorous demands of scientific inquiry.

Traditional quantitative psychologists often find the idea of qualitative research confusing. What counts as ‘qualitative’? I have often heard content analysis referred to as a qualitative method. While qualitative content analyses are beginning to appear in the literature (see Chapter 14), in most cases the term content analysis is used to describe the coding of data, typically in text or transcript form, for statistical analysis. More confusing still is the growing tendency for statistically-oriented psychologists to speak of ‘qualitative variables’ when they mean category variables (see Everitt 1998 for an example). I have even seen the term ‘qualitative’ used as synonymous with ‘interpretative’ (Macdonald 1993). Such usage may make sense within the context of these specialised discussions, but it does not help those studying psychology, or even those highly qualified, to grasp the essential points of difference between quantitative and qualitative research.

At the same time, I disagree with the common portrayal of qualitative and quantitative psychologists as two warring sets of academics who cannot understand each others’ perspective. There is no reason why an individual researcher should not simultaneously conduct a series of psychology experiments to test a causal hypothesis while conducting a discourse analysis to examine the social construction of the same phenomenon as part of the same research project. (Admittedly, I have yet to see such a project attempted, and would be surprised to see the conservative funding councils support it!). While different approaches may be underpinned by different epistemologies, I believe there is no need for an individual researcher to commit to a lifelong position.

Part V (‘Other approaches’) is something of an odds and ends section. It opens with (quantitative) content analysis, and in the same chapter I discuss log-linear analysis, a technique that is growing in popularity in psychological research, and is particularly appropriate for data derived from this type of content analysis. Statisticians may wonder why I did not cover log-linear analysis along with logistic regression (a related technique); the answer is that the organisation of the book is based more on common usage of methods rather than mathematical lines.

In this section I also cover meta-analysis (Chapter 18), which is growing increasingly popular as an alternative to the traditional literature review paper. Some would argue that the term ‘meta-analysis’ is merely a statistical technique for combining effect sizes. Again, from a theoretical perspective I could have placed this elsewhere, perhaps by combining this chapter with the one on statistical significance and power. However, in practice, meta-analysis is almost always conducted as a kind of quantitative literature review, and its usage provokes many questions about the way we evaluate research in the quantitative tradition. As with statistical power, I have strong reservations about how appropriate meta-analysis is when applied to psychological data, which is rarely consistent in its methods and measures, not to mention its sampling procedures.

Finally, the book ends with a section (Part VI, ‘Uses and abuses of research’) that contains issues of importance for budding researchers, but may also prove interesting for established psychologists. It is almost obligatory for a methods textbook to include a section on ethics; here I have tried to address the issue of what ethical clearance is actually for. Too often it is treated as a form of political correctness, and I sometimes wonder whether the idea of having separate chapters on ethics in books helps dispel this concept. Ethical concerns are at the heart of all research methods and should be design issues, not a list of spoilsport restrictions. Above all, they call into question the whole point of research. Chapter 21 is the aforementioned chapter on statistical significance and related issues. Here you will find details on how to calculate power and confidence intervals.

The book ends with two useful chapters on writing research papers and applying for grant money. These activities are of paramount importance in academic life today, but newly-qualified psychologists often find themselves abandoned after postgraduate study without any real guidance. The typical case is a young lecturer who, immediately on finishing a PhD, lands a lecturing post in a cash-strapped department which is desperately trying to plug a hole in its teaching timetable. Senior staff may be research-inactive, or too jaded to pass on advice; without any opportunity to build a research profile, the new lecturer has little chance of an escape route. These two chapters are dedicated to that lecturer, but postgraduates, or anyone thinking of starting a research career, may also benefit from the information.

Finally, a feature of the book is the frequent use of detailed examples from the psychological research literature. Recently, I read a meta-analysis chapter in a general psychology methods textbook which failed to give a single example, from the many available, of a meta-analysis that had been conducted by a psychologist. I fail to see how a reader unfamiliar with the technique could come away from that chapter any the wiser. Where possible I have used two examples of each technique, especially where there are variations in use, and I have tried to draw them from literature published in the five years before the book’s publication. I hope in doing so I have made the book directly relevant to psychology in the twenty-first century.

Part I

Beyond ANOVA

The first part of this book is devoted to statistical procedures that have emerged from the broad concept of analysis of variance, which dates back to Ronald Fisher’s work on experimental design in the 1920s. Fisher was primarily interested in the effects of different fertilisers on crop yields, which may seem rather remote from the kind of experiments you are likely to carry out in psychology. Nonetheless, from the 1930s onwards, psychologists began to adopt Fisher’s techniques as gospel, and they are fundamental to the way statistics and experimental methods are taught to psychology students today.

The more advanced techniques in this chapter were developed after the Second World War, as statisticians and psychologists found the basic ANOVA ...

Table of contents

• Cover Page
• Half Title page
• Title Page
• Copyright Page
• Contents
• List of figures
• List of Tables
• Preface
• 1 Introduction
• Part I Beyond ANOVA
• Part II Measures of Relationship Regression Techniques
• Part III Questionnaires and Scales
• Part IV Qualitative Methods
• Part V Other Approaches
• Part VI Uses and Abuses of Research
• Appendix 1
• Appendix 2
• References
• Index