Research Design

Creating Robust Approaches for the Social Sciences

Stephen Gorard

• Publisher: SAGE Publications Ltd
• Year: 2013
• ISBN:
  9781446290859
• Edition: 1
• Topic: Social Sciences
• Subtopic: Social Science Research & Methodology
• Index: Social Sciences

Research design is of critical importance in social research, despite its relative neglect in many methods resources. Early consideration of design in relation to research questions leads to the elimination or diminution of threats to eventual research claims, by encouraging internal validity and substantially reducing the number of alternative explanations for any finite number of research ?observations?.

This new book: discusses the nature of design; gives an introduction to design notation; offers a flexible approach to new designs; looks at a range of standard design models; and presents craft tips for real-life problems and compromises. Most importantly, it provides the rationale for preferring one design over another within any given context. Each section is illustrated with case studies of real work and concludes with suggested readings and topics for discussion in seminars and workshops, making it an ideal textbook for postgraduate research methods courses.

Based on the author?s teaching on the ESRC Doctoral Training Centre "Masters in Research Methods" at the University of Birmingham, and his ongoing work for the ESRC Researcher Development Initiative, this is an essential text for postgraduate researchers and academics. There is no book like Research Design on the market that addresses all of these issues in an easy to comprehend style, for those who want to design research and make critical judgements about the designs of others.

1 Introduction – ‘Design’ as Distinct from Methods

One What is Research Design?

Summary

• Attention to research design at an early stage is necessary for rigorous social research.
• Many areas of social science do not pay enough attention to design.
• Many existing resources for new researchers over-emphasise research methods at the expense of design.
• The elements of a research design include the cases studied, their allocation to sub-groups, the timing and sequence of data collection, and any interventions.
• These elements can be portrayed in a simple design notation to represent new or existing designs.
• An important part of research design involves thinking beforehand about the kinds of conclusions that you might want to draw.

1.1 Introduction to Design

This is a book about research design for social scientists. It argues that research design has been largely ignored in the development of new researchers, at the expense of a focus on methods of data collection and analysis. Perhaps this is because so many people generally care so little about their social science. To understand what I mean by this, consider areas of innovation where research design is strong. These might include the development of transport such as cars or elevators, of consumables such as medicines or packaged foods, and of gadgets from toasters to internet-capable televisions. In all of these areas, and many more, all of the products are tested before use. In many countries it is illegal to market such a product without rigorous testing. Even if it were not illegal, a strong pressure to test all products would come from the consumer. People want their aircraft to fly rather than crash, and their medicines to work rather than to poison them inadvertently. So, the research to test these things must be designed with a robust approach. Of course medicines and aircraft still fail, despite testing. This regrettable fact is not an argument against testing; it is an argument for more and better designed testing.

People should also care about the quality of studies in economics – witness the worldwide economic downturn in 2007/08 that was almost entirely un-predicted by the thousands of professional economic researchers in each country. The public should care about the billions of public money spent on school ‘improvement’ schemes that have no discernible impacts on the desired outcomes. Similar concerns should arise in research relevant to housing, crime, social services, business leadership, politics, international development, well-being, social inequalities, marketing, and a host of other fields. Perhaps people do not care as much as they might because, even where research in social science has serious public implications, the ‘result’ could be a long way off in the future, or hard to discern in the present. People rarely fall ill or die as a direct result of poor social science research. Now, this should not mean that they do not want improvements in public services like housing, education, or criminal justice. But perhaps their concern is less immediate than the fear that a badly designed plane might crash, because the consequences of poor design in social science could be less visually dramatic.

Two other reasons may be that social science research is often ignored by its potential users such as politicians, and practitioners in the public services, and that its research findings are often of very poor quality anyway. None of these reasons is an excuse, but in combination they might form an explanation for how and why social science research gets away with ignoring research design. What this book does is to imagine that more people genuinely care about the quality of social science research, in the way that they care about the effectiveness and safety of aircraft and medicines. The book imagines that when a child is taken into care, or a government changes the sentencing guidelines for criminal courts, then the public would demand that these decisions are made using the best possible evidence.

Design is not chiefly about techniques or procedures. It is more about care and attention to detail, motivated by a passion for the safety of our research-based conclusions. At its simplest, research design is about convincing a wider audience of sceptical people that the conclusions of the research underlying important decisions are as safe as possible. This is perhaps the major difference between the objects of design in medicine and engineering, where things can be seen to work or fail quickly, and in most social sciences, where we can only seek to be convincing. If something works, that is convincing in itself, but where we do not know whether something works, we can at least demand to be convinced that it ought to work. We should want to be convinced that it is worth risking the happiness of a family by removing a child from its parents, risking public safety by releasing prisoners early, or spending public money on almost any intervention. Such decisions might be correct, or they might be a wasted opportunity or worse. It is the task of social scientists to help make such decisions as foolproof as possible. At present, despite a small amount of excellent work in every field, this is just not happening sufficiently.

New researchers largely complete their development lacking any understanding of research designs, and this is reflected in the inadequate work of many areas of public policy research. There are many examples of public policy interventions, some covered in this book, that have been well-intended and rolled out into practice on the basis that they seem plausible and unlikely to do any harm. Yet when they have been rigorously evaluated, they have been found to be ineffective or even harmful. This means that ineffective and even harmful initiatives can divert scarce resources away from effective ones – a particular problem in the current economic downturn, when decisions are being made to abandon programmes on a whim rather than in terms of genuine cost-effectiveness. So, policy-makers and public auditors are increasingly calling for good research evidence on the development of cost-effective and efficient policy and practice solutions, establishing causal-type relationships between innovative changes and their desired effects. This is a key ethical issue for publicly-funded research.

In an attempt to improve the situation, this book is for a range of audiences. These suggested readers include newer researchers in those areas of social science where design is already important – including health promotion studies, for example. For them, the purpose of the book is to provide a relatively gentle introduction that can lead to more advanced templates for rigorous research design. The book is also for newer researchers in areas where research design is present only in a limited fashion. It should encourage them to go beyond focusing almost exclusively on longitudinal designs in sociology, or merely laboratory experiments in psychology. For them, the purpose of the book is to set the common design(s) within their disciplines into a wider context, and to suggest that a mature social science requires a greater variety of designs. Perhaps, most urgently, this book is for newer researchers in those many areas of social science where design is almost completely absent, where methods resources do not even address design, or it is confused with instrument design, post hoc statistical procedures, or bizarre issues like ‘paradigm wars’ (Gorard 2004a). This is probably the situation in most fields, including economics – the supposed ‘queen’ of the social sciences.

This is most definitely a book for readers who do not know what research design is, did not take a course on it as a doctoral researcher, who would otherwise feel content to continue with their existing approach to generating evidence for public consumption, and whose mentors, supervisors and colleagues feel the same. As this book argues, such complacency is unethical and unwarranted. In the example areas listed so far there are key issues of safety, efficiency and equality. People have lost their jobs as a result of an economic downturn caused partly by untested financial products, for example. The public should care about such things, but the researchers who work in such areas often claim to care about them even more. If they do care, they will want to ensure that they design their research to be as rigorous as possible. Ignoring design is one way of saying openly to the world – ‘I don't care about the quality of my research, the wasted opportunities it represents, the waste of peoples’ time participating in or reading it, or the dangers to the very people that the research is meant to help’.

1.2 Design and Methods

An important point for readers to understand is that research design is not about methods of data collection and analysis. What all rigorous research designs, and variants of them, have in common is that they do not specify the kind of data to be used or collected. No kinds of data, and no particular philosophical predicates, are entailed by common existing design structures such as longitudinal, case study, randomised controlled trial or action research. A good intervention study, for example, could and should use a variety of data collection techniques to understand whether something works, how to improve it, or why it does not work. Case studies involve immersion in one real-life scenario, collecting data of any kind ranging from existing records to ad hoc observations. The infamous ‘Q'-words of qualitative and quantitative, and mixed methods approaches are therefore not kinds of research design; nor do they entail or privilege a particular design. Of course, all stages in research can be said to involve elements of ‘design’. The design of instruments for data collection is one example. But research design, as usually defined in social science research, and as discussed throughout this book, is a prior stage to each of these. Thinking about methods before design is similar to an architect ordering building materials or setting deadlines for construction before settling on what the building will be (de Vaus 2001).

This point is quite commonly confused in the literature, where randomised controlled trial designs are seen as tied to ‘quantitative’ methods of data collection and analysis (Ceglowski et al. 2011), or it is assumed that a life-course research design must be ‘qualitative’ (Fehring and Bessant 2009). This point is also confused in some research methods resources, even those purportedly about design, including Creswell and Plano Clark (2007) who are really writing about methods issues not about research design. These writers and many like them contribute to the widespread misunderstanding of design issues. Do not be misled. Otherwise, judgement about what should be a design issue, such as how well the research will cater for rival explanations of the evidence, will be confused with judgement about the perceived merits of a method, such as whether to use a survey or interviews.

A study that followed infants from birth to adolescence, weighing them on 1 January every year, would be longitudinal in design. A study that followed infants from birth to adolescence, interviewing their parents about their happiness every year, would also be longitudinal. A study that did both of these would still be longitudinal, even though some commentators would distractingly and pointlessly categorise the first study as ‘quantitative’, the second as ‘qualitative’, and the third as ‘mixed methods’. In each example, the design – ‘longitudinal’, or collecting data from the same cases repeatedly over a period of time – is the same. This illustrates that the design of a study does not entail a specific form of data to be collected, nor does it entail any specific method of analysis; nor does any method require a specific research design.

Almost all existing research resources for newer researchers concern methods of data collection and analysis, and almost all of the rest concern red herrings about paradigms, or treating serious subjects like epistemology as though they were fashion items to be tried on and rejected on a whim. This is true even of many texts that claim to be about research design. This book is very different. Methods of investigation and the philosophy of social science are important, and aspects of both appear throughout the book. But they are not its starting point or its focus.

1.3 The Elements of Design

The elements of design covered in this book include the cases (participants) involved, the ways in which cases can be allocated to sub-groups, the time sequence of data collection episodes, and any manipulated interventions. These elements are the same, except perhaps for some terminology, as those presented by de Vaus (2001) and Shadish et al. (2002). The book presents these elements of design using a shorthand notation, as a convenient way of expressing more complex designs, and the differences between them. The notation is very simple, and all designs will also always be fully described and illustrated with examples where they first appear in a chapter. Do not be alarmed. What follows here is a brief introduction to the notation.

In a design, the cases are the participants in a study or the objects of a study. The letters R, C, M and N are used to denote groups of cases, allocated to their groups randomly (R), by using a cut-off point (C), through matching (M) or none of these (N). The letter O is used to represent an episode of data collection, which could be observation, measurements, conversations, text or indeed any form of data. If it is necessary to distinguish two or more different types of data collection, a sub-script will be added to the standard notation O. Thus, O₁ and O₂ might represent two different kinds of data taken from the same cases (such as a standard test and an interview). This vagueness about what the methods of data collection are is deliberate (see above). The letter X is used to represent an intervention or change of some sort that might influence the cases to which it is applied. Again, if it is necessary to distinguish two or more different types of intervention, a sub-script will be added to the standard notatio...

Table of contents

• Cover
• Half Title
• Publisher Note
• Title Page
• Copyright Page
• Contents
• Illustration List
• Illustration List
• About the Author
• Preface
• Glossary and Key to Symbols
• 1 Introduction – ‘Design’ as Distinct from Methods
• One What is Research Design?
• Two Introducing Designs in the Cycle of Research
• 2 The Preliminaries of Research Design
• Three Identifying Researchable Questions
• Four Warranting Research Claims
• Five The Nature of Causal Claims
• 3 Putting it All Together
• Six Identifying the Sample or Cases
• Seven Preparing for Comparative Claims
• Eight Matters of Timing and Sequence
• Nine Evaluating Controlled Interventions
• 4 More Advanced Considerations
• Ten Further Designs for Evaluation
• Eleven Challenges for Validity
• Twelve How Big is a Difference, and Related Matters
• Thirteen A Second Principle of Ethics
• 5 Conclusion
• Fourteen Revisiting the Need for Robust Design
• References
• Index