Research Design

Succesful Designs for Social Economics Research

Catherine Hakim

• Publisher: Routledge
• Year: 2012
• ISBN:
  9781135125721
• Edition: 2
• Topic: Social Sciences
• Subtopic: Social Science Research & Methodology
• Index: Social Sciences

Providing a practical overview for graduates and professional researchers, this book highlights the central issues involved in the design of medium to large scale social and economic research. Covering both theoretical and policy research Hakim sets out the key features, strengths and limitations of eight main types of study, with illustrations from real life research of the kinds of questions each can best be used to answer. This book also offers a more general pragmatic discussion of strategies for choosing between one design and another, and on how different types of study can be successfully combined in wider ranging research programmes.

In this expanded second edition the author has added new material on areas of contemporary significance across the social and economic sciences. New features to this edition are:
* a chapter on cross-national comparative studies
* more examples throughout the text of comparative research both within Europe and across modern societies
* discussions of student theses, advocacy research, selection effects and collaboration.

1 Introduction

Before a building of any consequence is built, there is an initial design stage. Architects are invited to present their ideas, sometimes on a competitive basis, on the shape, style and character of the building, while taking account of its functions, purpose, location and so forth. The design stage can attract substantial interest and controversy, far more interest than the actual building work. The architect who produces the design selected as the winner will then be responsible for supervising all subsequent work to implement the design, including that done by quantity surveyors, builders and other specialists who are hired in to help turn the blueprint into a reality. The architect may never lift a single brick, but famous buildings are known by the name of the architect rather than that of the construction company. Without wishing to push the analogy too far, this book is about the architect's role and design functions in relation to social research, while most methods texts are about the builder's job.

Design deals primarily with aims, uses, purposes, intentions and plans within the practical constraints of location, time, money and availability of staff. It is also very much about style, the architect's own preferences and ideas (whether innovative or solidly traditional) and the stylistic preferences of those who pay for the work and have to live with the final result. Methods texts are about how to produce a study, once the goal is defined or chosen, and can be very dull. I do not deal with methodological and philosophical issues (see Galtung, 1967, 1990; Sayer, 1984) or with theoretical issues, which depend heavily on the particular study, the researcher's discipline and also, arguably, national intellectual styles (Galtung, 1990).

Rationale for a focus on research design

The design function is virtually invisible when a researcher carries out a project singlehanded and unfunded, developing and revising the initial plan as the study progresses. The design stage becomes more visible with the development of large-scale studies, contract research for central government and other organisations, studies involving multi-disciplinary teams, and research programmes that involve several studies concerned with a central topic or set of issues. Large-scale social research does not automatically rule out the muddling-through approach, but it does force design issues out into the open at an early stage, so that the design function becomes more visible – even if visibly badly done by committee.

The design role is also becoming more visible as a result of the increasing division of labour and specialisation in social research. Experts in sampling, fieldwork and data analysis are now well accepted. But the idea of a research design specialism is still resisted among social scientists.

Research design issues rarely arise in certain social science disciplines. Economists, for example, use secondary analysis of existing data (both aggregate data and survey micro-data) to such a large extent that design issues are often reduced to the choice of dataset. Experiments are the domain of psychology. Most social science disciplines use more than one type of study, however, although sociology is distinctive in regularly making use of all the research designs identified here. The term ‘social research’ is used here in the broadest sense to encompass all social science disciplines, and one purpose of this book is to encourage disciplines that currently use a restricted range of research designs to branch out and consider other types as well. Although the choices researchers make are inevitably constrained by resources, most people become experienced with one type of study and then stick with it, safely and unadventurously.

While sociological examples will crop up with particular frequency, given their tendency to use the widest range of research designs, this text is multi-disciplinary in orientation and is intended to be of use to all social science disciplines. Economics might benefit from the use of qualitative or experimental research, for example, and a well-focused case study should be within the repertoire of any social scientist.

There is already a vast literature on particular research techniques and methods. Despite variations in content and style, they have in common a focus on how to do research and the technical details, with occasional forays into the philosophy of knowledge. The focus here is not on how to do any type of research but on when and why any particular type of study should be chosen for a project: the pros and cons of each type, including the frequently overlooked factor of relative costs and the time required; how they overlap and hence present partial alternatives; and how they can be combined or linked together logically in a research programme consisting of a number of individual projects. The focus is on choices and strategies in research design, and the reader who decides to opt for any study type will need to refer to the relevant how-to-do-it methods literature for further guidance on implementation. The information needed to make informed choices and develop a research strategy differs from that needed actually to carry out a project, and often it is needed by people who will not themselves be carrying the proposal through to completion. This text offers an assessment of the key strengths and weaknesses of the eight types of study, their relative appropriateness for different research issues, their usefulness in terms of output and the product obtained.

One positive consequence of increasing competition for research funds is that people are asking sharper questions about the value of any research project. This means that we have to look more closely at what is gained from any study, at the relative merits of different types of study, and at the cost-effectiveness of research designs. There is now an even balance between grant research and contract research in research funding, producing greater emphasis on value for money and a trend towards tightly defined research designs with clearly stated objectives and timetables. This is leading to greater rigour in research design and should have positive consequences for social science research in the longer term. In the short term, a well-designed study is more likely to get funding than a confused design. This book should also help those who have to assess research proposals, sometimes from a different social science discipline, or while working in a practitioner or policy-making role.

Theoretical research and policy research

There are substantial design differences between theoretical research and policy research. The outline that follows reiterates the distinction drawn by Scott and Shore (1979: 224–39) and Majchrzak (1984: 11–21) between knowledge for understanding and knowledge for action, and the distinction drawn by Gibbons et al. (1994) between Mode 1 (discipline-based) and Mode 2 (policy-oriented) knowledge production.

Theoretical research is concerned primarily with causal processes and explanation. The factors (or variables) considered are frequently abstract or purely theoretical constructs for which operational definitions and indicators of varying degrees of precision and validity are developed. The intended audience for theoretical research is the relevant section of the social science community (that is, academics in the main), who can all be assumed to speak the same specialist language. The long-term aim is the development of social science knowledge. Theoretical research is essentially concerned with producing knowledge for understanding, usually within the framework of a single social science discipline.

Policy research, in contrast, is ultimately concerned with knowledge for action, and the long-term aim is in line with the famous dictum that it is more important to change the world than to understand it. This broader objective means that policy research encompasses a more diverse variety of research, including theoretical research in many cases, but also descriptive research, which maps out the landscape of a topic, issue or problem, studies to monitor how existing policy is working, extending in some cases into formal evaluation research. A distinctive feature that differentiates it from theoretical research is a focus on actionable factors (or variables) either in addition to, or in preference to, theoretical constructs; and actionable variables are usually defined operationally from the very start, from the ground up rather than down from theory. The intended audience for policy research includes all the relevant groups of policy-makers, decision-takers, public pressure groups, managers of organisations, client groups, and so on. This diversity within the intended audience leads to specialist jargon being eschewed in favour of ‘plain English’ in reports on research results and summaries of key findings.

There is no firm dividing line between theoretical and policy research. For example, change may be brought about by (academic) research results that re-orientate the way an issue is perceived, by altering people's definition of the situation and, in due course, the way they deal with it. But the differences normally have implications for research design.

The concern with actionable factors can lead to quite different emphases in policy research and theoretical research on the same topic. For example, it may be that the home environment and personal characteristics are important determinants of educational attainment, but policy research would focus more on school characteristics and processes, particularly school organisation, atmosphere, leadership, curriculum goals and teacher expectations, with a view to minimising the effects of non-school factors and maximising the impact of those factors that are malleable.

Theoretical research is normally conducted within the framework of a single discipline, sometimes based on one particular approach within the discipline. As Smelser (1980: 28) notes, ‘to bring theoretical explanation to bear is simultaneously to select, exclude and thereby distort the whole historical record’. So theoretical research looks ‘ethnocentric’ or biased from the broader perspective of policy research, which is typically multi-disciplinary or transdisciplinary (Gibbons et al., 1994: 168) and is rarely anchored to a single discipline.

To a far greater extent than theoretical research, policy research is multi-dimensional (Majchrzak, 1984: 18). There is a greater propensity towards multi-method studies and research programmes, owing to the political pressures to get a fully rounded and balanced picture on any topic. Even single projects need to cover the conflicting interests that arise on an issue. Research designs must often be multi-level. For example, research on educational policy issues may need to address the matter at the level of the local neighbourhood or school, at the level of local or state government, and at the national level, as well as taking account of the potentially conflicting interests of teachers, parents and others who are affected by changes in policy.

The need for a comprehensive picture in policy research leads to a preference for studies that are nationally representative, or else permit extrapolation to the national level. In contrast, a great deal of theoretical research is carried out with small local studies, the results of which cannot readily be generalised.

Other differences between theoretical research and policy research are of a more peripheral nature, or are contingent rather than inherent distinctions. A good deal of policy research addresses respondents and informants as role-holders rather than as private individuals, as illustrated by interviews with employer/management representatives, trade union representatives or minority group representatives. Policy research is usually carried out within an agreed timetable, whereas theoretical research projects may run for a decade or longer before completion. In order to be accessible to all interested parties, the results of policy research must be written in ‘plain English’, in addition to (or instead of) specialist language, and must be presented in the context of policy options, sometimes extending into recommendations for action at the end of a report. Policy research is socially accountable and produces socially distributed knowledge (Gibbons et al., 1994).

Questions about causal processes can arise in both theoretical and policy research, but those arising in policy research tend to be more complex than in theoretical research – to judge by the simple-minded presentation offered in many methods textbooks (see, for example, Ackoff, 1953; Krausz and Miller, 1974; Miller, 1992). These and other texts tend to focus on the example of testing whether a particular antecedent is a necessary or sufficient cause of a known behaviour, attitude or other social phenomenon; the dependent variable is identified and the antecedents, or causes, are assessed. Some types of policy research deal instead with the consequences of a given social phenomenon; the consequences can be large in number and infinitely diverse, with no prior assumptions as to their nature and number. In this case there is a single independent variable and numerous dependent variables. Examples here would be research on the effects of putting platoons of black soldiers into white infantry companies (Stouffer, 1950), the effects of school climate on teenagers’ aspirations and achievements (Coleman, 1961), the effects of natural disasters (Wright et al., 1979) or the social consequences of unemployment at the individual level and the societal level (Hakim, 1982b). Sometimes a policy research study is designed to cover both antecedents/causes and consequences/effects of a given social phenomenon, and the resulting comprehensive reports can readily look ‘descriptive’ to readers unfamiliar with the broad range of issues being covered. Examples here might be the USA National Longitudinal Surveys of Labour Market Experience described in Chapter 8 or the British 1980 Women and Employment Survey discussed in Chapter 6. Also policy research addresses macro-level causal processes, which are poorly served by a methods literature that tends to use simple micro-level examples of explanatory research – such as whether fatigue and level of illumination influence performance (Krausz and Miller, 1974: 82). When policy research is required to provide answers quickly in areas that have attracted little academic interest, the answers on causal processes must inevitably be more broad-brush and imprecise than strictly desirable. In other cases, where the timetable and funds permit it, the causal analysis can be as sophisticated as the best theoretical research ever achieves – as illustrated by the guaranteed annual income experiments described in Chapter 9.

Accurate description plays a significant part in both policy research and theoretical research. In both cases the central question is often ‘did X happen or not?’, with the bulk of the study concerned with defining, describing and measuring X with a view to concluding that it did or did not happen. The single factor in question may be the key element of a complex theory (for example, the thesis that jobs became deskilled as a result of capitalist industrialisation, or the thesis that the working class adopted middle-class values as a result of increasing affluence), the key issue in a policy debate (for example, whether health inequalities declined as a result of the National Health Service in Britain), or both (for example, whether the sex segregation of jobs has declined, whether as a result of women's changing attitudes and increased labour force participation or due to legislation prohibiting sex discrimination). Being clear that something did, or did not, happen is a crucial first step before considering possible explanations, and it may even constitute the test of a thesis.

Another difference between theoretical and policy research is their contrasting approaches to the quantitive analysis of survey and other data. Theoretical research (and academic writing more generally) is orientated towards reporting statistically significant results, with a lesser emphasis on the size and strength of any association between the social factors studied. In contrast, policy research requires robust results on associations, the impact of any given factor and so forth. Generally, this means that results must report large and strong effects rather than small but statistically significant effects; research results should not be so dependent on (and hence possibly artefacts of) such finely tuned social measurement that they might readily disappear in a replication using less precise measurement; and results should concern factors of enduring social importance rather than trivial matters. Abstrusely technical academic conflicts over research results or social measurement sometimes reflect quite different approaches to assessing the importance of research results (as illustrated by Murphy, 1985). It also means that simple descriptive tables, which show clearly very large differences or unexpected patterns and associations, can be just as useful as complex multiple regression analyses in policy research, a point discussed by Cain and Watts (1970) and Coleman (1970). Unfortunately, the statistical significance of research findings (which is determined in large part by the size of the sample used for the study) is, quite wrongly, regularly confused and conflated with the substantive or practical importance of research results, which is a matter for judgment and cannot be determined mechanically by statistical techniques (Morrison and Henkel, 1970; McCloskey, 1985). When social scientists complain that ‘significant’ research findings are ignored by policy-makers, it is often because their results were of little or no substantive importance.

Evaluation research is a growth area in policy research, with numerous texts and journals devoted to it (Abt, 1976; Patton, 1990; Chelim-sky and Shadish, 1997; Rossi and Freeman, 1999; Evaluation Review, Evaluation Studies Review Annual). ...

Table of contents

• Cover Page
• Half Title Page
• Series Page
• Title Page
• Copyright Page
• Dedication
• Contents
• Preface
• Key to abbreviations
• 1 Introduction
• Part 1 Types of research study
• 2 Research reviews, meta-analysis and secondary analysis
• 3 Qualitative research
• 4 Administrative records and documents
• 5 Case studies
• 6 Ad hoc sample surveys
• 7 Regular surveys
• 8 Longitudinal studies
• 9 Experimental social research
• Part 2 Putting together a research design
• 10 Choices and combinations
• 11 Research programmes
• 12 Organisation and funding of research
• 13 Cross-national comparative studies
• 14 Conclusions
• Bibliography
• Author index
• Subject index