Quality Criteria

10 Key excerpts on "Quality Criteria"

• 

  eBook - ePub

  Doing Your Qualitative Psychology Project

  • Cath Sullivan, Stephen Gibson, Sarah Riley, Cath Sullivan, Stephen Gibson, Sarah Riley(Authors)
  • 2023(Publication Date)
  • SAGE Publications Ltd

    (Publisher)

  Research quality is about the strengths and weakness of research. All research has limitations – for example, a small-scale case study of a particular setting tells us only about how things work in that setting. But the study should be good enough to be credible, so we can take seriously what it tells us about that setting. While limitations are inevitable in research, low quality should not be.

  So how we do we work out what counts as high-quality research? Usually, we assess research quality by using a set of agreed Quality Criteria. This works in the same way as a marker comparing a piece of your work against a set of marking criteria. Using a set of criteria allows us to identify specific strengths and weaknesses, ensures that we are evaluating similar studies in the same way and tells us whether research meets a minimum quality standard.

  In a nutshell

  Quality Criteria

  Quality Criteria are specific standards against which research can be compared in order to:

  • illustrate each research study’s key strengths and weaknesses;
  • ensure fairness by evaluating similar studies in the same way as each other; and
  • judge whether research meets minimum quality standards and can usefully tell us things.

  Later in this chapter we offer suggestions for Quality Criteria that are useful for your project. But first we consider what shapes Quality Criteria and some debates about them.

  Why Does Research Quality Matter?

  Some argue that standardised Quality Criteria are not possible or desirable for qualitative research (see Majid and Vanstone, 2018). Others warn that attempts to assess quality can stifle methodological innovation (Parker, 2004). Our view is that methods for assessing the quality of all research are necessary. Psychology is an empirical discipline, which means using evidence to justify our claims and the actions we recommend. Most of this evidence comes from research, which is produced through social processes that are subjective and involve interpretation. Yet considering the quality of research remains important so that we can judge whether the ways in which evidence is produced and interpreted are reasonable.

  Another argument is that qualitative researchers cannot work with an ‘anything goes’ attitude if they want their work to be taken seriously. Although this is changing, commonly used Quality Criteria often assume that all research is quantitative. Therefore, qualitative research is often evaluated against inappropriate standards (Levitt et al., 2018) that lead to it being seen as poor research rather than being recognised as simply not quantitative. To prevent this, qualitative researchers must challenge inappropriate ways of judging quality and be clear about how evaluation should be done.

  Sign up to read

  Learn more about book
• 

  eBook - ePub

  Doing Research in Counselling and Psychotherapy

  • John McLeod(Author)
  • 2022(Publication Date)
  • SAGE Publications Ltd

    (Publisher)

  6 Criteria for Evaluating the Quality of a Research Study

  • Introduction 61
  • General criteria for evaluating research 62
  • The big divide: evaluative criteria associated with quantitative and qualitative research 65
  • How research quality is assessed within the psychotherapy professional community 71
  • Conclusions 73
  • Suggestions for further reading 73

  Introduction

  There are several ways in which an understanding of the criteria for evaluating research studies can be helpful to someone who is planning to embark on their own research project. Being guided by a sense of what makes for quality and excellence in research provides a basis for critical and appreciative reading of research articles. Being able to identify appropriate standards of good research practice is also necessary during the research planning and proposal-writing process. A thorough knowledge of criteria of research quality is vital at the stage of writing up the findings of a study, and submitting a thesis or dissertation, or a paper for journal publication. Finally, familiarity with Quality Criteria makes it possible to play a role within the research community, in supporting colleagues who are planning or writing up their own studies, and acting as a reviewer of journal articles and research proposals.

  In this chapter, the issue of Quality Criteria for evaluating research is approached from three perspectives. First, there is a discussion of some general Quality Criteria that are widely accepted across the whole field of science. The focus then turns to specific criteria that are associated with quantitative and qualitative research methodologies, and finally how research quality is assessed within the psychotherapy professional community.

  Exercise 6.1 Exploring your own Quality Criteria

  Identify some therapy research studies (or studies in other fields) that have been meaningful and inspiring for you. Then identify other studies that have left you feeling bored, annoyed or frustrated. What are the key attributes that you associate with each group of studies? What have you learned from this exercise, about the Quality Criteria that are most important to you?

  Sign up to read

  Learn more about book
• 

  eBook - ePub

  Doing Qualitative Research in Psychology

  A Practical Guide

  • Cath Sullivan, Michael A Forrester, Cath Sullivan, Michael A Forrester(Authors)
  • 2018(Publication Date)
  • SAGE Publications Ltd

    (Publisher)

  Nevertheless, it is of utmost importance that researchers can warrant the credibility of their research by demonstrating how it has been carried out, and to what standard. To do this, some researchers discuss the importance of ensuring rigour (e.g. Smith and McGannon, 2017) and validity (e.g. Yardley, 2015). Although these terms tend to be associated with quantitative approaches, when used in the qualitative research context they serve as umbrella terms that cover a range of Quality Criteria and provide the readers of the research with some means of evaluating it. Criteria for rigour and validity in qualitative research can include the transparency and coherence of the research process, and the researcher’s relationship with the research process. It is also valuable to show why the research is worthwhile, and that it will have academic or professional relevance through its contribution to the field.

  Definitions

  • Rigour: The practice of doing something with great care, thoroughly and systematically to ensure the process and outcome is of a high quality.
  • Validity: In the context of qualitative research, this refers to the credibility or believability of the research – are the findings trustworthy?

  Various sets of criteria for the assessment of quality in qualitative research have been proposed (e.g. Cho and Trent, 2006; Elliott et al., 1999; Yardley, 2015). Although criteria such as these are only guidelines, they offer some ways of demonstrating the quality and value of your research, thereby enabling those who read and use it to be confident that it is trustworthy.

  What Is Quality in Qualitative Research?

  A good place to start considering answers to this question is to think about what you might look for when reading other people’s research. What kinds of things would you want to know in order to be able to assess how well the research has been carried out?

  You will always be interested to know which method has been used, but you may also want to know why this particular method was used and not another one. You will want to know why and how particular participants were recruited to take part in the study, and what kind of information was sought from them. You will probably also want to know how data was gathered and what was done with this data. As an interested reader, you will want to know what theories and other influences were brought to the interpretation of the data analysis.

  Sign up to read

  Learn more about book
• 

  eBook - ePub

  Applied Psychology in Talent Management

  • Wayne F. Cascio, Herman Aguinis(Authors)
  • 2018(Publication Date)
  • SAGE Publications, Inc

    (Publisher)

  The challenge is to develop theories, concepts, and measurements that will achieve the twin objectives of enhancing the utility of available procedures and programs and deepening our understanding of the psychological and behavioral processes involved in job performance. Ultimately, our goal is to develop a comprehensive theory of the performance of men and women at work (Campbell & Wiernik, 2015; Viswesvaran & Ones, 2000).

  In the early days of applied psychology, according to Jenkins (1946), most researchers and practitioners tended to accept the tacit assumption that criteria were either given by God or just to be found lying about. It is regrettable that even today we often resort to the most readily available or most expedient criteria when, with a little more effort and thought, we could probably develop much better ones. Nevertheless, progress has been made as the field has come to recognize that criterion measures are samples of a larger performance universe and that as much effort should be devoted to understanding and validating criteria as is devoted to identifying predictors of them (Campbell & Wiernik, 2015). Wallace (1965) expressed the matter aptly when he said that the answer to the question “Criteria for what?” must certainly include “for understanding” (p. 417). Let’s begin by defining our terms.

  Definition

  Criteria has been defined from more than one point of view. From one perspective, criteria are standards that can be used as yardsticks for measuring employees’ degree of success on the job (Bass & Barrett, 1981; Guion, 1965; Landy & Conte, 2016). This definition is quite adequate within the context of personnel selection, placement, promotion, succession planning, and performance management. It is useful when prediction is involved—that is, in the establishment of a functional relationship between one variable, the predictor, and another variable, the criterion. However, there are times when we simply wish to evaluate without necessarily predicting. Suppose, for example, that the HR department is concerned with evaluating the effectiveness of a recruitment campaign aimed at attracting members of underrepresented groups (e.g., women for science, technology, engineering, and mathematics—STEM—positions). Various criteria must be used to evaluate the program adequately. The goal in this case is not prediction but rather evaluation. Fundamentally, one distinction between predictors and criteria is time (Mullins & Ratliff, 1979). For example, if evaluative standards such as written or performance tests are administered before an employment decision is made (i.e., to hire or to promote), the standards are labeled predictors. If evaluative standards are administered after

  Sign up to read

  Learn more about book
• 

  eBook - ePub

  Qualitative Research in Health Care

  • Catherine Pope, Nicholas Mays, Catherine Pope, Nicholas Mays(Authors)
  • 2019(Publication Date)
  • Wiley-Blackwell

    (Publisher)

  In part, this is because the choice and relative importance of different criteria of quality depend on the topic and the purpose of the research. If the key question for qualitative researchers is: ‘Why do people do what they do?’ then for Popay et al. research quality relates to the sampling strategy, adequacy of theory, collection and analysis of data, the extent to which the context has been understood, and whether the knowledge generated incorporates an understanding of the nature of the subjective meanings that people use to make sense of the world in their social contexts [ 16 ]. From this perspective, while there may be some broad similarities between quality standards in quantitative and qualitative research – that is, similar concerns with truth, applicability, consistency, and neutrality of research – the fundamental differences in the knowledge each approach generates require that quality is assessed differently in the two traditions [ 17 ]. Hammersley has attempted to summarise the different Quality Criteria and concerns of the relativists (or anti‐realists), as follows. [ 14 ] The degree to which substantive and formal theory is produced and the degree of development of such theory. The novelty of the claims made from the theory. The consistency of the theoretical claims with the empirical data collected. The credibility of the account to those studied and to readers. The extent to which the description of the culture of the setting would provide a basis for competent performance in the culture studied. The extent to which the findings are transferable to other settings. The reflexivity of the account – that is, the degree to which the effects of the research strategies on the findings are assessed and/or the amount of information about the research process that is provided to readers. These criteria are open to challenge. For example, it is arguable whether all research should be concerned to develop theory

  Sign up to read

  Learn more about book
• 

  eBook - ePub

  Advanced Qualitative Research

  A Guide to Using Theory

  • Michelle O?Reilly, Nikki Kiyimba, Michelle O?Reilly, Michelle O?Reilly, Michelle O?Reilly, Michelle O?Reilly, Michelle O?Reilly, Michelle O?Reilly, Michelle O?Reilly, Michelle O?Reilly(Authors)
  • 2015(Publication Date)
  • SAGE Publications Ltd

    (Publisher)

  In this chapter we present debates about how to assess the quality of qualitative research. Typically the literature on quality relates to the conduct and reporting of qualitative research (Spencer et al., 2003), and yet there is no consensus in relation to how to critically appraise qualitative work. The chapter opens by exploring a history of the use of quality checklists in quantitative fields and explores the relevance of the core quality concepts of reliability, validity and generalisability. We develop this discussion in terms of how these issues have translated to a qualitative context and take a critical position on attempts to construct universal checklists in the qualitative approach. We acknowledge that any research compromises its utility where defensible rigour is not made explicit; however, we question the uncritical universal application of particular markers. While checklists have contributed to the wider acceptability of qualitative approaches, these can be counterproductive if followed too prescriptively (Barbour, 2001) and it is necessary to take into account the heterogeneity of the different methodologies. The chapter concludes with some consideration of the diversity across methodologies and that, whilst at one level there may be a small number of general principles which may be universally appropriate, it is also necessary to account for the specific and unique criteria within different methodologies.

  Defining the Need for Quality Criteria for Qualitative Research

  In quantitative research there has been a long-standing acceptance that research should be judged against a predefined set of Quality Criteria. Although there is some debate about what constitutes quality markers, there is no universal acceptance of what these criteria are for qualitative work. The consensus within the academic community that scientific principles should be applied to research processes has meant that the quantitative paradigm still dominates public understanding of what equates to scientific rigour in research (Cheek, 2007). Given this dominance, it is perhaps unsurprising that these quantitative criteria have been hugely influential in appraising qualitative work. However, it is fundamentally accepted that qualitative research is different from quantitative work. The debate about whether to use the same Quality Criteria for both quantitative and qualitative is therefore complicated by the fact that there is a lack of uniform consensus as to precisely what qualitative research is (Mays and Pope, 2000). Although the standards offered by quantitative approaches have been used or at least adapted to judge quality in qualitative work, it is important to question how to recognise high-quality qualitative research (Easterby-Smith et al., 2008).

  Sign up to read

  Learn more about book
• 

  eBook - ePub

  Analysis in Qualitative Research

  • Hennie R Boeije(Author)
  • 2009(Publication Date)
  • SAGE Publications Ltd

    (Publisher)

  This perspective has received criticism from different angles. By posing alternative demands on qualitative research, it is said to disengage itself from mainstream social science, with all the expected debates on the (non-)existence of a different kind of social science ensuing. A second point of criticism is that the newly formed criteria are, in their nature, derived from the original ones. In other words: it is old wine in new barrels. Confirmability parallels objectivity, dependability resembles reliability, credibility resembles internal validity, and transferability resembles external validity that we come to speak of later on. A third point of criticism is made by those who feel that political or humanistic criteria should not have such a large place in social scientific research.

  In the third perspective on quality, the criteria validity and reliability are maintained as worthy aims (Seale, 1999; Kirk & Miller, 1986). Although the terms are retained, the procedures to realize reliability and validity are adjusted to the specific nature of qualitative research. In doing so, qualitative research does not alienate itself from conventional social science, and it does not need to be treated in isolation. Researchers using these procedures should reflect on them by addressing how the research process – the analytical methods, the data collection procedures and the researcher’s presence – affects the results. ‘It is research that looks at itself’ (Alexander, 2008: 355). That way, readers can judge what influence the researchers’ presence and research activities may have had on the findings.

  Within this perspective on quality in qualitative research, the relationship between reliability and validity is sometimes viewed differently from the way it is viewed in quantitative research. In quantitative research reliability is commonly seen as a necessary (but in itself not sufficient) condition for validity. It is assumed that researchers simply cannot observe what they want to observe (validity) when the measurements are unreliable. In qualitative research, it is difficult to establish reliability by repeating the measurement since it often does not involve measurement with standard instruments. Instead the measures have to be developed specifically for a particular study, and they are often further adapted during use in the field (Jorgensen, 1989). As a consequence it might be argued that reliability must be doubted when using a human being as a measuring instrument.

  Sign up to read

  Learn more about book
• 

  eBook - ePub

  Managing Quality in Qualitative Research

  • Uwe Flick(Author)
  • 2018(Publication Date)
  • SAGE Publications Ltd

    (Publisher)

  Chapter Three Criteria in Qualitative Research

  Contents

  • Transferring the discussion about criteria to qualitative research 26
  • Traditional or new criteria to answer the question of quality in qualitative research? 27
  • Reformulation of traditional criteria 28
  • Alternative, method-appropriate criteria 35
  • Quality of qualitative research – an issue for technology? 38

  Chapter objectives

  After reading this chapter, you should:

  • see the problems and limitations in the attempt to apply traditional criteria to qualitative research;
  • know more about suggestions for reformulating these criteria for qualitative research; and
  • be familiar with discussions about specific criteria for qualitative research.

  As the preceding chapter should have shown, the discussion of standards of or for qualitative research is complicated first by the fact that standards are often imported from a generalist view of empirical research or from a quantitative background. A second complication results from the question of whether these standards should be relevant for all qualitative research approaches or just for one specific understanding of qualitative research. The third source of complication comes from whether standards should be used for defining research programmes as good (and better than other ones) or for assessing the particular project done with a specific approach or with qualitative research in general. This last complication relates to how to nail down the general methodological question of research standards or quality to the concrete level of doing research. Instead of using the term ‘standards’ a bigger part of the literature on quality is based on the notion of criteria for judging the quality of qualitative research.

  Transferring the discussion about criteria to qualitative research

  When the discussion about criteria is transferred to qualitative research, we can distinguish two streams of discussion. Either, first, a narrowly focused and at the same time generalist concept is pursued. It is narrowly focused, because the (wider) question of what is qualitative is reduced to formulating and applying criteria (e.g. for the validity of interview statements). It is generalist, because – often at least – the hope in the background is that the criteria are the same or similar to those in quantitative research and that they at least will have the same function in both. Or, second, a rather widely conceived but at the same time specific concept

  Sign up to read

  Learn more about book
• 

  eBook - ePub

  Qualitative Research Methods in Mental Health and Psychotherapy

  A Guide for Students and Practitioners

  • David Harper, Andrew R. Thompson, David Harper, Andrew R. Thompson(Authors)
  • 2011(Publication Date)
  • Wiley

    (Publisher)

  Part III Establishing Good Quality Qualitative Research in Mental Health 16 In Pursuit of Quality Liz Spencer and Jane Ritchie Deliberation on the meaning of quality in qualitative research has a long established pedigree but calls for systematic appraisal have intensified over the past two decades. Numerous quality checklists have been developed; indeed, both authors were involved in an initiative commissioned by the UK Cabinet Office (Spencer et al., 2003). That project, together with the experience of teaching quality appraisal, confirmed the view that Quality Criteria should be viewed as part of an ongoing debate – and, if possible, should be framed as guidelines rather than prescriptive rules. Consequently, this chapter does not provide a quality framework as such, but it reviews some key concerns about quality in qualitative research and identifies some widely held quality principles. Each of these principles is examined in more detail through a set of quality questions that might be asked of a study and illustrated in relation to the qualitative methods discussed in this book. The ‘Quality’ Debate The debate in principle The very idea of judging the quality of qualitative research is contested in the literature. Some objections are made on philosophical grounds. For example, Smith (1984, 1990) has argued that the idealist and anti-foundational nature of qualitative research makes it impossible to assess quality in the sense of applying a set of formalized criteria. Other writers, however, argue that qualitative research is not based on a single or shared set of philosophical assumptions and therefore the idea of judging quality cannot be simply dismissed on philosophical grounds (Altheide & Johnson, 1994; Hammersley, 1992; Phillips, 1990)

  Sign up to read

  Learn more about book
• 

  eBook - ePub

  An Introduction to Qualitative Research

  • Uwe Flick(Author)
  • 2022(Publication Date)
  • SAGE Publications Ltd

    (Publisher)

  One starting point is to reflect upon the construction of social realities in the field under study and in the research process. The decisive question, however, is whose constructions were addressed and were successful in the process of knowledge production and in the formulation of the results – those of the researcher, or those met in the studied field? The problem of grounding qualitative research is made concrete with three questions: How far are the researchers’ findings based on constructions in the field? How are the translation and documentation of these constructions in the field made into texts, which are the empirical material? How did the researcher proceed from the case study to the developed theory or to the general patterns found? Grounding qualitative research becomes a question of analysing the research as process. After discussing the alternatives mentioned, the impression remains that both strategies – the application of traditional criteria and the development of alternative, specific criteria – have featured in recent discussions and that neither has yet given a wholly satisfactory answer to the problem of grounding qualitative research.

  The equation or connection of alternative and traditional criteria by Miles and Huberman (1994, p. 278) outlines an interesting perspective for structuring this field. They have structured the major traditional criteria together with the most prominent alternative criteria in the following five groups:

  • Objectivity/confirmability
  • Reliability/dependability/auditability
  • Internal validity/credibility/authenticity
  • External validity/transferability/fittingness
  • Utilization/application/action orientation.

  But at the same time, this equation makes clear that attempts to reformulate criteria for qualitative research did not really lead to new solutions. Rather, the problems with traditional criteria derived from different backgrounds have to be discussed in the case of alternative criteria, too.

  Quality Assessment as a Challenge for Qualitative Research

  The question of how to assess the quality of qualitative research currently arises in three respects. First, by the researchers who want to check and secure their proceedings and their results. Second, by the consumers of qualitative research, the readers of publications or the funding agencies, who want to assess what has been presented to them. And finally in the evaluation of research in reviewing research proposals and in peer reviews of manuscripts submitted to journals. In the last context, you will find a growing number of guidelines for evaluating research papers (articles, proposals, etc.). Seale (1999, pp. 189–92) presents a criteria catalogue of the British Sociological Association’s Medical Sociology Group, which includes a set of questions referring to 20 areas from research questions on sampling, collection and analysis of data, to presentation and ethics. The guiding questions are helpful, but when you answer them, you will find that you are drawn back to your own – maybe implicit – criteria: for example, when you want an answer in area 19 (‘Are the results credible and appropriate?’), the question ‘Do they address the research question(s)?’ (1999, p. 192) is suggested.

  Sign up to read

  Learn more about book