Reading Empirical Research Studies

The Rhetoric of Research

John R. Hayes, Richard E. Young, Michele L. Matchett, Maggie McCaffrey, Cynthia Cochran

• Publisher: Routledge
• Year: 2020
• ISBN:
  9781135441210
• Edition: 1
• Topic: Bildung
• Subtopic: Bildung Allgemein

For the most part, those who teach writing and administer writing programs do not conduct research on writing. Perhaps more significantly, they do not often read the research done by others because effective reading of articles on empirical research requires special knowledge and abilities. By and large, those responsible for maintaining and improving writing instruction cannot -- without further training -- access work that could help them carry out their responsibilities more effectively. This book is designed as a text in graduate programs that offer instruction in rhetoric and composition. Its primary educational purposes are: * to provide models and critical methods designed to improve the reading of scientific discourse * to provide models of effective research designs and projects appropriate to those learning to do empirical research in rhetoric. Aiming to cultivate new attitudes toward empirical research, this volume encourages an appreciation of the rhetorical tradition that informs the production and critical reading of empirical studies. The book should also reinforce a slowly growing realization in English studies that empirical methods are not inherently alien to the humanities, rather that methods extend the power of humanist researchers trying to solve the problems of their discipline.

Part 1

Chapter 1 Introduction to Empirical Research and Rhetoric: Reading in a Developing Tradition

Some months ago, one of the writers of this report mentioned to a colleague doing research in internal medicine that it was disappointing to see how little was really known about the teaching and learning of written composition, how inconclusive most of the research has been. The colleague replied that 95 percent of the research in his area was inconclusive or trivial. “Keep at it,” he said. “As you learn more, you’ll slowly learn to define your problems in a useful manner and to refine your techniques of analysis. Then you’ll be in a position to learn something substantial.”

Richard Braddock, Richard Lloyd-Jones, and Lowell Schoer,
Research in Written Composition (1963, p. 29)

Not only have we begun to develop a research tradition, but attempts to answer questions raised by Braddock, Lloyd-Jones and Schoer have led to new breadth and depth of interest in the teaching of writing.

George Hillocks, Jr., Research on Written Composition: New
Directions for Teaching (1986, xvii)

Neat Studies

As we worked out the conception of this collection and sifted articles for inclusion, we fell into the habit of calling the book “Neat Studies.” If somewhat facetious, our working title was also reasonably accurate. The studies all reveal scholars actively trying to come to grips with real problems in the discipline; the studies are all intelligently designed, carried out, and described; and they all tell us something that is new and useful. These characteristics were, in fact, the criteria we used in selecting the studies.

“Neat,” though, does not mean perfect. To drive home this point, we have prefaced each of the studies with a short critique pointing out not only some of the strengths we see in the studies but weaknesses as well. Furthermore, the authors of the articles often criticize their own work in the brief essays that follow each of their works. None of the articles, then, is beyond criticism. Nor, we insist, need they be in order to have value. If perfection were a precondition for usefulness, it’s unlikely that scholars would get very far in empirical research in rhetoric and composition, no matter what kind of method they were using. Perfection, in the sense of faultless work, is not a standard that is brought to bear in other areas of rhetorical scholarship, or for that matter in the scholarship in any other discipline. As we have all doubtless observed in so many areas of our experience, the insistence on perfection can have the effect of paralyzing action when action is needed. To put it another way, the best, which is often unobtainable, can be the enemy of the more obtainable good. Indeed, logicians (e.g., Fearnside, 1959, p. 131-2) recognize the call for perfection (that is, the establishment of impossible conditions as criteria for judgment) as an argumentative fallacy that can stifle constructive argument and consequent productive action. Whatever their shortcomings, all the articles in this collection are exemplary, in that they illustrate what it means to address genuine problems in the discipline of rhetoric with care and intelligence, and they set a standard of competence for those learning the methods of empirical research.

Empirical Research in Rhetoric

One of the most significant changes among the many significant changes in rhetorical studies over the last three decades is the growth of empirical research in rhetoric and composition. (It should be noted that Speech Communications has had a longer tradition of empirical research. Our comments here for the most part pertain to efforts by those working in English studies.) What has to be regarded as a revolution in humanistic research is readily apparent if Research in Written Composition, the 1963 study by Braddock, Lloyd-Jones and Schoer, is compared with George Hillocks’ 1986 study, which bears nearly the same title. Both survey prior empirical research in rhetoric: Braddock, Lloyd-Jones, and Schoer sought to survey all scientific studies of written composition done in this country up to 1962; Hillocks, surveying a much shorter span of time, investigated scientific studies done between 1963 and 1982. Out of an original list of over 1,000 titles, Braddock and his colleagues identified approximately 500 studies that could be regarded as examples of scientific research, about half of which were unpublished dissertations. They also selected five studies for close scrutiny that were in their opinion not perfect but the “most soundly based of all those studies available” (1963, p. 55). However, as they say, they were tempted not to select any study because of a general failure to recognize the complexities of scientific research design (p. 55).

In contrast, Hillocks originally identified over 6,000 titles, which were then screened using a much more rigorous set of criteria than was used in the earlier study; the result was a list of over 2,000 titles. Hillocks (1986, xvi) remarks that Braddock, Lloyd-Jones, and Schoer felt that few studies were exemplary:

“It is an unusual study,” they wrote, “which does not leave several important variables uncontrolled or undescribed.. .” While many studies included in the bibliography of this report suffer from similar flaws, there are also other studies which are, I believe, exemplary and which contribute to our knowledge of composition.

In 1963, Braddock and his colleagues likened empirical research on writing to “chemical research as it emerged from the period of alchemy” (p. 5). Whatever the shortcomings of present-day research, it has gone far beyond the rudimentary empiricism implied in their analogy. And in our comments on what it means to read empirical reports critically we acknowledge this by invoking not a special, and forgiving, set of criteria designed for applications of scientific methodology in rhetorical research (“science for humanists,” so to speak), but the same criteria that are generally used in the social sciences. One point of this brief comparison between the Hillocks study and its predecessor, then, is that in the twenty-five years between the two works both the number and quality of studies making use of scientific methods have increased remarkably.

Furthermore, as Lloyd-Jones notes in the “Introduction” to Hillocks’ study (1986, xiii), “the methods are also far more varied.” In the 1963 survey, error counts and comparative studies dominated the book so much that without them there would have been no review. Hillocks describes other procedures for systematically refining observations. He is as governed by empiricism as are the earlier reviewers. However, the practices he describes lead in more directions by many different routes.

Many methods that have now become commonplace in our journals were rare in rhetorical research just a generation ago. The significance of this development is that those interested in rhetorical research are in a much better position now than they were earlier to match appropriate methods of research to the questions they want to investigate. Our methodological toolbox has gotten much larger. In this collection of articles, we have provided a sampling of the methods now commonly used in rhetorical research: case studies, naturalistic observation, surveys, protocol studies, correlational studies, experiments, historical studies. We might add that the questions asked by our authors and the settings in which the questions are investigated are also increasingly varied: the investigations take place not only in primary, secondary, and college classrooms—the principal sites of earlier studies—but in rural black homes, inner city ghettos, and Japanese electronic firms.

Research and the Profession of English

One of the many paradoxes in the paradoxical tradition of writing instruction in departments of English is that much of the early research, whether empirical or otherwise, was carried out not in English departments but in schools of education, often by educational psychologists. That is, the teaching of writing, at least at the college level, was severed from the research on writing and on writing pedagogy. And those who did the teaching seldom were acquainted with the research, and, we might add, the researchers. However, as Anne Herrington points out (1989), the community of researchers today is growing and becoming more diverse. Research in the Teaching of English had a subscription of 1,171 at the time of its founding by the NCTE in 1967; ten years later the subscription had more than doubled to 2,834. In addition, those publishing in the journal are much more representative of those working in English studies. Indeed, a growing community of empirical researchers is discernible within the much larger community of scholar-teachers in English.

Such changes are encouraging to anyone interested in basing educational practices and policy on the best available knowledge. However, as encouraging as these changes are, the old division persists between, on the one hand, those who do the research on rhetoric and composition and, on the other, those who teach writing and those who make educational policy in English departments. (See, for example, Hartzog, pp. 14-71.) For the most part, those who teach writing and who administer writing programs do not do research on writing. Perhaps more significantly, they do not read the research done by others. One reason for this is that effective reading of articles on empirical research requires special knowledge and abilities that graduate training in English studies has in the past seldom sought to develop, though that tradition is changing in some schools around the country. By and large those responsible for maintaining and improving writing instruction in this country cannot, without further training, access the work that could help them carry out their responsibilities better.

Hence this book, which is designed primarily to serve as a text in graduate English programs that offer instruction in rhetoric and composition. The book has two basic purposes:

1. to provide models and critical methods designed to improve the reading of scientific discourse, and
2. to provide models of effective research designs and projects appropriate to those learning to do empirical research in rhetoric.

But the book is designed to serve other purposes as well. We want to demonstrate that one of the myths about empirical research commonly shared among humanists is without substance, for if you believe the myth, you will never make the effort to understand this kind of research. The myth is that empirical research never tells us anything new about rhetorical behavior and about the teaching of composition, that is, anything that isn’t already known by any reasonably perceptive layman. Another variant of the myth is that the findings of empirical research may be true but they are trivial. To see that the belief is problematic, consider the widely shared assumption that the longer one works on a task the greater one’s achievement will be. It is clearly not a trivial belief, but most would probably regard it as a truism, certainly not a claim requiring formal investigation. However, in one of the earliest studies in the history of teaching research, J. M. Rice found that there was no significant correlation between the amount of time devoted to spelling homework and achievement in spelling. After studying tests on 33,000 school children, Rice noted that “an increase in time... is not rewarded by better results.... The results obtained by forty or fifty minutes’ daily instruction were not better than those obtained where not more than ten or fifteen minutes had been devoted to the subject” (quoted in Gage, p. 12). What seem to be truisms sometimes turn out not to be. But, as N. L. Gage remarks (1991, p. 13), “even if the broad generalization is a truism, the specifics of its actualization in human affairs—to determine the magnitude of the probability and the factors that affect that magnitude—require research.” We think that an open-minded reading of the studies in this collection will lay the myth to rest.

The book will also, we hope, cultivate new attitudes toward empirical research: for example, we hope that it will encourage in the reader an appreciation of the rhetorical tradition that informs the production and critical reading of empirical studies. It is a tradition that has received little scholarly attention in rhetorical studies until recently. However, once one understands that the scientist does not seek to prove beyond the possibility of objection the truth of a particular claim but instead seeks to make persuasive arguments to a particular disciplinary community, the whole enterprise has to be examined in a different light. The scientist is to be seen now as constructing arguments, within a reasonably well-defined system of rhetorical constraints, that a particular belief is warranted. That is, the scientist is to be seen as a practicing rhetorician. That change in conception of the business of the scientist opens up a whole new area of rhetorical inquiry.

One last comment on the purposes of the book: we hope that the book serves to reinforce a slowly growing realization among those working in English studies that empirical methods are not inherently alien to the humanities, rather that the methods extend the power of humanist researchers trying to solve the problems of their discipline. The methods we explore in this collection are not merely various ways of collecting data; they represent ways of reasoning about problems we confront as scholars and teachers. In particular, they represent sophisticated ways of reasoning from observations to defensible conclusions and, more broadly, of reasoning about observation as a source of knowledge.

The Organization of the Book

The organization of the book also needs comment since it is a bit unconventional. Chapter 2 is an introduction to an art of reading scientific articles. Although the effective reading of scientific discourse is not a simple and straightforward intellectual activity, some may nevertheless think it strange to speak of an art of criticism in this context, since arts of criticism have traditionally been restricted to what is commonly referred to as “imaginative discourse,” that is, fictional discourse or discourse for esthetic purposes. Some of the skills associated with the effective reading of imaginative discourse are, in fact, applicable in the reading of scientific discourse. Nevertheless, the reading of scientific discourse entails more than a new application of abilities we may already have acquired in the study of literary texts. Effective readers have special skills, and these skills and their underlying principles can be taught. They are probably best taught through repeated use in the close reading of diverse texts, an approach this collection is designed to encourage.

Preceding each of the articles is a short critical analysis by the editors. The analyses are intended to help the reader see relationships among the articles, limitations and potentials of conventional designs and methods, and possible alternatives to the interpretation of the data discussed in the study. The analyses are also intended to illustrate various strategies for critical reading. In short they provide examples of the kind of analysis that a skillful reader might engage in when presented with the article.

Each of the articles in the collection is followed by a short statement by the author describing how the article came about, that is, what set the inquiry in motion. These informal autobiographical accounts serve, we hope, to make the process of scientific research more accessible to the reader, who can see that it begins where all inquiry begins, scientific or otherwise, in a moment of puzzlement, or curiosity, or need to know; more generally it begins, not in some impulse peculiar to th...

Table of contents

• Cover
• Half Title
• Title Page
• Copyright Page
• Table of Contents
• Preface
• Acknowledgements
• Part 1
• Part 2
• Part 3
• Glossary
• Additional Readings