Text Analysis for the Social Sciences

Methods for Drawing Statistical Inferences From Texts and Transcripts

Carl W. Roberts

• Publisher: Routledge
• Year: 2020
• ISBN:
  9781000149241
• Edition: 1
• Topic: Languages & Linguistics
• Subtopic: Linguistics
• Index: Languages & Linguistics

This book provides descriptions and illustrations of cutting-edge text analysis methods for communication and marketing research; cultural, historical-comparative, and event analysis; curriculum evaluation; psychological diagnosis; language development research; and for any research in which statistical inferences are drawn from samples of texts. Although the book is accessible to readers having no experience with content analysis, the text analysis expert will find substantial new material in its pages. In particular, this collection describes developments in semantic and network text analysis methodologies that heretofore have been accessible only among a smattering of methodology journals. The book's international and cross-disciplinary content illustrates the breadth of quantitative text analysis applications. These applications demonstrate the methods' utility for international research, as well as for practitioners from the fields of sociology, political science, journalism/communication, computer science, marketing, education, and English. This is an "ecumenical" collection that contains applications not only of the most recent semantic and network text analysis methods, but also of the more traditional thematic method of text analysis. In fact, it is originally with this volume that these two "relational" approaches to text analysis are defined and contrasted with more traditional "thematic" text analysis methods. The emphasis here is on application. The book's chapters provide guidance regarding the sorts of inferences that each method affords, and up-to-date descriptions of the human and technological resources required to apply the methods. Its purpose is as a resource for making quantitative text analysis methods more accessible to social science researchers.

CHAPTER
1

A Matter of Definition

Gilbert Shapiro
John Markoff
University of Pittsburgh

Researchers with the most diverse purposes have examined texts, recorded codes denoting some characteristic of those texts and then used a collection of such codes as data. Definitions of content analysis have tended to stress some of these purposes or some possible methods of accomplishing such purposes and have excluded others. Although there are significant differences among research projects, there are some significant common elements as well. In this chapter we offer a minimal definition of content analysis that covers a wide range of concrete research agendas. We also show how differences among standard definitions reflect central methodological issues that need to be addressed, rather than defined out of existence.

The Need For Definition

Normally, we hold that few strategies can so confuse a discussion as can an author’s announcement of his sincere determination to use a common term in some clearly delimited manner. Nevertheless, a consideration of the boundaries of content analysis seems inescapable; at least, we find such discussions very frequently in the methodological literature. This need to define the term, we believe, stems from two opposite difficulties. At times, the explicit definition of the term, or the definition implicit in its use, can be so inclusive as to encompass any reading of text. At other times, explicit or implicit definitions restrict a term’s application to a very small portion of the research activities that have been so labeled in the past. The boundaries of content analysis are problematic; hence we, like many other writers on the subject, find it necessary to attempt to delineate the range of research activities to which we believe it should apply.

Such broad categories of procedures are encompassed within some definitions and, more important, some patterns of usage of the term, that it is questionable that any common methodological principles can apply to them all. There are authors who seem to embrace literally any reading of text, albeit the most conventional of literary work. One example is Lerner’s (1950, fn. 1, p. 212) description of his polemical answer to critics of the World War II research of the War Department embodied in The American Soldier as “a preliminary sketch for a more detailed thematic content analysis of responses to The Soldier and other significant books.” Holsti (1969, p. 36) saw this article, which reads much like any other piece of controversial scholarly rhetoric, as an example of a content analysis. In such cases, we do not find any methodical marshaling of evidence, any explicit account of procedures for the selection of documents or sampling, or any indication of the invariable application of coding categories or tests of reliability. What we typically find is an extensive discussion of conclusions regarding the contents of some vaguely defined collection of documents, backed up by occasional citations; the systematic laying bare of methodology, warts and all, which we take to be characteristic of scientific work, is not seriously attempted.

Of course, there are the inevitable borderline cases. Davis (1938), at least, provided us with a list of the sources he studied to arrive at his description of the ideology of the mental hygiene movement. He also declares that these works were “gone through with the aid of a fixed questionnaire designed to discover certain things about each book” (p. 57), although we are not granted the privilege of reading the questionnaire, nor are we told the frequency with which he found each of its categories. Again, in his analysis of the American social pathologists’ ideology, Mills (1942) generalized over the content of roughly 25 textbooks, of which he provided a list. He made no explicit claim, as did Davis, that he addressed identical queries to each text, but, in the time-honored fashion of literary or journalistic study, presented his general impressions of the contents speckled with occasional quotations.¹ We have no argument against the value of such studies but only against the claim that they are somehow content analysis² and hence partake of the authority of science if performed by a professor in one department, whereas they are mere literary work if performed by a professor in another department or by a writer outside of the academy.

¹‘As usual, Mills knew exactly what he was doing: “no one of the texts to be quoted exemplifies all the concepts analyzed; certain elements are not so visible in given texts as in others, and some elements are not evidenced in certain texts at all” (Mills, 1942, fn. 1, p. 165). “The quotations in the footnotes are merely indications of what is usual. The imputations presented must be held against the reader’s total experience with the literature under review” (Mills, 1942, fn. 3, p. 166). Fair enough; we have no objections to the way Mills conceived of his work, nor, for that matter, to how he performed it. Note that Mills acknowledged that to properly assess his imputations, the reader must read all the texts that Mills read!

²Berelson (1971, p. 218) listed the Mills study as an instance of qualitative content analysis.

We believe that the designation of any work as content analysis as distinguished from the ordinary reading of text makes an implicit claim that the researcher is participating in a scientific enterprise. Such a claim must be based on something on the order of systematic or methodical procedures, and we would reserve the term content analysis for scientific work so conceived. At a minimum, there must be some rules (as opposed to feelings or intuitions) determining the text that is to be studied and some standard set of coding decisions that are to be applied to all of that text. The completion of this thought, specifying what, beyond the minimal requirement of systematic procedures, must be included among the defining criteria of content analysis, takes diverse forms for different authors, and much of this chapter is devoted to mapping the alternatives.

Representative Definitions

The following six quotations seem to us to provide a fair indication of the range of explicit definitions of content analysis in the methodological literature (cf. Berelson, 1971, pp. 14-18; and Holsti, 1969, pp. 2-3):

Bernard Berelson: “Content analysis is a research technique for the objective, systematic, and quantitative description of the manifest content of communication” (Berelson, 1954, p. 489).³

Dorwin Cartwright “In the subsequent discussion, we propose to use the terms ‘content analysis’ and ‘coding’ interchangeably, to refer to the objective, systematic, and quantitative description of any symbolic behavior” (Cartwright, 1953, p. 424).⁴

Irving L. Janis: “ ‘Content analysis’ may be defined as referring to any technique a) for the classification of the sign-vehicles, b) which relies solely upon the judgments (which, theoretically, may range from perceptual discrimination to sheer guesses) of an analyst or group of analysts as to which sign-vehicles fall into which categories, c) on the basis of explicitly formulated rules, d) provided that the analyst’s judgments are regarded as the reports of a scientific observer” (Janis, 1949, p. 55 [emphases in original]).

³This definition is identical with that in Berelson (1971, p. 18), but neither the restriction to the quantitative nor the restriction to the manifest is sustained for the rest of the book.

⁴Although this particular definition best fits our polemical needs, we find Cartwright’s other formulation (1953, p. 421) more stimulating philosophically, in that he regards text or “qualitative material” as “phenomena” which, by “coding,” are “converted into scientific data.”

Klaus Krippendorff: “These observations suggest that content analysis be restricted to the use of replicable and valid methods for making specific inferences from text to other states or properties of its source” (Krippendorff, 1969, p. 70).⁵

Charles E. Osgood. “If we define content analysis as a procedure whereby one makes inferences about sources and receivers from evidence in the messages they exchange, then the problem falls precisely in the field of special interest of the psycholinguist” (Osgood, 1959, p. 35).

Philip J. Stone: “Content analysis is any research technique for making inferences by systematically and objectively identifying specified characteristics within text” (Stone, Dunphy, Smith, & Ogilvie, 1966, p. 5).⁶

The variety in these definitions is striking.⁷ We find the following issues raised by their similarities as well as their differences:⁸

What classes of symbolic objects are studied in content analysis?
Text Only	Text and Other Symbolic Material
Krippendorff Stone	Berelson (“communication”) Cartwright (“symbolic behavior”) Janis (“sign-vehicles”) Osgood (“messages”)

What kind of intellectual product comes from a content analysis?
Description	Inference	Classification
Berelson	Krippendorff	Janis
Cartwright	Osgood
	Stone

⁵Krippendorff’s later formulation is more ambiguous: “a research technique for making replicable and valid inferences from data to their context” (1980, p. 21). This formulation does not specify the kind of data, nor the meaning of the context, so we stay with the earlier statement in further discussion.

⁶This definition was repeated almost identically by Holsti (1969, p. 14). Holsti extended the possibility of content analysis beyond text to all kinds of messages, but, unlike Stone, he was not engaged in the development of a computer system for the content analysis of text alone.

⁷Recent literature on content analysis is scarce and adds no new elements to the range of definitions presented here. See, for example, Weber (1990).

⁸Some remarks on this (content?) analysis seem to be in order. We have relied exclusively on each author’s explicit definition in arriving at our characterization of his positions. If we had paid attention also to his other text (viz., to the context of his definition), as well as, perhaps, to his other writings (especially to his actual content analyses) we would undou...

Table of contents

• Cover
• Half Title
• Title Page
• Copyright Page
• Dedication
• Table of Contents
• Introduction
• 1 A Matter of Definition
• PART I: METHODOLOGIES
• PART II: APPLICATIONS
• PART III: PROSPECTS
• References
• Author Index
• Subject Index