Using Statistics in Small-Scale Language Education Research

Focus on Non-Parametric Data

Jean L. Turner

• Publisher: Routledge
• Year: 2014
• ISBN:
  9781134055586
• Edition: 1
• Topic: Education
• Subtopic: Education General

Assuming no familiarity with statistical methods, this text for language education research methods and statistics courses provides detailed guidance and instruction on principles of designing, conducting, interpreting, reading, and evaluating statistical research done in classroom settings or with a small number of participants. While three different types of statistics are addressed (descriptive, parametric, non-parametric) the emphasis is on non-parametric statistics because they are appropriate when the number of participants is small and the conditions for use of parametric statistics are not satisfied. The emphasis on non-parametric statistics is unique and complements the growing interest among second and foreign language educators in doing statistical research in classrooms. Designed to help students and other language education researchers to identify and use analyses that are appropriate for their studies, taking into account the number of participants and the shape of the data distribution, the text includes sample studies to illustrate the important points in each chapter and exercises to promote understanding of the concepts and the development of practical research skills. Mathematical operations are explained in detail, and step-by-step illustrations in the use of R (a very powerful, online, freeware program) to perform all calculations are provided.

A Companion Website extends and enhances the text with PowerPoint presentations illustrating how to carry out calculations and use R; practice exercises with answer keys; data sets in Excel MS-DOS format; and quiz, midterm, and final problems with answer keys.

Section I
Foundations

1
What is Research?

What is research? When Professor Nataliya Borkovska was teaching technical English, she realized her students were challenged by the task of learning so much new terminology. She systematically tried out several different graphical approaches to presenting new vocabulary to see if they helped her students retain new technical vocabulary (Borkovska, 2007). Is her systematic trial of these techniques research, though it was done by a teacher in her own classroom, using techniques and data collection tools she had designed, with the primary purpose of gaining a deeper understanding of that learning environment? Professor Pablo Oliva teaches Spanish to graduate students who use Spanish in their workplaces. He wants to help them develop a habit of lifelong learning (Oliva, 2011) and provides feedback on their written assignments that he designed to promote their autonomy. He collected and analyzed questionnaire data to determine whether his students’ autonomy was enhanced by the special feedback. Can his investigation be considered research, though the only participants were his students and the questionnaire data he collected consisted of students’ self reports?

Professor Jennifer Grode wrestled with deciding whether to use language textbooks or authentic materials as the basis for her English language lessons. She had read a lot of expert opinion and theory but decided she really needed a sense of learners’ opinions. She designed an online questionnaire to collect language learners’ perceptions of the usefulness of these two types of materials and their degree of enjoyment of them (Grode, 2011). She used her findings to guide her decisions about when and how to use textbooks and authentic materials. Is her investigation research because she collected information from learners she didn’t know?

Language educators’ daily lives are filled with complex, professional decisions. Like Professors Borkovska, Oliva, and Grode, I believe that when educators make decisions about their pedagogical practices, the outcomes of those decisions are more useful when the decisions are based on knowledge of the students and the specific educational settings in which the decisions will be implemented. As McMillan (2000) writes, “[I]n fields such as education, where practice is heavily influenced by complex interactions among students, environments, and teachers, there is room for experts to disagree about what is known” (p. 3). In such complex environments, teachers who rely simply on what an authority has said or written, or even on their own past experiences, may not make the best decisions for their students. I believe that decision making should be informed by teachers’ deep knowledge of the contexts in which they practice their profession as well as by their familiarity with current theory and practice. I also believe that teachers’ decision making is best guided by a combination of careful study and thoughtful reflection in three areas: (1) their personal experiences and observations, (2) consideration of information from theory and experts or authorities, and (3) evidence gained from the systematic collection and analysis of information from specific learning environments.

The third of these perspectives is examined in this book; the systematic collection and analysis of information for the purposes of seeking answers to questions about specific learning environments, exploring the effectiveness of practice, or forming hypotheses and theories. This systematic process is research, though it doesn’t involve large numbers of randomly selected, anonymous participants and the primary goal is a deeper understanding of the local learning environment rather than drawing conclusions about learners in general. Nunan and Bailey (2009) and Burns (2010) noted that research planned and carried out by teachers in the educational contexts in which they work and having the primary goal of improving practice is becoming increasingly important as a basis for informed practice. The work of educators such as Professors Borkovska, Oliva, and Grode reflects this trend.

In this text, I focus on statistical research with the goal of assisting language education researchers and others in reading and understanding statistical research and in planning and performing research that involves the statistical analysis of quantitative information. I recognize that language education researchers often collect and analyze qualitative data too. However, in this text I address the analysis of quantitative data because many of the teachers I’ve worked with in my research methods classes have told me they’re not confident doing research that involves what they call “numbers.” They recognize that quantitative information is useful, but they’re unfamiliar with or uncomfortable reading statistical studies and doing math themselves, or they simply mistrust statistical research. If they previously took a statistics class, the focus was usually on formulas designed to be used in large-scale research, with the primary goal of applying the findings to people outside the study. This research requires anonymous participants and test scores distributed neatly in a bell-shaped curve; teachers may not see the relevance of this type of statistical analysis to their smaller, local investigations.

Though I address some of these statistics for large-scale research in this text, the emphasis is on non-parametric statistics, formulas that accommodate the features of small-scale language education research. These features include the relatively small number of participants and the fact that they’re not anonymous recruits from a vast population of learners. Non-parametric statistics are also appropriate for analyzing information from tests, questionnaires, and other data collection tools designed to measure students’ learning or opinions, regardless of whether the outcomes fall in a bell-shaped curve.

To help teacher/researchers who have concerns about math, I include step-by-step explanations of how to carry out the calculations and I illustrate how to use a powerful, free statistical software program called R. I appreciate the fact that it’s free, but what’s really great is that it’s powerful and flexible enough to be used for almost all of the calculations I discuss in this textbook. I like the program’s graphics capabilities too. (R can be used to make many different types of useful charts, graphs, and tables, though honestly, one feature I particularly like is the range of color options.)

Types of Research

One of the important defining characteristics of all types of research is systematicity. Some of the systematicity of statistical research is embodied in rules, of which there are two types. The first type is a set of procedural steps known as statistical logic, the platform for the probability statements that characterize statistical research. The second type of rules consists of mathematical operations embedded in formulas. Before studying either the rules of statistical logic or the mathematical rules expressed in statistical formulas, it’s important to have a general understanding of what statistical research is, what it’s not, and how statistical research differs from other types of research, because there are a lot of adjectives used with the term research. Some of these adjectives indicate the philosophy that underlies the research approach. Others designate the venue for the research, who does the research, or who might be included as participants. Some of the adjectives highlight the purpose of the research or the tools or techniques used to conduct the research or data analysis. Others indicate the types of models that are used in performing analyses and interpreting the results.

Consider the term action research. Bailey (2005) describes action research as systematic inquiry with “a reiterated cycle of procedures… taken in an attempt to improve a situation” (p. 25), so the term indicates the purpose of a research study. Burns (2010) notes that action research involves a teacher’s “self-reflective, critical, and systematic approach” to identifying and exploring an issue in his or her educational context (p. 2). So another characteristic of action research is who does it—practitioners of a profession rather than people who are trained primarily as researchers. In language education, action research is typically done by teachers, typically to investigate the impact of an innovation or change in their usual practice on students’ learning or well-being. Professor Oliva’s research can be categorized as action research for several reasons. First, he himself identified an issue that he considers relevant and important—how features of the feedback he provides his students might impact their autonomy as learners and long-term interest in and commitment to studying Spanish. Second, he designed and carried out the research, though he’d probably tell you that he’s not trained primarily as a statistician or researcher. Third, he made changes in his practice both as part of his investigation and as a result of his investigation. Finally, the goal of his investigation was to enhance his practice.

Like the term action research, the terms exploratory research and confirmatory research indicate the purpose of the research. People who do exploratory research conduct their studies to gain a deeper or broader understanding of the phenomena present in a particular learning environment or to form hypotheses. Confirmatory research is conducted to collect evidence to support a theory or hypothesis. Unlike the adjective action, the terms exploratory and confirmatory give no indication of who the researchers are, but much of the research in language education has features of exploratory or confirmatory investigation. I believe that both Professor Borkovska’s and Professor Oliva’s research can be considered exploratory—they conducted their studies to gain a deeper understanding of factors that might contribute to their students’ learning.

Survey research, defined by Johnson (1992) as research done “to learn about characteristics of an entire group of interest (a population) by examining a subset of that group (a sample)” (p. 104), is a term that indicates the type of instrument used to collect information from the participants, a survey. The term survey research tells us something about the tool used to collect informa...

Table of contents

• Cover
• Title
• Copyright
• CONTENTS
• Preface
• Acknowledgments
• Section I Foundations
• Section II Analyzing Differences Between Two Sets of Data
• Section III Analyzing Differences Among More Than Two Sets of Data
• Section IV Analyzing Patterns Within a Variable and Between Two Variables
• References
• Index