Evaluating Scholarship and Research Impact

Jeffrey W. Alstete, Nicholas J. Beutell, John P. Meyer

• Publisher: Emerald Publishing Limited
• Year: 2018
• ISBN:
  9781787563896
• Topic: Education
• Subtopic: Higher Education

Faculty members, scholars, and researchers often ask where they should publish their work; which outlets are most suitable to showcase their research? Which journals should they publish in to ensure their work is read and cited? How can the impact of their scholarly output be maximized?
The answers to these and related questions affect not only individual scholars, but also academic and research institution stakeholders who are under constant pressure to create and implement organizational policies, evaluation measures and reward systems that encourage quality, high impact research from their members. The explosion of academic research in recent years, along with advances in information technology, has given rise to omnipresent and increasingly important scholarly metrics. These measures need to be assessed and used carefully, however, as their widespread availability often tempts users to jump to improper conclusions without considering several caveats. While various quantitative tools enable the ranking, evaluating, categorizing, and comparing of journals and articles, metrics such as author or article citation counts, journal impact factors, and related measures of institutional research output are somewhat inconsistent with traditional goals and objectives of higher education research and scholarly academic endeavors.
This book provides guidance to individual researchers, research organizations, and academic institutions as they grapple with rapidly developing issues surrounding scholarly metrics and their potential value to both policy-makers, as evaluation and measurement tools, and individual scholars, as a way to identify colleagues for potential collaboration, promote their position as public intellectuals, and support intellectual community engagement.

CHAPTER 1

A PRIMER ON POLICY AND RESEARCH

Policies are an institution’s way to guide decision-making and conduct operational practices. Proper development and implementation of effective policies is especially challenging in times of rapid technological development, changing societal expectations, and other turbulent internal and external environmental forces. Different types of institutions implement policies that reflect varying levels of importance on learning activities, research endeavors, and service commitments. Historically, colleges and universities educated a small, elite portion of the population using a specified curriculum. Other endeavors such as faculty research and service activities had a very limited role until relatively recently (Rudolph, 1990; Thelin, 2004; Veysey, 1965). The ancient Greek education system involved students researching their own information and debating it, while instructors directed students to develop critical-thinking skills. Many people today see career preparation as the primary or only goal of higher education, while various external and governmental agencies see research as the fundamental purpose (Altbach et al., 1999).

There have been, and still are, vastly different perspectives on developing clear foundations for policies in higher education (Brubacher, 1965; Christensen and Eyring, 2011; Zemsky et al., 2005). For those who understand that crafting and executing organizational strategy is an important priority, it is often stated or implied that internal and external policies should support postsecondary institutions in the aforementioned vocational endeavors as well as practical or applied research, without question, even by people in academe. It is interesting to note that concerns and conflicts about the appropriateness of policies to support this perspective have been going on for well over a century, from a time when only a small portion of the population attended college at all, and the exclusive institutions serving this elite group largely followed the English model to provide broad liberal education.

Classic literature such as the essays by Cardinal Newman in the Idea of a University (first published in 1852) praised this model of pursuing knowledge for its own sake and uplifting the soul (Newman, 1947). However, the early nineteenth-century Germanic models with a strong research emphasis also made headway into the realm. Other visible higher education scholars such as Veblen and Flexner noted that important changes were occurring (Flexner, 1930; Veblen, 1918). These leaders and writers urged that American universities end obligations to extraneous pursuits, such as service to the community and vocational education. In their view, postsecondary education should be devoted exclusively to the pursuit of knowledge and research endeavors.

In The Academic Revolution (Jencks and Reisman, 1968), the authors boldly identified the increasing prominence of professional scholars and scientists in top universities and examined some of the revolution’s results. They perceived the changes as intensifying generational and class conflicts, as well as delicately transforming the types of pursuits to which capable people aspire, while unfortunately contributing to the decline of entrepreneurship and the rise of professionalism. They concluded that mass higher education, which has now become universal (Trow, 1973), despite its benefits, has had no considerable influence on the amount of social progress or equality in society. Jencks and Riesman believed that the revolutionary movement into academic professionalism was an advance over nineteenth-century higher learning, yet warned of its hazards and drawbacks such as the elitism and haughtiness inherent in meritocracy. Further, they argued against the shortsightedness that originates from a harshly academic assessment of human experience and understanding, as well as the complacency that may present methodological capability as an end rather than a means. This thought echoed Newman’s earlier beliefs that knowledge and research endeavors have an end in themselves, the pursuit of which should seek to make better people who love learning.

There were counter-revolutionaries as well (Harris, 1970). Notable critics of twentieth-century higher education such as Irving Babbitt, Albert Jay Nock, Abraham Flexner, Robert Maynard Hutchings, and Alexander Meikeljohn “opposed the new general and professional education; they disliked research of operational utility; and they believed education through extension services (is) wrong” (p. 15). These individuals had a common belief that practicality, whether in regard to teaching, research, or service, had no proper place in institutions of higher learning. The rightful purpose of the collegiate instruction should be studying for character development. Research activities should not focus on responding to immediate needs of society. Universities were seen as instruments for promoting the general welfare of the nation through:

• The conservation of knowledge.
• The interpretation of knowledge and ideas.
• The search for truth.
• The training of students who will appreciate knowledge and become the scholars of tomorrow (Flexner, 1911).

If these counter-revolutionaries and like-minded thinkers from that era could see the current state of colleges and university strategies, funding, philosophy, and faculty scholarship measurements, it is doubtful they would recognize the institutions and their stated missions.

Nevertheless, the multiple purposes of institutions continued to evolve and grow. The developments were aptly labeled as a creation of the “multiversity” that combines liberal general education, research, as well as a variety of services to society (Kerr, 1963). All of these different models exist today, continuing to confound internal and external policy-makers. A somewhat less familiar yet prescient book on Bases for Policy in Higher Education examines the various expectations of higher education institutions and states that there are philosophical, practical, and policy considerations that should be noted (Brubacher, 1965). Brubacher further proposed that higher education policies must first question whom should we seek to serve and what is our purpose? Should we seek to educate citizens to be knowledgeable members of an enlightened republic? Or merely provide a “consumatory good” (p. 18) that is something to be used up or appreciated by itself? Or offer something that provides material wealth and goods as a resultant benefit?

Therefore, the ensuing development in higher education policies, from teaching general education to increased specialization, combined with rapid enrollment increases, and greater emphasis on research in the post-Sputnik era have greatly complicated these matters because of the different institutional histories and varied opportunities that persist today. The three primary elements of higher education strategies identified by Brubacher are fundamentally philosophical, practical, and policy-related. Therefore, questions arise as to how disparate institutional goals and policy development schemas can be reconciled for modern higher education with very diverse institutional types. Is measurement of research scholarship activities appropriate? Can research output be accurately measured across a wide array of disciplines? Can scholarly metrics be used to guide policy development?

The purpose of this monograph is to conduct a comprehensive analysis of scholarly metrics, research impact, and research evaluation. Strategies will be proposed to support, acknowledge, and encourage scholarship for different academic disciplines and types of institutions. In order to achieve these goals, the literature on bibliometrics, citation analysis, research impact, and policy development will be reviewed guided by the tenets of qualitative research methodology (Miles and Huberman, 1994). This involves a few broad constructs that incorporate a large number of particulars that are categorized in bins of discrete perspectives, events, and actions, overlaid with narrative framework to study the key ideas, paradigms, and variables. In this work, the bins or categories will be subsequent chapters that first examine the history of scholarly metrics and research impact, criticisms of metrics and scholarly impact practices, benefits of using quantitative and qualitative evaluation methodologies, and finally, concluding with policy recommendations for institution leaders to consider.

It is commonly believed that the purpose of conducting research in colleges and universities is the generation, processing, preservation, and dissemination of knowledge (Allen, 1988; Bess and Dee, 2010; Bok, 2006; Bonewits and Soley, 2004; Boyer, 1990; R. F. Fisher, 2009; Geiger, 2004; Gibbons, 2003; Humbolt, 1970; Lipset, 1994; Neave, 2006; Newman, 1947; Rowley, 1999; Tuckman and Hageman, 1976; UNESCO, 2006; Whitehead, 1929). The pursuit of truth and intellectual honesty has also been identified (Turk, 2000) as one of the goals in seeking to create at research universities a metaphorical “city of intellect” (Brint, 2002). Yet, this goal is challenged by some criticisms regarding intellectual quality and the often counteracting pressures for practicality (Collins, 2002). However, even these criticisms have been theoretically reconciled by some thinkers, such as Fisher (2009, 2010) who states that “the primary research purpose is to enhance and extend the core college mission by enriching the student experience and the quality of college graduates, keeping faculty current and engaged, and contributing to the social and economic communities that colleges serve” (R. Fisher, 2009; Fisher, 2010; R. F. Fisher, 2009, p. iii). This “research-for-student-skill-development” concept is both old and new. It may be seen as a desire to prepare a new generation of researchers and innovators, as well as prepare students for participation in a knowledge-based economy and society. Equally or perhaps more importantly is examining and understanding the tremendous increase in the number of scholarly journals and the spread of the research culture in academe that have contributed to the wider usage of research evaluation methods.

FORMS OF RESEARCH AND MEASURES

Medieval universities were institutions that largely focused not only on operational utility by preserving and teaching classical knowledge, but also functioned as professional schools for clergy, medical doctors, and lawyers (Rashdall, 2012; Rudolph, 1977; Veysey, 1965). Universities began to assume an important role in the latest scientific and social knowledge in early nineteenth-century Germany (Fallon, 1980). German universities started to require that all faculty conduct research as well as teach in order to deliver the latest knowledge to students. The research university model was then emulated by some institutions in the United States and elsewhere, either initiated in this model or replacing and overlapping with the traditional English collegiate model into the various types of institutions operating today (Kerr, 1963). A fundamental and ongoing concern is the differing opinions on the proper goals of research, types of research that are appropriate, and whether such research should be done for practical utility to respond to immediate societal needs or the broader expansion of knowledge in seeking truth. Faculty and institutions were criticized by factions on both sides of the issue and continued with the previously mentioned revolutionary and counter-revolutionary ideas as higher education further evolved in the twentieth century.

Therefore, it is important to understand that the changing perspectives on research policy have a long history that might be overshadowed by the extensive scrutiny of faculty scholarship today. The entire higher education system, from student admission policies, to hiring practices, to graduate success, and other aspects are continually assessed by internal and external forces. Various conceptual frameworks for theories in higher education research and overall expectations have been identified and generally include:

• Outcomes or the products and results of the activities of institutions.
• Institutions or the structures that perform higher educational activities.
• Goals or the purposes, intentions, and objectives of higher education.
• Activities of institutions of higher education.
• The people or the individuals and groups of individuals involved in the activities of higher education.
• Activities or the characteristic goal-seeking functions of people in institutions of higher education.
• Environments and the settings wherein institutions of higher education pursue their goals (Williams, 1973).

These ideas are still timely for the purpose of exploring research measurement activities, systems, problems, and suggestions for effective policy development. It has long been understood that colleges and universities are places that pursue goals leading to the creation of new knowledge for its own sake as well as the betterment of society at large. This is particularly true in the sciences, economics, and social sciences, as well as the arts and humanities. People in organizations are logically grouped or self-organize into these various disciplines of study where research activity is conducted and around which the public recognition originated (Neave, 2002). The traditional structure of basic or pure curiosity-driven scholarship represents the production of knowledge in the framework of academic preferences for fixed hierarchical structures represented by the departments we commonly see today (Allen, 1988; Boyer, 1990; Dewey, 1938; R. Fisher, 2009; Kaplan, 1964; Lipset, 1994; Neave, 2002; Rowley, 1999; Tuckman and Hageman, 1976; Whitehead, 1929).

These structures provide guidelines about what the important research problems are; they also create a social dimension for knowledge sharing. As the literature evolved, alternatives to traditionally recognized forms of research were identified. Scholars such as Boyer proposed a well-known typology of four priorities that seek to bring additional authenticity to the full range of academic work. This more inclusive framework specifies the following forms of scholarship:

1. The scholarship of discovery. This includes basic and subject area/discipline-based research, as well as a commitment to knowledge for its own sake.
2. The scholarship of teaching. This is central to the mission of colleges, renews and revitalizes institutions, while also identifying scholars as learners.
3. The scholarship of application. Involves applied research that is related to the larger society outside academe, where theory and practice connect.
4. The scholarship of integration. This makes connections across the traditional academic research disciplines; therefore, it is interdisciplinary, integrative, and interpretive (Boyer, 1990).

While these forms of scholarship have become widely recognized, they are often still clumped together under the general function of research when considering higher education and research institution policies. An additional model of research classification is offered by Gibbons (2003), who identified modes of knowledge production (Gibbons, 2003). The concept is that there are two forms of research: Mode One and Mode Two. Traditional research is Mode One, which is no longer sufficient to describe the complete variety and intricacy of research activities in modern higher education and in other research institutions. Gibbons’ Mode Two states the idea that there is a distributed knowledge production, not hierarchical or fixed, with heterogeneous transdisciplinary skill sets instead of homogeneity. Furthermore and related to the upcoming examination of scholarly metrics and policy development is that:

• There are an increasing number of places where recognizably competent research is being carried out.
• These sites communicate with one another and thereby broaden the base of effective interaction; knowledge is thus derived from an increasing number of tidal flows that both contribute to and draw from the stock of knowledge.
• The dynamics of socially distributed knowledge lie in the flows of knowledge and in the shifting patterns of connectivity.
• The number of interconnections is accelerating; the ebb and flow of connections follow the paths of problem interest; which are no longer determined by the disciplinary structure of research.
• Knowledge production exhibits heterogeneous rather than homogenous growth, providing new points of intellectual departure for further combinations and configurations of researchers (Gibbons, 2003, pp. 111–112).

It should be added that multi-, inter-, and sub-disciplinary research activities conducted by individuals, departments, and institutions ought to also be considered when constructing a framework for understanding, tabulating, and evaluating researchers and research output. Quality assurance is therefore increasingly complex because of the ways in which disciplinary research structures are shifting, and knowledge networks are connecting, interacting, and generating new research configurations.

In the general schema of research activities at colleges and universities, the paradigm of research policies denotes the methods and people who are involved in planning, financing, organizing, and capability building, in addition to the moral/ethical, freedom of thought, and intellectual property aspects of creating and administering the said policies (Birnbaum, 1988; Bonewits and Soley, 2004; Clark, 1983; Kyvik and Skodvin, 2003; Neave, 2002; Powers, 2003; Richardson and Martinez, 2009; Rowley, 1999; UNESCO, 2006). Institutions, people, and organizational processes need to examine how and with whose input decisions about research are made. Fisher (2009) conceived a broad framework for research in higher education that places the aforementioned research forms together with purpose, outputs, funding, personnel, and governance (i.e., policy) as shown in Figure 1.1.

Figure 1.1. Framework of Research.
Adapted from Fisher (2009, p. 53), used with permission.

The conceptual model illustrates a schematic representation of a suitable working standard to examine policy development and implementation for different purposes, forms, and forces. This concept has expanded and developed in recent decades in both higher education, as well as private and publicly funded research institutions. It can serve as a structure to analyze the consequences of developing an effective research evaluation policy and as a comparator for the policy developments to be examined and proposed. Governance and policies of research administration that are now using metrics are important aspects in the framework and is evident by its relationship to academic capitalism (which includes the development, marketing, and selling of research products), increased calls for accountability, changes in funding, corporate-style decision-making, and allocation of resources. These different internal and external forces are complicated and inter-related and require a proper understanding of the current states, recent changes, and original purposes that have evolved yet retain some of the terminology and the options for further development.

It is widely known that the growing influence of Academic Capitalism is manifest in the framework of research purpose and policy. Slaughter and Rhoades (2004) recognized a revolution in corporate management style at higher education institutions that is affecting research topics selected and other policy aspects. Other scholars have also recognized a movement toward commercialization that is bringing in an entrepreneurial management culture (Breton, 2003), with a new batch of corporate leaders that are insistent in guiding university research so that it will ...

Table of contents

• Cover
• Title Page
• Chapter 1 A Primer on Policy and Research
• Chapter 2 History and Evolution of Scholarly Metrics and Impact Factors
• Chapter 3 Concerns and Problems
• Chapter 4 Benefits and Possibilities for Research Policy Improvements
• Chapter 5 Conclusion
• Appendix: San Francisco Declaration on Research Assessment
• References
• Author Index
• Subject Index