International Large-Scale Assessments in Education

Insider Research Perspectives

Bryan Maddox, Bryan Maddox

• Publisher: Bloomsbury Academic
• Year: 2018
• ISBN:
  9781350023611
• Edition: 1
• Topic: Education
• Subtopic: Educational Policy

This book explores the often controversial international large-scale assessments (ILSAs) in education and offers research-based accounts of international testing as a social practice. Assessment exercises, such as the Organisation for Economic Co-operation and Development's Programme for International Student Assessment (PISA), produce comparable international statistics and rankings on educational performance, and are influential practices that shape educational policy on a global scale. The chapters in this volume, written by expert researchers in the field, take the reader behind the scenes to document a broad range of ILSA practices – from the recruitment of countries into ILSAs, to the production and performance of large-scale testing, and the management, media reception and use of test data. Based on data that is only available to expert researchers with inside access, the international case study material includes examples from Australia, Ecuador, Germany, Japan, Mexico, Norway, Russia, Scotland, Slovenia, Sweden, the UK and the USA. The volume provides important insights for teachers, researchers and policy-makers who use and study assessment data and who wish to evaluate its significance for educational policy and practice.

Part One

Theory and method

1

Researching inside the international testing machine: PISA parties, midnight emails and red shoes

Camilla Addey

Introduction

This chapter discusses the methodological and ethical challenges of researching international large-scale assessments (ILSAs) from inside international organizations, government bodies and international businesses involved in ILSA decision-making processes. In particular, the chapter focuses on access and its constant negotiation, carrying out interviews and participant observations, and the ethics of data gathering when investigating ILSAs. The interviewees and observed actors are described as ILSA elites, given the significant role they play in shaping the development of ILSAs. The aim of the chapter is to contribute to methodological scholarship on ILSA research from the inside, an approach which is made challenging by the secretive and poorly accessible nature of ILSAs (this is in great part due to the confidentiality of the test items and the sophisticated psychometric methodologies). The chapter describes the methodological and ethical challenges of gaining access to the elite ILSA community and contributes to scholarship on ethical and methodological questions faced by young, female researchers carrying out research with predominantly male, elite education policy actors (Ozga & Gewirtz, 1995). This is a controversial dimension of empirical scholarship which is rarely discussed in literature.

The chapter draws on the data-gathering experience carried out for the independent research project ‘PISA for Development for Policy¹’ (PISA4D4Policy), which focuses on the ILSA known as PISA-D. PISA-D is the redevelopment of the main PISA, which the OECD claimed was poorly policy-relevant for lower and middle income countries since the main PISA background questionnaires do not sufficiently capture their context variables and the test items cluster students in the lower levels of the PISA metric (Bloem, 2013;² OECD, 2013).

The PISA4D4Policy project looks into how the development of PISA-D is negotiated amongst high level staff at the OECD, in the private sector, and in the participating countries who are involved in its development (see Addey, 2016). It also looks into how PISA-D is made to resonate in Ecuador and Paraguay (two of the nine participating countries)³, and how the OECD is redefining processes of global education governance as it moves into contexts where it had previously not worked in education (see Addey, 2017). The research project adopts a critical policy analysis approach and follows in the tradition of a qualitative research design (Diem et al., 2014). To investigate the above-outlined research questions, I carried out interviews with staff at the OECD and The Learning Bar (a private contractor developing the PISA-D questionnaires) and with high level policy actors in Ecuador and Paraguay, in 2015 and 2016.⁴ I describe these interviewees as ILSA elites. Interviews were carried out in English and Spanish, transcribed verbatim and then checked by each interviewee (who was given the opportunity to revise his/her interview). The interviews were all coded by institution, year and interviewee number or name (i.e. OECD2015#31). I also had the opportunity to carry out participant observations of a PISA-D International Advisory Group (IAG) meeting in Paraguay, and attend a PISA-D cocktail party organized by the OECD and a PISA-D dinner party organized by the Ministry of Education of Paraguay. Data was also gathered through analysis of key documents (all the PISA-D meeting power points and working documents made available online after each PISA-D meeting,⁵ and other key OECD publications.⁶

Getting inside the ILSA machine and staying inside

It was the second of July 2014 when the PISA-D doors were opened to me. From 2013, I had been emailing and meeting up with high level OECD staff to discuss carrying out research on PISA-D (which was still at a very early stage). One such meeting had gone very well: after an hour talking about my PhD research findings on rationales for ILSA participation in lower and middle income countries and my OECD interlocutor substantiating my research findings with his stories about PISA participation, I was handed some PISA-D documents and asked if I would consider participating in an OECD commissioned study on countries’ PISA experience. I was excited about this opportunity but above all I was sure the doors to do research on PISA-D had been opened. It was not to be. The study was carried out by researchers with quantitative research training and I was told that it would be very difficult for me to do research on PISA-D: representatives from all the PISA-D partners⁷ and PISA-D countries would have to formally agree first. The doors were politely closed. About a year later, on 2 July, after Andreas Schleicher had given the opening keynote at the International Testing Commission annual conference, all conference participants gathered for the welcome reception. I found myself standing with a glass of white wine on the terrace of the Teatro Victoria Eugenia overlooking the Urumea river in San Sebastian, Spain, next to the only person I knew (by email, as I had corresponded a couple of times with him when I worked at UNESCO years before): Andreas Schleicher, the Director of Education and Skills at the OECD (also widely recognized as the father of PISA). Like me, he was standing alone. I decided he was my best chance to open the OECD doors to carry out research on PISA-D from the inside. After a brief chat, Andreas Schleicher asked me to email him so that he could put me in touch with the right person at the OECD. Within the next hour, I had run back to my hotel to email him, and within a week, I was in Paris meeting OECD staff again. As discussed by scholars writing about interviewing, access depends greatly on one’s ‘sponsor’ or access point, whose endorsement grants access and cooperation whilst also acting as one’s professional credential and standing (Welch et al., 2002; O’Reilly, 2009; Busby, 2011). As Welch et al. (2002) suggest, one’s sponsor affects the kind of data one gathers, especially when it is a person or an institution which plays an influential role in the network of research participants. Busby (2011) adds that the power relationships and authority structures in this network also shape the data one gathers. Andreas Schleicher briefly became the ‘sponsor’ of my project, before this role was transferred to OECD staff. Throughout this project, I have been aware of how my point of access influenced who I have met and not met, how people have reacted (and not reacted) to me, the relationships I have built (and not built), and the data I have gathered (and not gathered).

My access to OECD staff was built over many years. It was an opportunity to build trusting relationship that would lead to the OECD formally introducing me and putting me in touch with all the PISA-D partners identified in the PISA4D4Policy research design. Andreas Schleicher’s support within the OECD and then the OECD’s staff support within the PISA-D network, meant that I accessed everyone the PISA4D4Policy research design considered crucial informants. Being introduced by the OECD had implications for my study: on the one hand, my new contacts were willing to support my research needs to please the OECD, on the other hand, I risked being perceived as an OECD-researcher carrying out an undercover evaluation (or that I might be reporting to the OECD). Initially, my new contacts (The Learning Bar staff and representatives of the government in Ecuador and Paraguay) were very willing to support my research. I developed research project outlines (in English and Spanish) which explained what kind of support I was seeking (interviews) and what the implications for my research participants were. This correspondence was followed by Skype chats. I approached all Skype meetings as an opportunity to build trust, but the OECD’s introduction had already built it on my behalf. My contacts were keen to help me in any way I requested and insider information was shared with me from the first contacts (as these were not formally arranged interviews and I had not informed my future interviewees of the rights, I did not use these exchanges as interview data).

My interviewees were so supportive that the organizers of the next IAG meeting invited me to participate and present my research. The foundation funding my research understood the value of this opportunity and increased my funds for me to travel to Latin America, and stay in an expensive hotel where the meeting would take place. I was excited about this opportunity, given that no independent researcher has yet been able to carry out participant observations of the OECD’s PISA advisory meetings. I also knew the OECD would not have invited me, and that this invitation was the mistake of someone who was not accustomed to the fact that PISA meetings happen behind closed doors. Rather than creating a diplomatic accident and being turned back once I arrived in Latin America, I decided to inform the OECD staff. Twenty-four hours later, walking down the busy central streets of Zurich with my grandmother, my smart phone suddenly went crazy. People I had only ever emailed and Skyped with, were urgently calling, whatsapping, emailing and texting me all at the same time. Very politely I was told that all PISA-D partners had been consulted, and had agreed that an external participant would make country representatives feel uncomfortable when expressing their concerns. Worried this crisis would close the PISA-D doors forever, I made clear that I knew the rules of the game and had no intention of attending without everyone’s acknowledgement and permission. It was worrying, especially since the next emails I needed to send required actual commitment to an interview appointment. I soon discovered that some doors were still open, but others had been firmly closed.

The OECD staff remained supportive. Indeed, upon insistence and increasingly shorter emails, I even obtained an interview with Andreas Schleicher: after not getting a reply to my many emails, I made one last attempt from the Berlin–Paris night train. Two minutes later, at midnight, I received an email from Andreas Schleicher suggesting I make an interview appointment via his secretary for the following week. Trying my luck further, I asked OECD staff if I could attend the next PISA-D IAG meeting (not the one I had been invited to by mistake). I was told that if I were to find myself in Paraguay on 31 March the following year, I could attend one day of the three-day meeting. I made sure I would be there.

At the same time, my request for interview appointments with other contacts I had established through the OECD were rejected. Upon my insistence, further clarifications on my research project were requested. Given that ILSAs tend to polarize (Gorur, 2017), it would be difficult to gain access if one outlined the research project as being pro- or anti- ILSAs. Indeed, Busby (2011) shows that interviewees provide access based on how they classify the research and researcher. I thus tried to position myself and the project as neutrally as possible, neither pro-ILSAs nor ILSA-sceptic. I re-wrote the outline of my research in such a way that my research participants would see it as a mutually beneficial research project. Once again, the doors that had been closed were opened, so widely that my research participants suggested they would like to discuss a consultancy opportunity. These consultancy offers from my research participants put me in an unexpected position, challenging me to think about my research focus from my research participants’ perspectives. Busby (2011) suggests this requires the researcher to switch roles and become an active participant in the process under investigation. As with the OECD consultancy offer, this one did not take place either. I was relieved.

Although the support I received to carry out my research from the inside required extensive and diplomatic negotiating and trust building, my research participants were motivated by more than the relationship I built with them. In the case of the OECD, the organization committed from the beginning of PISA-D, to make the entire process as transparent as possible. For this reason, the OECD has published online all the documents and Power Points presented at the PISA-D meetings. Allowing and facilitating my research project can be understood as making the PISA-D process more transparent, but also a reflection of the OECD’s commitment to ILSA improvement and innovation through research. Allowing research from the inside is also a way for the OECD staff to tell their version of the story and influence scholarship on PISA-D. For The Learning Bar, supporting my research needs became an opportunity to draw on my expertise (in particular knowledge gained from fieldwork in Ecuador and Paraguay to understand how to respond to the PISA-D countries’ needs) but also to position themselves favourably with the OECD’s request, as was also the case for my contacts in Ecuador and Paraguay. In Ecuador, my presence was used to further the international legitimization which is sought through PISA: beyond the formal dinners and a busy agenda of high level meetings they arranged for me, I was asked to give lectures and photos were taken of me shaking hands with ministers and directors. I was also video-interviewed for video news items to be edited and disseminated. Articles were published claiming an international expert of international education policy approved of and commended the work being carried out with PISA in Ecuador. In the case of Paraguay, participating in PISA clearly fits with the global ritual of belonging (Addey, 2014; Addey & Sellar, 2018) which interviewees suggested also includes attracting international scholars from prestigious universities (being based at Humboldt University in Berlin was considered very highly) to carry out research on Paraguay (being put on the global data map, includes the scholarly research map).

These newly established contacts led to more contacts (known as the ‘snowball approach’ to access), thus allowing me to develop an extensive network of research participants and contacts. Within this network of research participants, I experienced different levels of access. Through the OECD I experienced supportive but guarded access: during the IAG meeting, OECD staff ensured all PISA-D participants at the meeting were informed by email about who I was and why I would be attending the meeting.⁸ Through some newly established contacts, I experienced unlimited access: during the IAG meeting some contacts went out of their way to make sure I was introduced to everyone and informed about everything that could be informative for my research project. Although my network of research participants and trust was expanding constantly, I was also aware that this access and trust were not stable and could easily be broken (Barbour & Schostak, 2005; Busby, 2011), requiring constant renegotiation and demonstrations of trustworthiness before, during and after⁹ the data gathering (Maxwell, 2005; Gains, 2011). This will be discussed in the ethics section in relation to sharing research findings with research participants and keeping doors open for further research.

Finally, it is worth expanding on the relationship between the researcher and the researched as dynamics are often reversed with research participants who occupy high level positions and have significant decision-making power – as was the case in the PISA4D4Policy research project. Apart from constantly negotiating access and building delicate trust relationships, I was also taking decisions about how to position and present myself, evaluating the impact of relationships on the data I gathered, and reflecting on how all the issues related to access were impacting on the data I was gathering (as described by Busby, 2011). Young female scholars carrying out research in elite policy settings are faced with further methodological and ethical challenges. The data-gathering process requires continuously making decisions about how to position oneself, how to use that positioning and their gendered implications. Although some may criticize this as a form of manipulation of the data-gathering process, to deny these dimensions would be to deny the complex reality which scholars face in fieldwork. Being self-reflective and honest about the way we take these gendered decisions (i.e. dressing down to no...

Table of contents

• Cover
• Halftitle Page
• Title Page
• Contents 
• Notes on contributors
• Foreword
• Introduction
• Part One: Theory and method
• Part Two: Observing data production
• Part Three: Reception and public opinion
• Index
• Imprint