Assessing Outcomes in Child and Family Services

Comparative Design and Policy Issues

Anthony N. Maluccio

• Publisher: Routledge
• Year: 2017
• ISBN:
  9781351328104
• Edition: 1
• Topic: Politica e relazioni internazionali
• Subtopic: Politica sociale

In this collective portrait, editors and authors do not attempt to draw systematic, country-by-country comparisons. Given the magnitude of the issues, they believed that it would be inappropriate to paint with too broad a stroke. What they have accomplished, however, is to codify and identify what the participants repeatedly noted in regard to issues and difficulties inherent in conducting outcome evaluation. These include: varying definitions of outcome; complexities in measuring outcomes of particular interventions with different groups of consumers and documenting the effectiveness of the intervention; the tendency to focus on evaluation of process more than outcome; the challenge of involving practitioners in the evaluation task, in part because its value is unclear to them or perceived as distant or untrustworthy; the typical inadequacy of resources available for systematic evaluation; and the need to inject rigor into the design and execution of evaluation projects. The authors demonstrate strong conviction about sharing research expertise across national boundaries; learning through each other how to cope with organizational impediments to cross-national collaboration; and strengthening the interaction between practice and research. Their contributions suggest that there is wide interest in pursuing cross-national collaboration. In recent years, largely in response to demands by their funding sources for accountability, assessment of performance, and cost effectiveness, researchers in human services have been devoting increased attention to outcome evaluation. Limited attention, however, has been given to the findings of evaluation studies conducted in different countries. The present volume has been organized and edited to address the task of learning from outcome research across the world. Its goal, an extension of a major goal of the human services in any one country, is to improve life chances of vulnerable children and youth.

1

Current Initiatives in the Development of Outcome-Based Evaluation of Children’s Services

Harriet Ward

Over the last century, England, along with most other developed countries, has introduced a wide range of services designed to promote the satisfactory development of its children. Services such as physical and mental health care, dental care, and education are universally available and, despite a flourishing private sector, are still largely delivered, without charge, by public agencies. Nevertheless, of the eleven million children and young people under eighteen years of age in England today, about four million are thought to be vulnerable in that they “would benefit from extra help from public agencies in order to make the best of their life chances” (Department of Health, Department for Education and Employment and Home Office, 2000:2). Of these, 381,500 are defined as children in need, that is, children whose vulnerability is such that they are unlikely to reach or maintain a satisfactory level of health or development, or those whose health and development will be significantly impaired without the provision of additional services (Children Act 1989, Section 17.10). Children Act 1989, which introduces the concept of children in need, also lays on local councils an obligation to provide these additional services, generally through their social services departments, although working closely with other agencies.

While it is possible to trace the development of child welfare services in Britain over the last century, it is only over the past twenty years or so that serious attempts have been made to evaluate their outcomes or effectiveness. The introduction of outcome-based evaluation of children’s services is, as this book shows, a developing initiative throughout the Western world; in Britain it has been particularly engendered by a loss of confidence in the welfare state, a perceived lack of accountability of public services accompanied by rising costs, and a thriving consumerist movement that demands that services meet the requirements of users (Parker, Ward, Jackson, Aldgate & Wedge, 1991). The systematic evaluation of children’s services is part of a much wider movement to introduce outcome-based accountability into all public sector services: health, education, transport, social security and now social services are all increasingly required to provide demonstrable evidence of satisfactory delivery and proof that public money is being well-spent.

Within the above context, this chapter explores the introduction of outcome- based evaluation in services for children in need at both micro and macro levels and considers the issues raised by initiatives such as the Looking After Children project, a methodology designed to help practitioners and their managers assess outcomes for children placed in out-of-home care (Ward, 1995), and the “Performance Assessment Framework” (Department of Health, 1999b)—a series of performance indicators for all the “personal” social services required annually by government and published in the national press.¹ Poor implementation and problems with management information systems have meant that the link between aggregate and individual data has sometimes been lost, a situation that a new Integrated Children’s System aims to redress.

THE LOOKING AFTER CHILDREN PROJECT:OUTCOME-BASED EVALUATION AT THE MICRO LEVEL

On almost any criterion, children who have been placed in the care of public agencies tend to be among those most obviously in need, and for many years concerns have been raised about the poor outcomes that they appear to achieve. Recent figures suggest that 70 percent of care leavers have no educational qualifications; 50–80 percent are unemployed a year after discharge; 30 percent of single homeless people have been in care; and one in seven young women leaving care is pregnant or already a mother (Acheson, 1998). These women also receive the most expensive level of service provision: on average, each child looked after away from home currently costs her or his social services department £435 per week, about five times more than each child supported in his or her own family or living independently (Department of Health, 2001:para. 20b). It is therefore unsurprising that the first initiatives to introduce outcome-based evaluation measures have concentrated on the assessment of the effectiveness of services for children placed in care or accommodation. However, although there is some evidence that children’s experiences in public care do little to compensate for previous adversities and may indeed compound them, we do not yet know how far this is the case, or whether this extremely disadvantaged group would have displayed even less successful outcomes without the provision of services.

The Department of Health (England) initiative on assessing outcomes for children looked after away from home is known as the Looking After Children project (Parker et al., 1991; Ward, 1995). This long-standing research and development program has been running since 1987, and is now in its fourth stage. In the first stage (1987–1991), a working party of academics and practitioners devised a theoretical framework for assessing outcomes in child care. They argued that appropriate outcome measures should adopt the child’s perspective, and therefore assess services by asking how far they promote children’s long-term chances of achieving satisfactory well-being in adulthood. Although the working party’s remit was to produce measures designed to assess outcomes for children in care or accommodation, they argued that the program should, at least in theory, be applicable to all children rather than focusing solely on issues specific to those in receipt of a particular service.

The working party therefore identified seven developmental dimensions along which all children need to progress if they are to achieve long-term well-being in adulthood: health, education, identity, family and social relationships, social presentation, emotional and behavioral development, and self-care skills. They argued that, at a very basic level, all parents have broadly similar objectives for their children in each of the above dimensions; for instance, they want them to be healthy, literate, confident, popular, attractive, and competent, and are concerned if they show signs of emotional or behavioral disturbance. The objectives should be no different for those children for whom the state has acquired parental responsibility. However, children will only make satisfactory progress if their needs in each of these dimensions are adequately met, primarily by parents but with assistance from social workers, foster carers, and other child welfare professionals where necessary. Both parents and child welfare agencies can be held accountable for the extent to which they attempt to meet children’s needs, and the outcome of their efforts can be measured with reference both to children’s experience of parenting and to their progress, although it is important to recognize that there is not always a causal relationship between the two (Parker et al., 1991).

The practical application of the theoretical perspectives noted above produced the Assessment and Action Records, a series of six practice tools that ask age- related questions about significant aspects of children’s experience and progress, drawn from research knowledge about normative development (Department of Health, 1991). In the second stage of the project, the Assessment and Action Records were extensively piloted: social workers and children and young people in five local authorities tested their viability as practice tools; in addition, data from Assessment and Action Records completed by a representative sample of parents and children not looked after away from home were used to evaluate the relevance of their content to normative parental practice within the community. The records were extensively revised in the light of these findings, and additional materials were produced in order to allow them to be set within existing procedures for information-gathering, planning, and review (Department of Health, 1995; Ward, 1995).²

During the third stage of the project (1995–1998) widespread use of the materials was facilitated by a structured program for implementation, which was sup-ported and funded by central government and informed by both theoretical and empirical work undertaken in the second, developmental stage. Although implementation was only made mandatory in Wales, virtually all local authorities in England and Wales made it their policy to use the materials as case management tools for all children who were looked after for substantial periods. Versions adapted to local circumstances and requirements are now also being implemented in Scotland and Northern Ireland. In addition, extensive pilot and implementation projects have been undertaken in Canada, Australia, Germany, Sweden, Hungary, and Russia (Ward, 1998).

Social workers are expected to ensure that an Assessment and Action Record is completed annually for each child looked after away from home for whom they hold responsibility. In theory, therefore, a system is in place in English and Welsh local authorities that should allow for outcome data on all looked after children to be routinely collected and recorded as part of the everyday interaction between social workers, carers, and the children for whom they hold responsibility. It should be possible to aggregate the data held on individual children to assess the outcomes of the service as a whole.

However, perhaps we should not be surprised to find that reality falls consider-ably short of the theoretical picture. The evidence from an extensive audit program undertaken between 1996 and 1999 demonstrates that, even with substantial support from central government, the use of these materials, though improving over time, is often inconsistent and patchy. While those parts of the system that replicate existing procedures for information-gathering, planning, and review are generally found in 70–80 percent of case files, the innovative Assessment and Action Records, designed to gather data on outcomes, are only used in about 36 percent of relevant cases, and are rarely used repeatedly to gather longitudinal information (Moyers, 1996; Peel, 1997; Scott, 1999).

There is considerable difference of opinion as to whether such findings are indicative of major difficulties inherent in the system as a whole, or whether they demonstrate teething problems in the introduction of a program that requires a radical change in social work culture, compounded by the well-documented difficulties of translating social policy into social work practice (Baldwin, 2000). It seems likely that all of these factors have impeded comprehensive implementation.

Undoubtedly the current version of the Assessment and Action Records is too long, and the format, which repeatedly invites respondents to give tick-box answers to a range of primary and secondary questions, is too mechanistic and off-putting for users. Moreover, the design currently bears an unfortunate resemblance to a public examination paper and could also be improved. A major weakness has been caused by the delay in producing an electronic version of the records, so that at present data can only be aggregated and analyzed by keying them separately into a database. We also know that, in their current version, the records are not always appropriate for use with children and young people with certain disabilities: in particular, the educational progress of children with learning disabilities cannot be adequately measured against normative standards, while the development of self-care skills raises very different issues when children’s functioning is substantially impaired by physical conditions affecting, for instance, mobility, sight, hearing, or cognition. However, as noted below, while plans to make radical changes to the materials may make them more user-friendly, even if it were possible to construct a computerized system that could be presented in a manner thoroughly acceptable to all the participants, it would not be adequately implemented unless a number of other issues are addressed.

Systematic outcome-based evaluation of this nature requires participants to gather accurate data on a routine basis in order to demonstrate changes to the service and/or developmental progress in the child or young person concerned. However, the evidence from the audits, and also more particularly from Stage Four of the project (1996–2002), where researchers are demonstrating how the data from case files of242 children can be transformed into management information, is that the need to gather and record accurate information is not always understood by practitioners.³ Even where all the materials are completed, case files demonstrate substantial gaps and inaccuracies in key information. It is often impossible to find the answer to simple questions such as the date on which a child was admitted to care or accommodation, or the number of placements he or she has experienced since then; other information such as academic progress or the child’s health needs may be out of date or no longer relevant.

Discussions with social workers reveal that many of them have the information in their heads, but they do not regard writing it down as necessary. Indeed, this is at the heart of one of the main criticisms raised by practitioners: that by requiring them to monitor children’s progress by answering questions on forms, the system bureaucratizes the social work task (Ward, 1995). This debate is taken further forward by Garrett (1999), who argues that by suggesting that there are a number of recognized indicators of progress that can be monitored in order to evaluate outcomes, the system requires social workers to act as controlling agents who require children in need to conform to societal norms rather than giving free rein to their individuality. It is difficult to see how such criticisms can be addressed by any system designed to produce routine evaluations of outcome.

Up to a point, these criticisms reflect conflicting perceptions of social work accountability. In England, central government produces legislation that local councils are required to implement, but the manner of implementation is largely left to their discretion; procedures followed by social work staff often mirror this relationship, in that the council for which they work translates the legislative requirement into local policy, but as professionals who work in bureaucracies, they are given considerable discretion as to how this is operationalized. An acknowledgment of professional freedom to act is sometimes interpreted as meaning that social workers are relieved of the responsibility of implementing policy, as Garrett (1999) appears to argue. The audits of Looking After Children showed that the success or failure of implementation was often related to middle managers’ perception of the amount of discretion they could exert in deciding whether or not to encourage fieldworkers to follow local policy directives. Even in Wales, where the system has been made mandatory, middle managers vary in the extent to which they urge compliance, and implementation frequently follows the letter rather than the spirit of policy: the documentation is more frequently found on files, but questions are more often left unanswered (Nicholson and Ward, 1999).

Implementation of the system, however, while imperfect, has by no means failed. Not only has no council withdrawn from the program, but there is substantial evidence to suggest that, even if the practical completion of the documents is inadequate, the conceptual framework upon which the materials are based is gradually becoming part of accepted social work theory. A comprehensive study of social workers’ understanding of children’s needs found that those working in local authorities which had implemented the Looking After Children materials were able to describe them in the structured, developmental terms provided by the seven dimensions used in the project (Department of Health, 2000a). A number of councils, having introduced Looking After Children for young people looked after away from home, have begun to use the theoretical framework as a basis for developing procedures for identifying and assessing other children in need; these initiatives were reinforced when the new government guidance for the assessment of need was released in 2000 (Department of Health et al., 2000). This program now requires social workers to build on the developmental basis of the Looking after Children model in assessing all children in need, by identifying how far factors related to parenting capacity, the wider family, and the environment are impeding or promoting children’s progress in the seven developmental dimensions.

Not all social workers in England have yet grasped the message that, in their interactions with individual children, they need to gather accurate, structured information that allows them to assess how far services are affecting developmental progress. Nevertheless, this argument is constantly being reinforced and will continue to be so through new procedures and training. In England and Wales, the new Post Qualifying Award in Child Care, introduced in 2000, places an added emphasis on the importance of social workers acquiring a greater understanding of children’s developmental needs; there are plans in progress for this also to become a more central part of the curriculum for the basic diploma.⁴ New information requirements imposed by government on social services will also make it likely that, in the future, social workers will be expected to devote more attention to the need for comprehensive and accurate recording (Ward, 2000). Moreover, such initiatives are also now being matched at the macro level by a similar emphasis on the need for outcome based evaluation.

PERFORMANCE ASSESSMENT FRAMEWORK: OUTCOME-BASED EVALUATION AT THE MACRO LEVEL

As part of the government program for modernizing and improving social care services (launched in 1998), a performance management system has been introduced, providing a comprehensive framework for measuring the effectiveness of services against identified objectives and setting targets for improvements (Department of Health, 1998, 1999a, 1999b). English councils are now required to work toward eighteen objectives for services for children and adults; their progress will eventually be monitored through fifty performance indicators. Councils are also required to submit data on most of these indicators annually to central government, although the full range will not be available until late 2002. A series of National Priorities Guidance targets has also been set...

Table of contents

• Cover Page
• HalfTitle Page
• Title Page
• Copyright Page
• Contents
• Contributors
• Preface
• Acknowledgments
• Introduction: Outcome-Based Evaluation: Cross-National Perspectives
• 1 Current Initiatives in the Development of Outcome-Based Evaluation of Children’s Services
• 2 The Black Box: Accounting for Program Inputs When Assessing Outcomes
• 3 The Evaluation of “Community Building”: Measuring the Social Effects of Community-Based Practice
• 4 Outcome-Based Evaluation of National Health and Social Programs
• 5 Nonexperimental Methods of Evaluating Social Programs: Applications for Child and Family Services
• 6 An Intervention to Reduce Smoking Habits through Counseling from the General Practitioner
• 7 Improving Mental Health Care for Children and Adolescents: Strategies and Lessons
• 8 Evaluating Social and Health Services with Children, Adults, and Elderly Person
• 9 Outcomes Are Dependent on Inputs: Does Risk Assessment Inform Service Delivery?
• 10 A Learning-Organization Approach to Evaluation
• 11 Outcome Studies in the Context of Organizational Inertia and Political Ideology
• 12 Family Service Centers: Lessons From National and Local Evaluations
• Author Index
• Subject Index