Agency-Based Program Evaluation

Lessons From Practice

Stephen A. Kapp, Gary R. Anderson

• Verlag: SAGE Publications, Inc
• Jahr: 2009
• ISBN:
  9781483342689
• Auflage: 1
• Thema: Social Sciences
• Thema: Social Work

This text aids both students and practitioners in articulating the elements of program evaluation, deepening their understanding of the contextual issues that surround and shape an evaluation. Authors Stephen A. Kapp and Gary R. Anderson offer readers details on the application of useful and accepted evaluation methods. It also shows readers how to make sound decisions in balancing techniques and strategies with the realities of the agency environment.

Key Features

• Ethical considerations related to collecting data from service consumers and working with agency personnel
• Strategies for accommodating cultural and ethnic differences when collecting program evaluation data
• Techniques for engaging agency personnel and key constituents in the entire evaluation process

Accompanied by robust ancillaries

This text is accompanied by a robust companion site which you can view here.

The password-protected Instructor?s Site features:

• Sample Syllabi
• PowerPoint Slides
• Test Bank

The open-access Student Study Site offers:

• Quizzes
• Journal Articles
• Web Resources and Activities

This text is designed for upper-level undergraduate and graduate courses in program evaluation within departments of social work, mental health, and human services. It is also a great resource for practicing professionals, supervisors, and managers.

CHAPTER 1

---

Making the Case for Program Evaluation

Program evaluation has unlimited potential for improving services. It can help to clarify and develop the kinds of services that recipients need. It also can help to identify and/or adjust services that are not being delivered to more effectively serve the appropriate people. Crucial information can be discovered about the benefit of services through questions like these: Are program participants getting anything out of the programs? Are recipients doing better after receiving the services? Is everyone better off? If not, who is better off and who is not? What aspects of the program are associated with those who seem to be doing well? How can the program be managed to ensure that everyone gets the service that seems to have the greatest impact?

While this list of questions seems very relevant to social work practice, social worker practitioners and research/evaluators seem to operate on different planets. Practitioners are usually committed to delivering the best possible service that directs attention to their immediate interactions with the client, for example, such as a counseling session or the completion of a referral. This orientation does not leave excess time or energy for evaluation. Additionally, research and evaluation activity is often separated from practice. Many agencies have distinct departments and/or staff functions that perform these duties. Research or quality assurances arms in organizations usually are separate from direct-service wings. The schism is further reinforced in most social work curricula where there is a clear distinction between practice courses and evaluation/research courses.

The primary focus of social work intervention is to improve the lives of individuals, families, and their communities by working with them to assist them with helpful services and resources. Consistent with organizational and educational reinforcements, when thinking about this mission to improve the lives of clients, it is unlikely that program evaluation is one of the first topics that comes to your mind. While we understand that, we would like to argue that an effective evaluation can be a very powerful tool for improving the quality of services and increasing the positive impact on service recipients.

ETHICAL OBLIGATIONS

---

As social workers, we have an ethical responsibility to exploit all resources in our repertoire in the interest of improving the lives of the clients we are serving. While our Code of Ethics guides us in the treatment of clients and the clarification of moral dilemmas that may arise in social work practice, there is also attention focused on the importance of providing services that are effective and helpful to clients (NASW, 1999).

Additionally, our Code of Ethics emphasizes a social worker’s mandate to focus on social justice in the interest of interrupting the oppression of high-risk groups dispensed by prominent social institutions (NASW, 1999). If you are truly committed to at-risk individuals, families, and/or their communities, then program evaluation is a great ally. You can discover if services are actually being delivered to targeted groups or find out if services are being delivered as intended or if services are having the desired impact. Evaluation can also help to identify the services that are actually needed. When you consider the array of tools a social worker has at his or her disposal, program evaluation techniques have great potential for assisting in this fundamental mission to improve lives.

Checking Vitals

The Code of Ethics focuses on the importance of providing services that are effective and helpful to clients (NASW, 1999). The Code of Ethics emphasizes a social worker’s mandate to focus on social justice in the interest of interrupting the oppression of high-risk groups dispensed by prominent social institutions (NASW, 1999).

A recent trend in social work service is the choice of services models based on evidence-based practice. Social workers with this orientation would choose service treatment modalities that have been proven to work with the specific target population of interest. Choosing a model with an empirical history of effectiveness is an excellent practice; however, it does not ensure that the service will benefit the service recipients. In addition, evidence-based practitioners need to evaluate specific services to ensure that the model’s effectiveness generalizes to that setting (Yeager & Roberts, 2003).

Social workers in managerial and supervisory capacities often practice from a consumer-centered management framework. The use of evaluative information to assess consumer needs, the effectiveness of the implementation of services, and the eventual impact of services is central to a consumer-centered approach to administrative practice (Poertner & Rapp, 2007).

Very similar arguments about the needs for service and its respective effectiveness are critical in the process of securing funds. Evaluation findings can be invaluable in the process of seeking new funding or continuing existing funds for services. When seeking grant monies, public or state contracts, or foundation dollars, it is likely that some type of evaluation will be required to enhance the likelihood of securing the award.

Checking Vitals

Evidence-based practitioners need to evaluate specific services to ensure that the model’s effectiveness generalizes to that setting (Yeager & Roberts, 2003). The use of evaluative information to assess consumer needs, the effectiveness of the implementation of services, and the eventual impact of services is central to a consumer-centered approach to administrative practice (Poertner & Rapp, 2007). When seeking grant monies, public or state contracts, or foundation dollars, it is likely that some type of evaluation will be required to enhance the likelihood of securing the award.

PUTTING IT INTO CONTEXT

---

In addition to the practical and ethical arguments, evaluation is useful in service delivery efforts that are exceedingly complex. Clients bring complex sets of needs to the service setting, with economic and social challenges often at the center of those needs. Client families often come from racial or ethnic groups who have faced a lengthy history of injustice and discrimination. Additionally, individuals and families may have received services from established systems that do not have a track record of helping their clients. Despite these challenges, individuals and their families seem to find personal strengths that allow them to endure and survive.

In most cases, social work services are housed in extremely complex organizations. Ongoing discussion of organizational politics does not necessarily cast a favorable light on the individuals who participate in processes that impede the use of evaluation data (Rubin & Babbie, 2005). Our experience indicates that various players within these organizations engage in these processes as a part of their jobs, and the accompanying complexities come with the territory. When you look at these services systems you will find that different individuals design, implement, fund, and often evaluate the same services from different vantage points in the organization. For example, while direct service staff actually deliver the service, supervisors, managers, and administrators hire, fire, and manage these staff. Depending on the size of the organization, other staff functions may manage the payroll and benefits of these staff, deal with computers and technology, and also raise funds for the services. Finally, executive directors and upper management personnel usually oversee the operation at the top level of the organization.

All of these parties rely on external constituencies for many critical things. Funding and contractual arrangements may come from public or private entities. Favorable laws that support the mission of the organization require the agency to lobby legislators (local, state, and federal) and their supporters. The point of this discussion is not to review your organizational theory, but to keep in mind that doing evaluation and using data to improve services often gets lost in the shuffle. The staff at various levels of the organization are forced to deal with pressing matters, and unfortunately, in most cases, evaluative work is not viewed as the most pressing matter.

The low priority of evaluation within social service organizations is also complemented by the aforementioned historical schism that exists in the social work world between research and practice. Research is often viewed as something outside of practice. This is often reinforced in schools of social work where research and evaluation courses are taught in isolation from direct practice. In our conversations with students about research and taking additional research courses, even those who do well in the research course see it as a separate and less critical part of their professional development. This is further supported in organizations where ongoing evaluation is often excluded from the “real work” of the agency.

SO WHY BOTHER?

---

Despite the challenges, the potential payoffs for program evaluation efforts are significant. We have said it before and we will say it again: Program evaluation data can be used toward the betterment of social work practice in a number of ways. When agency staff at all levels participates in evaluation, it enhances the quality of the evaluation, the evaluation process, and the prognosis for securing credible and useful information. Additionally, social workers who participate in the evaluation process pick up skills that can improve their skills as practitioners. Patton (1997) describes the skills that participants in the evaluation process acquire that make them better prac...

Inhaltsverzeichnis

• Cover Page
• Dedication
• Title Page
• Copyright
• Contents
• Foreword
• Acknowledgments
• Introduction. The Purpose of This Book
• Chapter 1. Making the Case for Program Evaluation
• Chapter 2. Steps in Program Evaluation
• Chapter 3. Ethics and Program Evaluation: Applying a Code of Ethics to Field-Based Research
• Chapter 4. Ethical Challenges for Evaluators in an Agency Setting: Making Good Choices
• Chapter 5. Agencies and Academics: The Social and Political Context of Program Evaluation
• Chapter 6. Cultural Competency and Program Evaluation
• Chapter 7. Program Definition: Using Program Logic Models to Develop a Common Vision
• Chapter 8. Program Description: Evaluation Designs Using Available Information
• Chapter 9. Evaluation Design: Options for Supporting the Use of Information
• Chapter 10. Group Designs and Methods
• Chapter 11. Evaluation Design: Qualitative Designs and Applications
• Chapter 12. Consumer Satisfaction
• Chapter 13. Dissemination: Spreading the News
• Addendum: Families as Evaluators
• References
• Index
• About the Authors