Service Learning Program Evaluation

by Salvatore Alaimo

Definition

Evaluation is a systematic process for an organization to obtain information on its activities, its impacts and the effectiveness of its work, so that it can improve its activities and describe its accomplishments (Mattessich 2003). Program evaluation for service learning primarily involves measuring learning objectives of the participating students and the impact on the recipients of their service.

Program theory is descriptive information that provides a coherent account of how and why your program will generate its intended outcomes (Mattessich 2003). The visual framework for presenting your program theory is a logic model that incorporates the program´s outcomes, outputs, activities and inputs.

Outcomes are intended and unintended consequences of a program, usually in the context of knowledge, attitudes, values, skills, behavior, condition or status concerning the program´s participants (Mark, Henry and Julnes 2000). A program may have initial, intermediate and/or long-term outcomes. For example, an initial outcome for students involved in a service learning program may be knowledge gained while the long term outcome may be a change in behavior based on that knowledge.

Experiential learning, the basis for service learning, is learning by doing, by using, or experiencing a concept or testing a skill. The experiential learning model is an inductive learning process consisting of five phases: experiencing, reporting or publishing, processing, generalizing and applying (State of Michigan 2002).

Historic Roots

Program evaluation of service learning has its roots in experiential learning, a concept of connecting education with personal experience largely attributed to John Dewey from his Experience and Education written in 1938. However, the basis for evaluative methods of experiential learning didn´t surface until the 70´s with the Cooperative Assessment of Experiential Learning Project in 1973 and David Kolb´s experiential learning model in 1975. Kolb´s model included a four-stage cycle moving from concrete experience to observations and reflections to formation of concepts and generalizations to testing implications of concepts in new situations (Chickering, 1977). The early attempts at evaluating experiential learning largely concentrated in higher education, and used school grades and observations of student´s behavior as the basis for measuring results.

The 1980´s saw a movement towards more formalized evaluation of youth service and a migration into K-12 schools. However such efforts concentrated primarily on the number of hours of service contributed, and the components of assessing individual student performance included their written work, participation in group discussion, one-on-one interviews, and student self-assessments (Conrad and Hedin 1987). The context of such assessments largely were comprised of the student´s experience and behavior in the service activity and inquired as to whether the student learned something through the service or not.

The decade of the 90's would prove to be a landmark period for program evaluation of service learning. Federal legislation specifically defined service learning in the National and Community Service Act of 1990 as: "a method under which students or participants learn and develop through active participation in thoughtfully organized service (Pearson 2002)." The act stated that this service is "integrated into and enhances the academic curriculum of the students, or the educational components of the community service program in which the participants are enrolled (Pearson 2002)." This put the focus on the learning component of service and called for evaluation to determine specific, measurable outcomes. Complete outcomes included three components: the condition under which the learning took place, the behavior while learning and the standard used to judge the degree of learning (Cairn 1993). The measuring of learning constituted the five domains of verbal, intellectual, cognitive, attitude and motor. The amending of the 1990 act in 1993 created the Corporation for National and Community Service (CNCS) which administers grants through several initiatives including Learn and Serve America that deals with K-12 students (Corporation for National Service 2003). The accountability component of these grants has stiffened over time, therefore driving more formal program evaluations of service learning.

The United Way of America introduced the basic logic model and outcome measurement framework to their grantees in 1996, in Measuring Program Outcomes: A Practical Approach (United Way of America 1996). Program evaluation began to be thought of at the planning or development phase of a program, rather than after it had been implemented. Program evaluation of service learning has evolved into the models we now have today including those in the Compendium of Assessment and Research Tools (CART), the Educator´s Guide to Service-Learning Program Evaluation and Shumer´s Self-Assessment for Service-Learning (National Service-Learning Partnership 2004). These are arguably the most formal examples of program evaluation of service learning.

Importance

Effective service learning is now seen as a major vehicle for education reform and for improving the learning experience of students. Comprehensive School Reform (CSR) is a federal program incorporated under the No Child Left Behind Act of 2002. A school must address eleven components to be considered a CSR program. The seventh component reads "Provides for meaningful involvement of parents and the local community in planning, implementing and evaluating school improvement activities (Pearson 2002)." Almost half of all Americans believe that improving our schools should be our nation´s highest priority. Ninety percent agree that service-learning helps students build the skills they need later in life (Learning In Deed 2000). Teachers are seeking new and innovative ways to ensure their students acquire these skills, and program evaluation of service learning is critical to determining the effectiveness of such programs.

The increasing call for accountability required more stringent evaluation of programs incorporating measurable outcomes. The Corporation for National Service´s CEO has authority to (1) work with the grantees to establish performance measures for each grantee; (2) require corrective plans for those not meeting goals; and (3) reduce or terminate grants if corrections are not made (Corporation for National and Community Service 2003). It is important to note that funding decisions for the overall initiatives from CNCS are appropriated in Congress. Such decisions are arguably made based on funding priorities, but those priorities are largely driven by the effectiveness of programs. On a more micro level, program evaluation helps teachers, administrators and school officials improve the effectiveness of their service learning programs and the efficiency in which they allocate resources for such programs.

When students engage in service to the community, in addition to learning, they are contributing to their long-term social and civic development. When youth engage in service, they are more likely to carry those values through to adulthood. This has important consequences for many community organizations that are the recipients of such service. Effective service learning from students can greatly assist these organizations in serving their constituency and working towards their missions. In this way, service learning can contribute to community building by connecting citizens to community issues at a young age and fostering relationships between schools and the local community organizations. Most schools with service learning programs cited strengthened relationships among students, the school, and the community is key reasons for practicing service learning (Skinner and Chapman 1999).

Ties to the Philanthropic Sector

Service learning evolved partly out of the movement towards formalizing volunteerism, which is a big component of philanthropy. The debate over whether required service is really volunteering notwithstanding, the ultimate impact of service learning can contribute to the "greater good" and attend to community needs. Program evaluation of service learning can yield indicators demonstrating the effects on the recipients of such service. Such results can support altering service learning programs so that the students meet their learning objectives while the community organizations benefit from their service. Service learning is also a major component of most of the youth-serving nonprofits in the United States such as 4-H, The Boy Scouts, the Girl Scouts, Girls, Inc. and others. Program evaluation can assist those organizations in determining whether their programs are achieving their intended outcomes for service learning.

The funding through CNCS enables service-learning programs to exist in public schools throughout the country. Funding also exists for service learning research through organizations such as The Pew Charitable Trusts, The William T. Grant Foundation and The Spencer Foundation (Learning In Deed 2000). The Surdna Foundation takes a slightly different take on service learning in that they state that they "don't have an educational focus." But they "do have a strong interest in community problem-solving, service and service-learning." The foundation's two main grant areas are Community Revitalization and Effective Citizenry (Surdna Foundation 2004), so they see service learning as a vehicle to address the revitalization of communities through effective engagement of citizens. Corporate philanthropy for service learning also comes in the form of grant awards for successful service learning projects. Examples include Prudential with their Spirit of Community Awards and the State Farm Companies Foundation with their Harris Wofford Awards to honor service-learning achievements.

Lastly, many community foundations, such as the Community Foundation of Greater Lorain County in Ohio, quite often distribute small grants to teachers and schools for service learning programs and/or projects. By funding these initiatives for service learning, these philanthropists are leveraging dollars to enable youth to address community needs while learning new skills, behaviors and values. Without such support from the philanthropic sector, these service-learning programs would not exist.

Key Related Areas

A framework for program evaluation is a practical tool for summarizing and organizing the essential elements of program evaluation. It involves the steps and standards for an effective program evaluation. The recommended framework to use is the one developed by the Center for Disease Control. The six steps of a program evaluation include: engaging stakeholders, describing the program, focusing the evaluation design, gathering credible evidence, justifying conclusions, and ensuring use and sharing lessons learned (CDC 1999).

An outcome measurement framework is a guide that portrays the system for measuring your program´s outcomes. It contains indicators, data sources and data collection methods for each of your outcomes. An indicator is the information that indicates how well the program has met a particular outcome (United Way of America 1996). For example, the outcome of your students cleaning up a park may be a change in attitude towards their local environment. An indicator of this outcome for your class would be the percentage of students treating their local environment differently i.e. picking up litter, starting recycling programs or indicating in other ways they will now treat their local environment differently. Data collection methods may be observing their behavior, group discussion, written surveys or a combination thereof.

Phases of an evaluation typically include planning and design, data collection, data analysis and reporting (Mattessich 2003).

A process evaluation evaluates how your program is implemented. The Alliance for Service Learning in Educational Reform (ASLER) standards for an effective program is recommended as a guide for assessment. Examples of the standards include: students are engaged in tasks that challenge and stretch them cognitively and developmentally; communication and interaction with the community are promoted and partnerships and collaboration are encouraged; student reflection takes place before, during and after service (Billig 2000).

The standards for program evaluation are grouped into four categories. Utility standards ensure that information needs of evaluation users are satisfied. Feasibility standards ensure that the evaluation is viable and pragmatic. Propriety standards ensure the evaluation is ethical and conducted with regard for the rights and interests of those involved and affected. Accuracy standards ensure that the evaluation produces findings that are considered correct (CDC 1999). (For more information on standards for evaluation practice, see The American Evaluation Association http://www.eval.org)

Important People related to the Topic

• Shelley Billig: Billig is a nationally recognized researcher, evaluator, and technical-assistance provider who has written extensively about service learning. She currently serves as Vice President of RMC Research Corporation, director for the Research Network portion of the W.K. Kellogg Learning In Deed initiative, advisor to the National Commission on Service-Learning, and director of several national, state, and local studies of service-learning in K-12 settings. She directed a four-year technical assistance project for the Corporation for National Service that explored ways in which federal programs can adopt service-learning as a key strategy of educational reform.
• John Dewey (1859-1952): Dewey was a major figure in American intellectual history whose areas of work included philosophy, psychology, education, politics, and social thought. He was a noted author and a preeminent voice in American educational philosophy, with emphasis on what is generally called "progressive education." He is considered to be the father of experiential learning who professed in his book Experience and Education (1938) that students would learn more effectively and become better citizens if they engaged in service to the community and the service was incorporated into their academic curriculum.
• Harry P. Hatry: Hatry is an influential contributor to the field of performance measurement and evaluation and has been a leader in developing and promulgating procedures for measuring the performance, especially the outcomes, of government and private nonprofit organizations. He has contributed to the United Way's focus on outcome measurement. Hatry is currently a principal Research Associate at the Urban Institute where he conducted and published research for the Corporation for National and Community Service on Outcome Indicators and Outcome Management for their various initiatives.
• David Kolb: David Kolb is a noted author, founder and chairman of Experience Based Learning Systems, Inc. and professor of Organizational Behavior at the Weatherhead School of Management, Case Western Reserve University. He wrote Experiential Learning: Experience as the source of learning and development, and created the Kolb Learning Style Inventory and the Experiential learning Model (Kolb Cycle) which is largely cited as an applied theory of experiential learning that service learning models evolved from.

Related Nonprofit Organizations

• American Evaluation Association is an international professional association of evaluators devoted to the application and exploration of program evaluation, personnel evaluation, technology, and many other forms of evaluation. The work of their members involves assessing the strengths and weaknesses of programs, policies, personnel, products, and organizations to improve their effectiveness (http://www.eval.org).
• The National Service-Learning Partnership works with and through over 8,000 individual and organizational members to educate, organize, and mobilize service-learning supporters around a strategic agenda promoting and strengthening the practice of service-learning. They provide their members with an array of services targeted at increasing the odds for service-learning to become a common educational practice endorsed by a broad range of stakeholders--teachers, administrators, parents, school board members, young people, community leaders, teacher educators, researchers, and policymakers (http://www.service-learningpartnership.org).
• National Youth Leadership Council is a leader in the service-learning movement that provides practitioners with training and technical assistance through the National Service-Learning Exchange. The organization's National Service-Learning Conference draws nearly 2,500 attendees from around the world for three intensive days of speakers, workshops, and networking. Their initiatives such as Growing to Greatness: The State of Service-Learning Project provides research intended to transform service-learning and influence public policy (http://www.nylc.org/).
• W.K. Kellogg Foundation is one of the top 20 largest foundations in the United States with respect to giving. They have four grant areas, two of which are Youth and Education and Philanthropy and Volunteerism, and they have been a leader in funding evaluation initiatives in these areas. Most notably was Learning in Deed, a national initiative started in 1998 to engage more young people in service to others as a part of their academic life (http://www.wkkf.org).

Bibliography and Internet Sources

• Billig, Shelley. "Research on K-12 School-Based Service-Learning: The Evidence Builds." Phi Delta Kappan, 81 No. 9: (658-664) May, 2000.
• Cairn, Rich. Learning by Giving: K-8 Service-Learning Curriculum Guide. St. Paul, MN. National Youth Leadership Council, 1993.
• Chickering, Arthur W. Experience and Learning: An Introduction to Experiential Learning. New Rochelle, NY: Change Magazine Press, 1977. ISBN: 0915390108.
• Conrad, Dan and Diane Hedin. Youth Service: A Guidebook for Developing and Operating Effective Programs. Washington, DC: Independent Sector, 1987.
• Corporation for National Service. "About Us: Principles and reforms for a Citizen Service Act." January, 2003. https://www.nationalservice.gov/about/role_impact/mission.asp
• Learning in Deed: Making a Difference Through Service Learning. "Service-Learning Delivers What Americans Want from Schools." Academy for Educational Development, November, 2000. https://essaypro.com/essay-help
• Mark, Melvin M., Gary T. Henry and George Julnes. Evaluation: An Integrated Framework for Understanding, Guiding and Improving Public and Nonprofit Policies and Programs. San Francisco, CA: Jossey-Bass, 2000. ISBN: 0787948020.
• Mattessich, Paul W. The Manager´s Guide to Program Evaluation: Planning, Contracting and Managing for Useful Results. St. Paul, MN: Wilder Publishing Center, 2003. ISBN: 0940069385.
• National Service Learning Partnership. Teaching Resources: Assessment. http://www.service-learningpartnership.org/teaching/Assessment.cfm
• Pearson, Sarah S. Finding Common Ground: Service-Learning and Education Reform: A Survey of 28 Leading School Reform Models. American Youth Policy Forum, 2002.  https://www.aypf.org/resource/finding-common-ground-service-learning-and-education-reform/​
• Skinner, R. and Chris Chapman. "Service-Learning and Community Service in K-12 Public Schools." Education Statistics Quarterly. National Center for education Statistics, 1999.  http://nces.ed.gov/programs/quarterly/vol_1/1_4/3-esq14-h.asp
• Skloot, Edward. Service Learning: The View from an Engaged Foundation. Surdna Foundation, 1999. https://surdna.org/speeches/service.html
• United Way of America. Measuring Program Outcomes: A Practical Approach. Alexandria, VA: United Way of America, 1996.

This paper was developed by a student taking a Philanthropic Studies course taught at The Center on Philanthropy at Indiana University. The class is offered by Learning to Give and The Center on Philanthropy at Indiana University.