Last updated 3 months ago. Our resources are updated regularly but please keep in mind that links, programs, policies, and contact information do change.

Every year, the federal government spends trillions of taxpayer dollars on programs designed to improve education, healthcare, infrastructure, and countless other public services. But how do we know if these programs actually work?

Program evaluation provides the answer. It’s a systematic method government agencies use to determine whether their programs achieve their goals efficiently and effectively. Think of it as a thorough health check-up for government efforts.

The U.S. Government Accountability Office defines program evaluation as “an assessment using systematic data collection and analysis of one or more programs, policies, and organizations intended to assess their effectiveness and efficiency.” This definition highlights two core objectives: determining if programs achieve their intended results and if they use resources wisely.

Program evaluation serves as a key mechanism for ensuring government accountability to the public and taxpayers. When programs undergo evaluation, the findings can lead to improved services, more efficient resource allocation, and increased transparency.

Federal bodies like the Centers for Disease Control and Prevention, the GAO, and the Department of Health and Human Services consistently emphasize evaluation for accountability, improvement, and informed decision-making. For citizens, this means there’s a recognized system designed to scrutinize and enhance government programs.

The widespread practice of making evaluation findings public transforms individuals from passive service recipients into informed stakeholders equipped to engage in democratic oversight. Agencies like USAID and HHS regularly publish their evaluation findings, as do guidance platforms like Digital.gov that promote transparent reporting.

Two Essential Types of Evaluation

Among the various evaluation approaches, two stand out as particularly significant: formative evaluation and summative evaluation. While these terms might sound technical, their purposes are straightforward and vital for making government programs work better for everyone.

Formative Evaluation: Building Better Programs

What Is Formative Evaluation?

Formative evaluation functions like a series of check-ups or progress reports conducted while a program is still being developed or implemented. Its primary objective is to gather feedback that can improve the program as it unfolds, rather than waiting until the end to identify issues.

The GAO defines formative evaluation as “an evaluation that is conducted when researchers want to examine the extent to which a program is being implemented as intended, producing expected outputs, or could be improved.”

The CDC elaborates that formative evaluation aims to assess “whether a program, policy, or organizational approach is feasible, appropriate, and acceptable before it is fully implemented” and “focuses on learning and improvement.”

A helpful analogy comes from the Administration for Children and Families Formative Evaluation Toolkit: imagine a chef preparing a new soup. The chef tastes the soup multiple times during cooking (this is the formative evaluation), adjusting seasonings like salt and spices, or adding other ingredients as needed to perfect the recipe before serving it to guests.

The core purpose is to enhance efficiency, effectiveness, and overall success by identifying and addressing challenges early. This proactive approach serves as crucial risk mitigation for government programs. By pinpointing problems early, formative evaluation helps prevent minor issues from escalating into significant failures or inefficient uses of public funds that might only be discovered at a program’s conclusion.

When It Happens

Formative evaluations typically occur during the early to mid-stages of a program’s implementation. They can even begin during initial planning and design phases, allowing for adjustments before significant resources are committed.

The fundamental idea is to make necessary changes while there’s still ample time to influence the program’s ultimate outcome. The earlier and more consistently these evaluations are performed, the greater the chance of steering the program toward success.

Key Questions Formative Evaluation Answers

Formative evaluations provide real-time understanding of how a program functions. They seek to answer critical operational and developmental questions:

• Is the program being implemented according to the original plan?
• How can the program be improved, and how can existing strengths be built upon?
• Is the program approach feasible, appropriate for the context, and acceptable to the target audience and stakeholders?
• What types of data need to be collected to effectively monitor progress and success?
• Is the program successfully reaching the intended individuals or communities?
• What challenges, roadblocks, or unexpected issues are emerging during implementation?
• What specific modifications could enhance the program’s operation and effectiveness?

Common Methods

Formative evaluation often employs qualitative methods to gather rich, contextual insights into program operations and stakeholder experiences. However, it can also incorporate quantitative measures.

Interviews involve in-depth conversations with program staff, participants, administrators, and other relevant stakeholders to gather detailed perspectives.

Focus Groups are facilitated discussions with small, diverse groups to explore opinions, attitudes, and experiences related to the program.

Observations involve directly watching program activities as they occur to understand implementation fidelity and identify practical challenges or successes.

Pilot Testing implements the program on a small scale before a full rollout to test its components and gather initial feedback.

Peer Review has experts in the relevant field review program plans, materials, or early implementation efforts to provide constructive criticism.

Surveys collect standardized feedback from larger groups of participants or stakeholders to assess perceptions or gather specific data points.

Process Evaluation is a specific type of formative evaluation that meticulously assesses how well the program’s implementation aligns with its original plan. It examines the content, quality, quantity, and structure of program activities. The GAO clarifies that process evaluation “assesses the extent to which essential program elements are in place and conform to statutory and regulatory requirements, program design, professional standards, or customer expectations.”

The emphasis on qualitative methods and assessing acceptability underscores that formative evaluation isn’t solely an internal review but a process that deeply values stakeholder input. Engaging program participants, staff, and community members through interviews and focus groups is crucial for understanding real-world feasibility, identifying unforeseen challenges from diverse viewpoints, and ensuring the program aligns with the actual needs of those it aims to serve.

Real-World Government Examples

Education Programs

In educational settings, teachers frequently use formative assessment to monitor student understanding during lessons and adapt their teaching strategies in real-time to meet student needs. At a broader level, the U.S. Department of Education supports evaluations designed to guide program improvement.

An example includes efforts to refine technology-based approaches for assessing children with disabilities, ensuring these tools are effective and appropriate. These initiatives, detailed in Federal Register notices, demonstrate how formative evaluation helps adapt educational tools before widespread implementation.

Public Health Initiatives

The CDC employs formative evaluation to determine if a public health program or policy is feasible, appropriate, and acceptable before wide implementation. For instance, a new health communication campaign aimed at promoting vaccination might be pilot-tested with a sample of the target audience to refine its messaging and delivery channels for maximum impact.

The Rural Health Information Hub, a national clearinghouse on rural health issues, notes that formative evaluation is used in health promotion and disease prevention programs to provide information that aids in achieving program goals or improving the program itself.

Youth Programs

The Administration for Children and Families, an agency within HHS, funded the Youth At-Risk of Homelessness (YARH) program. During YARH’s initial phases, grantees utilized planning periods and formative evaluation techniques, such as usability testing and assessment of early outcomes, to refine their service models before any large-scale summative evaluation was considered.

This iterative process is crucial, as formative evaluation can help determine if a program is stable and well-defined enough for a meaningful final judgment. ACF also provides resources like the Formative Evaluation Toolkit to assist child welfare agencies in evaluating programs during their early implementation stages specifically for improvement purposes.

Department of Labor Initiatives

Formative evaluation helps the DOL and its partners understand how to effectively translate research findings into practical applications that improve service quality. In contexts similar to DOL-supported vocational training or employment programs, formative evaluation is used to study the complexities of implementing new strategies, identify factors influencing progress, and suggest necessary adaptations to optimize success.

Environmental Protection Agency Programs

While the EPA might not always explicitly use the term “formative evaluation” in every document, the underlying principles of reviewing processes and making adjustments for improvement are evident.

For example, the EPA’s review of Cumulative Risk Assessment case studies to identify “lessons learned” that can inform future implementation has a strong formative quality—learning from past and ongoing work to enhance future efforts.

Similarly, when the EPA amends procedural rules for chemical risk evaluation based on accumulated experience and public comments, this reflects a formative approach to refining regulatory processes.

Benefits and Challenges

Benefits

Formative evaluation enables early identification and correction of problems, preventing them from becoming larger issues. It increases the likelihood of program success by facilitating real-time adjustments and improvements.

The approach helps ensure programs are relevant, appropriate, and acceptable to target audiences and stakeholders. It fosters a culture of continuous improvement, organizational learning, and adaptability.

Challenges

The process can be time-consuming, requiring significant time investment to regularly assess progress, collect feedback, and implement changes.

Objectivity presents challenges when different stakeholders hold varying opinions on what aspects work well and what needs improvement, potentially leading to disagreements on necessary changes.

Resistance to change can occur when individuals or organizational systems hesitate to adapt or modify established procedures, even if formative evaluation findings indicate clear needs for improvement.

Summative Evaluation: Measuring Final Success

What Is Summative Evaluation?

Summative evaluation is conducted at the conclusion of a program, project, or significant phase to assess its overall effectiveness, outcomes, and ultimate impact. It’s about making a comprehensive judgment on the program’s merit, worth, or value based on collected evidence.

The GAO defines summative evaluation as “an evaluation that is conducted when researchers want to determine the extent to which a program has achieved certain goals, outcomes, or impacts.”

Using the earlier cooking analogy, if formative evaluation is the chef tasting and adjusting soup during preparation, then summative evaluation is like a food critic’s final, published review of the meal after it has been served to diners. Alternatively, it can be likened to a student receiving their final grade on a report card, summarizing their performance over an entire academic period.

The core purpose is to determine whether the program successfully achieved its stated goals and objectives, measure its impact on the intended target audience or broader community, and inform critical decisions regarding future program development, continuation, modification, or funding.

The findings aren’t merely for internal review; they’re intended as crucial inputs for high-level decision-making that can shape program futures and influence broader policy directions. This makes summative evaluation a powerful tool for ensuring government efforts evolve based on solid evidence, aiming for more effective and efficient public services long-term.

When It Happens

Summative evaluations typically occur at the end of a policy’s or program’s designated implementation period or at the conclusion of a grant cycle. This timing allows for assessment of the cumulative effects and achievements of the program.

Key Questions Summative Evaluation Answers

Summative evaluations concentrate on end results and overall accomplishments. Key questions include:

• Did the program achieve its intended goals and objectives as originally defined?
• What was the overall effectiveness of the program in addressing the identified need or problem?
• What measurable impact did the program have on its intended participants, beneficiaries, or the wider community?
• Based on its performance and impact, should the program be continued, replicated in other settings, expanded, scaled back, or discontinued?
• What was the ultimate value, merit, or worth of the program, considering its costs and benefits?
• Can the observed outcomes be confidently attributed to the program itself, rather than to other external factors?

This last question of attribution is often a central focus of impact evaluation, a specific type of summative evaluation.

Common Methods

Summative evaluations often employ a combination of quantitative and qualitative methods to provide comprehensive assessment of a program’s impact.

Quantitative Data Analysis involves analyzing numerical data such as test scores in education programs, employment rates for job training initiatives, or health statistics for public health interventions.

Surveys collect data from large and representative samples of participants, stakeholders, or target populations to gauge outcomes, satisfaction, or other relevant measures.

Standardized Tests assess participant performance or knowledge against pre-established benchmarks or standards, commonly used in educational evaluations.

Outcome Evaluation measures how well a program has achieved its intended short-term, intermediate, and long-term outcomes. The GAO defines it as assessing “the extent to which a program achieves its outcome-oriented objectives.”

Impact Evaluation is a more rigorous form of outcome evaluation that aims to determine the net effect of a program by comparing outcomes of those who participated with an estimate of what would have happened in the program’s absence. This often involves control or comparison groups and helps establish causality—whether the program itself caused the observed changes.

Cost-Benefit Analysis/Economic Evaluation examines the relationship between a program’s effects (benefits) and its costs to determine if the program represents a worthwhile and efficient use of resources. The GAO defines cost-benefit analysis as “a method of identifying and comparing relevant quantitative and qualitative costs and benefits associated with a program or activity, usually expressed in monetary terms.”

Performance Indicators are specific, measurable criteria defined at a program’s outset and used to assess whether the program successfully met its objectives. Clear and relevant performance indicators are crucial for conducting objective and meaningful summative evaluation.

The success and utility of summative evaluation heavily depend on the clarity of pre-defined program goals and measurable indicators. If objectives are vague or unmeasurable, the evaluation lacks a clear benchmark against which to judge performance, underscoring the importance of laying this groundwork during program design and formative stages.

Real-World Government Examples

Education Programs

Statewide assessments, such as the Iowa Statewide Assessment of Student Progress, serve as summative tools to meet federal accountability requirements under laws like the Every Student Succeeds Act (ESSA).

The National Assessment of Educational Progress (NAEP), often called “the Nation’s Report Card,” provides summative data on student achievement across the U.S.

The U.S. Department of Education also commissions summative evaluations of various federally funded programs to understand their long-term impacts, such as those examining the effects of initiatives like Parent PLUS loans or income-driven student loan repayment plans.

Youth Homelessness Prevention

The U.S. Department of Health and Human Services, through its Administration for Children and Families, is spearheading a federally-led summative evaluation of Colorado’s “Pathways to Success” model. This rigorous evaluation aims to determine the effectiveness of this comprehensive service model in preventing homelessness among youth who have been involved in the child welfare system. It includes both an implementation study and an impact study.

Workforce Development

A summative evaluation of a government-funded job training program would typically examine key outcomes such as employment rates and job retention statistics of participants after they have completed the program.

Environmental Programs

An evaluation of an environmental protection program might focus on its ultimate impact on specific environmental outcomes, such as successful protection of endangered species’ natural habitats or measurable reductions in greenhouse gas emissions attributable to the program’s interventions.

Health Programs

The Centers for Medicare & Medicaid Services subjects its Innovation Center models to rigorous evaluation. For instance, reports such as the “Primary Care First Model Options – Third Evaluation Report” and the “Kidney Care Choices (KCC) Model – Second Evaluation Report” assess the models’ impacts on critical factors like hospital admission rates, overall Medicare expenditures, and quality of patient care.

These reports often include accessible summaries like “Findings-At-A-Glance” and executive summaries to communicate key results. The public availability of such reports turns these documents into a public record of government achievements and areas needing improvement, allowing citizens and researchers to independently assess performance.

Benefits and Challenges

Benefits

Summative evaluation provides comprehensive and definitive assessment of a program’s overall effectiveness and impact. It informs critical, high-level decisions about future funding allocations, program continuation, expansion, or termination, and broader policy development.

The approach promotes accountability to funders, policymakers, and the public by providing evidence of program performance. It supports evidence-based decision-making, helping ensure public resources are directed toward effective interventions.

Summative evaluation can identify strengths and weaknesses that inform the design and improvement of future programs or policies.

Challenges

Data collection issues present significant challenges. Gathering comprehensive, accurate, and reliable data at a program’s end, especially for long-term outcomes, can be complex and resource-intensive.

Lack of clear performance indicators creates problems. If a program’s goals and measurable performance indicators weren’t clearly defined from the outset, assessing success in a summative evaluation becomes significantly more difficult.

Attribution poses major challenges. Determining whether observed outcomes were solely or primarily caused by the program, as opposed to other external factors or confounding variables, can be difficult. Rigorous impact evaluation designs attempt to address this but can be complex to implement.

Cost and time requirements are substantial. Conducting thorough and rigorous summative evaluations, particularly those involving large-scale data collection or complex impact analyses, can be expensive and take considerable time to complete.

Comparing Formative and Summative Evaluation

While both formative and summative evaluations are indispensable tools in governance and management of public programs, they serve distinctly different purposes and are employed at different stages and in different ways. A clear understanding of these distinctions is vital for appreciating how government programs are developed, continuously improved, and ultimately judged for effectiveness and value.

Purpose

Formative Evaluation primarily aims to improve and refine a program while it’s still in progress or under development. It’s fundamentally about learning, adaptation, and enhancement.

Summative Evaluation mainly seeks to judge overall effectiveness, outcomes, and impact of a program after it has been completed or reached a significant milestone. It’s about accountability, making conclusive assessments, and informing future decisions.

Timing

Formative Evaluation is conducted during the program’s development and implementation phases, often in early to mid-stages.

Summative Evaluation takes place after the program or a significant phase is complete, or at the end of a funding cycle.

Focus

Formative Evaluation concentrates on processes, how the program is being implemented, its operational feasibility, early outputs, and its acceptability to participants and stakeholders. It seeks to answer questions like “How is the program going?” and “How can we make it better right now?”

Summative Evaluation focuses on outcomes, overall impact of the program, the extent to which goals were achieved, and ultimate value or worth. It addresses questions like “Did the program work as intended?” and “What difference did it ultimately make?”

Primary Users of Findings

Formative Evaluation findings are primarily used by program managers, staff, and developers who are directly involved in the program and can implement immediate changes or adjustments.

Summative Evaluation results are typically intended for a broader audience, including funders, policymakers, the public, and other stakeholders who will use the information to make decisions about the program’s future, such as continuation, expansion, replication, or termination.

Types of Questions Asked

Formative Evaluation inquires about operational aspects: “Are activities being completed as planned?” “What are the barriers to implementation?” “Is the program acceptable and engaging for participants?”

Summative Evaluation addresses overall results: “Did the program achieve its stated objectives?” “What were the measurable short-term and long-term outcomes?” “Was the program cost-effective in achieving these outcomes?”

How They Work Together

Formative and summative evaluations aren’t mutually exclusive; rather, they’re most effective when used in a complementary fashion. Formative evaluation can play a crucial role in ensuring that a program is well-implemented, refined, and operating effectively, thereby setting the stage for a more meaningful and valid summative evaluation.

The insights gained from summative evaluation can then provide valuable lessons that inform the design, planning, and formative evaluation processes for future programs or subsequent iterations of existing ones. This creates a continuous cycle of learning, improvement, and accountability in public program management.

The distinction isn’t about which type is superior, but how they fulfill different, yet equally critical, roles in the lifecycle of a government program. Formative evaluation is about making the program effective during its course, while summative evaluation is about knowing if the program was ultimately effective and determining the path forward.

The way these evaluations influence resource allocation differs significantly. Formative evaluation findings often trigger internal, tactical reallocations of resources or adjustments in program activities to address immediate operational issues. In contrast, summative evaluation results can impact much larger, strategic resource allocation decisions, such as continuation of funding for entire programs, their expansion to new populations or regions, or their termination if found ineffective or inefficient.

Broader Context of Evaluation Types

Formative and summative evaluations are part of a larger family of evaluation approaches, each with specific nuances:

Process/Implementation Evaluation is often considered a component of formative evaluation. This type focuses intently on how a program is delivered, examining its activities, adherence to design, and operational aspects.

Outcome Evaluation is typically summative in nature and measures the extent to which a program has achieved its intended outcomes or results.

Impact Evaluation is a rigorous type of summative evaluation that seeks to determine whether the program itself caused the observed outcomes, often by comparing results to what would have happened in the program’s absence.

Economic Evaluation (like Cost-Benefit Analysis) assesses a program’s effects in relation to its costs, often as part of a summative assessment, to determine its financial viability and efficiency.

Needs Assessment is frequently conducted for formative purposes and helps understand the problem a program intends to address and the resources required for the program to achieve its goals successfully.

Side-by-Side Comparison

How Understanding Evaluation Helps Citizens

Asking the Right Questions

Knowledge of the distinctions between formative and summative evaluation—along with related concepts such as process, outcome, and impact evaluation—equips citizens to ask more insightful and pertinent questions about government programs operating within their communities and at the national level.

For instance, if a new local initiative is launched, an informed citizen can inquire whether formative evaluations are planned to monitor its early progress and ensure it stays on track. Similarly, if a long-standing public program is under review, one can ask about the findings and recommendations from its most recent summative evaluation. This ability to ask targeted questions fosters greater engagement and scrutiny.

Understanding How Tax Dollars Are Used

Program evaluation provides critical insights into whether government initiatives are delivering their intended benefits and are being managed in a manner accountable to taxpayers. Formative evaluation contributes by helping to prevent the misdirection or waste of resources on ineffective strategies during a program’s lifecycle.

Summative evaluation, in turn, helps determine if the overall financial and societal investment in a program was justified by its results. The CDC aptly notes that one of the key products of program evaluation is the ability to “document progress and ensure optimal use of resources.”

Promoting Government Transparency and Accountability

When citizens understand the principles and purposes of program evaluation, they’re better positioned to hold government agencies accountable for their performance and the outcomes of their programs. Publicly available evaluation reports serve as tangible evidence of what government entities are doing, how effectively they’re performing, and what improvements are being implemented based on findings.

Effective evaluation inherently supports transparency by making the methodologies, data, and findings of program assessments open to public examination and critique. The U.S. Government Accountability Office itself is a cornerstone institution in this regard, conducting numerous evaluations across the federal government to ensure accountability and provide Congress and the public with objective information.

This understanding allows citizens to move beyond simply receiving services to becoming active participants in ensuring government effectiveness. This can create a positive feedback loop: an informed public that asks pointed questions and demands accountability may encourage agencies to conduct more rigorous and transparent evaluations, which, in turn, can lead to better-designed and more effective programs. This dynamic elevates the role of the citizen from a passive observer to an active participant in the quality control of government.

Finding Public Evaluation Reports

Many U.S. government agencies are committed to transparency and make their evaluation reports accessible to the public. This allows individuals to review the evidence and results for themselves.

Agency Websites

Federal agencies such as the Department of Health and Human Services maintain dedicated sections for research and evaluation reports. For example, the Office of the Assistant Secretary for Planning and Evaluation at HHS provides access to numerous evaluations. The Substance Abuse and Mental Health Services Administration also shares program evaluations.

Similarly, the Centers for Medicare & Medicaid Services posts evaluation reports from its Innovation Center.

The Department of Education makes evaluation studies available through the National Center for Education Evaluation and Regional Assistance and its Office of Planning, Evaluation and Policy Development.

The U.S. Agency for International Development has historically published evaluations through its Development Experience Clearinghouse.

U.S. Government Accountability Office

The GAO is a primary source for evaluations of federal programs. Its reports, covering a vast spectrum of government operations, are published online and are searchable by topic or agency. Notable GAO resources on evaluation itself include “Program Evaluation: Key Terms and Concepts” and “Performance Measurement and Evaluation: Definitions and Relationships.”

Digital.gov

This resource hub for government digital services offers guides and case studies on how agencies can effectively report outcomes and performance, with an emphasis on transparency. Their guide on public participation evaluation provides valuable insights.

Federal Register

While not a repository of evaluation reports, the Federal Register publishes notices that can provide insights into how evaluation is integrated into government processes. These include announcements of funding opportunities that mandate evaluation components or proposed changes to program regulations that may be based on prior assessments or evaluations.

Our articles make government information more accessible. Please consult a qualified professional for financial, legal, or health advice specific to your circumstances.