SILC-NET Presents… A National Two-Part Teleconference & Webcast Measure Twice, Cut Once: An Introduction to Outcome Measures for Statewide Independent Living Councils Part I – October 2, 2008 (3:00 PM – 4:30 PM Eastern) Part II – October 7, 2008 (3:00 PM – 4:30 PM Eastern) Presentation Guide Materials Developed by: Mike Hendricks, Bob Michaels, Darrell Jones, Richard Petty, Eleanor Canter, Tim Fuchs © September 2008 National Council on Independent Living (NCIL) 1710 Rhode Island Ave. NW, 5th Floor Washington, DC 20036 202.207.0334 (V) 202.207.0340 (TTY) 202.207.0341 (FAX) ncil@ncil.org www.ncil.org This material is a publication of the SILC-NET, a program of the IL NET national training and technical assistance project for centers for independent living (CIL- NET) and statewide independent living councils (SILC-NET). The IL NET is operated by the Independent Living Research Utilization (ILRU) Program at TIRR Memorial Hermann in partnership with the National Council on Independent Living (NCIL) and the Association of Programs for Rural Independent Living (APRIL). Support for development of this publication was provided by the U.S. Department of Education, Rehabilitation Services Administration under grant number H132B070003. No official endorsement of the funder should be inferred. Permission is granted for duplication of any portion of this guide, providing that the following credit is given to the project: Developed as part of the SILC-NET, a program of the IL NET, an ILRU/NCIL/APRIL National Training and Technical Assistance Project. AGENDA Part 1: Developing Desired Outcomes in Your State 3:00 PM EDT – Welcome and introductions 3:05 PM EDT – Why write and measure outcomes? 3:25 PM EDT – Q&A Session 3:35 PM EDT – What’s the current state of SILC outcome measures? 3:40 PM EDT – What is an “outcome” anyway? 4:00 PM EDT – Q&A Session 4:10 PM EDT – How can you develop your desired outcomes? 4:20 PM EDT – Wrap-up and final Q&A 4:30 PM EDT – Presentation Ends Part 2: Measuring Your Desired Outcomes 3:00 PM EDT – Welcome and review of Part 1 3:05 PM EDT – Q&A Session 3:10 PM EDT – How can a SILC think about the relationships among various desired outcomes? 3:30 PM EDT – Q&A Session 3:40 PM EDT – How can a SILC measure desired outcomes? 4:00 PM EDT – Q&A Session 4:10 PM EDT – How can a SILC gather and use outcome data? 4:15 PM EDT – Wrap-up and final Q & A 4:30 PM EDT – Presentation Ends ABOUT YOUR TRAINER Mike Hendricks received his Ph.D. in Methodology and Evaluation Research from Northwestern University, and ever since, he’s worked to help programs become as effective as possible. In particular, Mike believes it’s very important that SILCs know exactly what we’re trying to achieve, have a way to measure how we’re doing, monitor our progress on a regular basis, and consistently use what we learn to improve our efforts. He’s successfully used this approach with a wide variety of clients – governments at all levels (city, county, state, national), foundations, national nonprofit associations, and local service delivery agencies. Mike is also a well-known and respected member of the program evaluation community. A recently-elected board member of the American Evaluation Association, he serves on the editorial board of the American Journal of Evaluation, speaks and publishes regularly, and is often invited to train others. Mike recently spent two weeks in England training front-line service providers in London and Manchester. MEASURE TWICE, CUT ONCE: AN INTRODUCTION TO OUTCOME MEASURES FOR STATEWIDE INDEPENDENT LIVING COUNCILS A Two-Part National Teleconference & Webcast Training Materials — Table of Contents Copyright & Usage Requirements Page ii Agenda & Training Introduction Page iii Table of Contents Page iv Measuring Outcomes in the Independent Living Movement Page 1 2003 PART Results — Independent Living Program Page 2 RSA’s 704 Report — FAQ Introduction Page 4 Why Measure Outcomes? Page 5 Who’s Focused on Outcomes? Page 6 Program Outcome Model Page 7 Sample Program Outcomes Page 9 Outputs vs. Outcomes Page 10 Sample, Illustrative-Only Possible Outcomes for a SILC Page 11 “Inputs Through Outcomes” Page 12 Activities to Outcomes — The Chain of Influence Page 13 Sample, Illustrative-Only Possible Logic Model for a SILC Page 15 Glossary of Key Terms Page 16 Ways A Logic Model is Useful By Itself Page 18 About Indicators Page 19 Sample Outcomes & Indicators Page 20 Key Traits of a Good Indicator Page 21 Sample, Illustrative-Only Possible Indicators for a SILC Page 22 Possible Sources of Outcome Data Page 24 Possible Methods of Collecting Outcome Data Page 25 Appendix A: Measuring Outcomes in the Independent Living Movement Page 26 Measuring Outcomes in the Independent Living Movement One of the most significant changes in the nonprofit world during the past few years has been the big emphasis on outcomes – outcomes being defined as changes in the persons, systems, or communities we work with. We can hardly turn around these days without hearing about outcomes, results, impacts, improvement, or accountability. Many of our colleagues in human services and advocacy are already identifying the outcomes they want to achieve, regularly measuring their progress, and using what they learn to improve their programs and to document their successes. The outcome measurement train isn’t coming down the track – this train’s already here. Those of us in the IL movement are ready, even eager, to jump on this train. We know without a doubt that we help the people we serve, and we know we change communities for the better. For that reason, we look forward to showing ourselves, our funders, our partner agencies, the persons we serve, and the general public that we do an excellent job. But we also know in our hearts that we could probably accomplish even more. So we welcome the chance to better define our hoped-for outcomes, to find ways to measure how well we’re doing, and to make improvements where they’re needed. Someone described not measuring program outcomes as like managing a baseball team, but without being able to see the scoreboard; or driving to a new destination, but without having a map. It’s simply not possible to manage well if we don’t know how we’re doing right now. Obviously it’s also important to use that information wisely when we have it, but first we have to know our outcomes. That's what this training session is all about. 2003 PART Results for the Independent Living Program PART Question -- (note that some questions have been shortened) OMB Answer Yes No Section 1 – Program Purpose and Design 1.1 Is the program purpose clear? X 1.2 Does the program address a specific and existing problem, interest or need? X 1.3 Is the program designed so that it is not redundant or duplicative of any other effort? X 1.4 Is the program design free of major flaws that would limit its effectiveness or efficiency? X 1.5 Is the program effectively targeted so that resources reach beneficiaries and address purpose? X Section score 100% Section 2 – Strategic Planning 2.1 Does the program have specific long-term performance measures that focus on outcomes? X 2.2 Does the program have ambitious targets and timeframes for its long-term measures? X 2.3 Are there annual performance measures that track progress toward the long-term goals? X 2.4 Does the program have baselines, targets, and timeframes for its annual measures? X 2.5 Do all partners commit to and work toward the program’s annual and/or long-term goals? X 2.6 Do independent evaluations improve the program and assess effectiveness and relevance? X 2.7 Are Budget requests tied to accomplishing the annual and long- term performance goals? X 2.8 Has the program taken meaningful steps to correct its strategic planning deficiencies? X Section score 12% 2003 PART Results for the Independent Living Program OMB Answer Section 3 – Program Management Yes No 3.1 Does the agency regularly collect and use timely and credibly performance information? X 3.2 Are Federal managers and program partners accountable for cost, schedule and performance? X 3.3 Are all funds obligated in a timely manner and spent for the intended purpose? X 3.4 Does the program measure and achieve efficiencies and cost effectiveness? X 3.5 Does the program collaborate and coordinate effectively with related programs? X 3.6 Does the program use strong financial management practices? X 3.7 Has the program taken meaningful steps to address its management deficiencies? X 3.CO1 Are grants awarded based on a competitive process that includes an assessment of merit? X 3.CO2 Does the program have oversight practices that provide knowledge of grantee activities? X 3.CO3 Does the program annually collect and make available grantee performance data? X Section score 40% Section 4 – Program Results/Accountability 4.1 Has the program shown progress in achieving its long-term outcome performance goals? X 4.2 Does the program (including program partners) achieve its annual performance goals? X 4.3 Has the program improved its efficiency or cost-effectiveness each year? X 4.4 Does the program’s performance compare favorably to relevant other programs? No studies 4.5 Do independent evaluations indicate that the program is effective and achieving results? Small extent Section score 8% Overall rating ------- “Results Not Demonstrated” (Included under “Not Performing”) Revised 704 Annual Performance Report Frequently Asked Questions (FAQ) In this document, the Rehabilitation Services Administration (RSA) offers answers to substantive questions posed by representatives of centers for independent living (CILs), statewide independent living councils (SILCs) and designated state units (DSUs) regarding the revised annual section 704 Report. Part I of the 704 Report is completed jointly by DSUs and SILCs. Part II of the 704 Report is completed by CILs. This FAQ document is a follow-up to the National Teleconference and Webcast conducted by Independent Living Research Utilization (ILRU) and RSA in August 2006. Areas addressed here are taken from a variety of sources including questions received directly by RSA as well as those posted on the national training discussion board. RSA would like to thank the many individuals who contributed to this process, with special thanks to Ms. Julia Sain for her efforts in this training and subsequent discussion board activity. The revised 704 Report incorporates newly established performance goals and measures for the Independent Living (IL) program. These measures aim to better reflect the IL impact on individuals and the community. The primary difference in the new 704 Report is its increased focus on “outcomes,” in comparison with “outputs.” Outputs represent the independent living activities and services provided by centers for independent living, DSUs and other IL service providers. Outcomes represent the impact that the IL services and activities have on the consumer, that is, the extent to which consumers achieved greater independence and community integration. Outcomes are the ends, whereas outputs are the means. Both are important yet, ultimately, the independent living program will be evaluated based on its outcomes. Full document is available at: http://www.ilru.org/html/publications/704/704_Report_FAQ_Final.doc * Who’s Focusing on Outcomes? * Government at all levels * National nonprofit associations * Local nonprofit agencies * Foundations * Accrediting bodies * Private sector * International development agencies Outputs versus Outcomes: What’s the Difference? Program Possible Outputs (volume of work accomplished) Possible Outcomes (changes in people) Comprehensive child care * Number of children cared for * Number of child- days of care * Children exhibit age-appropriate physical, mental, and verbal skills * Children are school-ready for kindergarten General Equivalency Diploma (GED) preparation * Number of participants tutored * Number of tutoring sessions held * Number of participants tested and re-tested * Participants obtain their GED certificates * Within 6 months after obtaining their GED, participants are employed full- time Congregate meals for senior citizens * Number of seniors served lunch * Number of seniors led in exercise programs * Number of tables of seniors organized for after-lunch games * Seniors interact socially with peers * Seniors are not home-bound * Seniors eat a nutritious and varied diet * Seniors exhibit fewer health problems Sample, Illustrative-Only Possible Outcomes for a SILC Possible outcomes likely to be within a SILC’s “reasonable influence” * Needs of PWD in this state are known * All relevant organizations (public, private, nonprofit) collaborate on IL issues * A state plan exists to enhance IL services * Both PWDs and non-PWD citizens support the IL network * Implementation of the state plan is monitored, evaluated, and revised as necessary * There are sufficient IL resources and organizational capacities * The IL network is enhanced and expanded * New options are available for PWD Possible outcomes likely to be outside a SILC’s “reasonable influence” * PWD experience greater dignity, inclusion, independence, nondiscrimination, and productivity * PWD maximize their independence and community participation Glossary of Key, SILC-Related Terms Used in Measuring Outcomes Term Definition (with SILC-related examples) Activities The efforts the SILC makes; the actions a SILC takes; the things SILC staff and members do with their time; often described with “-ing” words, such as convening key stakeholders, conducting a needs assessment, etc. Frequency How often an indicator is measured; some indicators are best measured monthly, while others are best measured annually; the frequency should be determined on an indicator-by-indicator basis Goals The mid-level desired outcomes a SILC aims to achieve, ranking above the objective and under the mission; achieving the goals contributes to achieving the mission; possible goals might be “The CIL network is enhanced and expanded” or “New options are available for PWDs” Indicators Also called measurable indicators; the specific items of information we measure in order to learn about (indicate) progress on a desired outcome, since outcomes cannot be measured directly; each desired outcome has its own, unique set of indicators; e.g., an indicator of the desired outcome “The CIL network is enhanced and expanded” might be “# and location of new CILs opened in the state during the past year” or “# and % of persons with disabilities living on September 30 in areas of the state designated as unserved or underserved” Methods How information for an indicator is gathered from a source; sometimes there can be several possible methods we can use to gather information from a source; it is important to choose the best possible method for gathering information from a source; for example, if persons with disabilities are a source of information for an indicator, possible methods to gather the needed information from those persons might include a written questionnaire, a telephone interview, a face-to-face interview, an email, etc. Mission The top-level desired outcomes a SILC aims to achieve; the very top of the logic model; the ultimate reasons a SILC exists; the final product of a SILC conducting its activities and achieving its objective and goals; many times the actual mission is outside the reasonable influence of a SILC; possible missions might be “Persons with disabilities maximize their independence and community participation” or “Persons with disabilities experience greater dignity, inclusion, independence, nondiscrimination, and productivity” Glossary of Key, SILC-Related Terms Used in Measuring Outcomes Term Definition (with SILC-related examples) Objectives The first-level desired outcomes a SILC aims to achieve; those outcomes which are achieved directly from a SILC’s activities; achieving the objectives contributes to achieving the goals; possible objectives might be “Needs of persons with disabilities in this state are known” or “All relevant organizations (public, private, nonprofit) collaborate on independent living issues” Outcomes Also called desired outcomes; the desired end results of a SILC’s efforts; the benefits or changes the SILC aims to achieve as a result of its activities; the improvements in individuals and systems the SILC hopes to effect; the reasons why the SILC conducts its activities; outcomes are often described as existing at different levels, most commonly initial outcomes (objectives), intermediate outcomes (goals), and ultimate or top-level outcomes (mission); on a logic model, outcomes include everything above the SILC’s activities Sources The location where information about indicators can be found; where a SILC needs to go to “extract” information in order to measure an indicator; possible sources of information for indicators might be persons, documents, events, natural conditions, etc. Strategy The combined set of activities a SILC undertakes in order to achieve its desired outcomes; what a SILC does, overall, to accomplish its objectives, goals, and mission; a strategy is made up of several different activities * Ways a Logic Model Is Useful By Itself * Create a shared vision of the program * Make the program’s theory very explicit * Recruit, orient, and train staff * Help allocate resources properly * Communicate the program’s intent to stakeholders * Explain the program to potential clients * Negotiate “fair” accountability with funders and others * Guide a systematic review of outcome data in order to identify improvement opportunities Outcome Indicators The specific things we measure in order to show how fully the desired outcome is being achieved. Key Traits of a Good Indicator A good indicator is SMART: Specific –The indicator is clear enough that different people would measure the same thing in the same way Measurable – The necessary information can be observed, counted, or weighed with reasonable efforts and costs Ambitious –The indicator sets a high standard for program participants to achieve the desired outcomes Realistic – The standard is not so high that program participants cannot achieve it with reasonable effort Timely –The indicator is measured during an appropriate period of time to provide useful information Sample, Illustrative-Only Desired Outcomes and Measurable Indicators for a SILC Sample, Illustrative-Only Desired Outcomes Sample, Illustrative-Only Measurable Indicators for each Outcome 1. Needs of PWD in this state are known ? # of months since the SILC last undertook a formal needs assessment effort to learn about the needs of PWDs in the state ? # of needs assessment activities attempted by the SILC during the last federal fiscal year ? # and % of persons with disabilities (PWD) surveyed regarding their needs ? Other, better indicators? 2. All relevant organizations (public, private, nonprofit) collaborate on issues related to the statewide program of independent living ? Did the SILC, DSU, CILs, and other relevant public agencies, along with relevant private and nonprofit organizations, all participate in developing and implementing the current state plan? ? # and types of organizations consulted during the SPIL development process ? Other, better indicators? 3. A state plan exists to enhance services under the statewide program of independent living ? Have the SILC and DSU signed a formal planning document as per the timetable required by RSA? ? Has the SPIL been reviewed and approved by RSA? ? Other, better indicators? 4. Both PWDs and non- PWD citizens support the statewide program of independent living ? % of PWDs and non-PWDs (measured separately) responding positively to the question “Do you support using state resources to expand independent living opportunities for PWDs?” ? $ contributed by the public during the past federal fiscal year for the statewide program of independent living ? # of volunteer hours contributed by the public during the past federal fiscal year for the statewide program of independent living ? Other, better indicators? Sample, Illustrative-Only Desired Outcomes and Measurable Indicators for a SILC 5. Implementation of the state plan is monitored, evaluated, and revised as necessary ? # of months since all signers of the state plan met to discuss progress on the plan and changes that might be needed ? # and % of SILC meetings during which SPIL goals and objectives are reviewed and modified if necessary ? Other, better indicators? 6. There are sufficient resources and organizational capacities for the statewide program of independent living ? $ available for the statewide program of independent living during the past federal fiscal year ? Do SILC minutes indicate that funding gaps are identified and prioritized and steps taken to address those with highest priority? ? Other, better indicators? 7. The IL network is enhanced and expanded ? # of CILs operational during the past federal fiscal year ? # and location of new centers opened during the past federal fiscal year ? Total $ allocated to all CILs within the state ? # and % of PWDs living in unserved or underserved areas of the state ? # and % of I&R requests in areas designated as unserved or underserved ? Other, better indicators? 8. New options are available for PWDs ? What new statutes, policies, or codes were created within the state during the past federal fiscal year to support the independence of PWDs? ? # and % of PWDs who report that they have services that address their needs ? Other, better indicators? Possible Sources of Program Outcome Data Existing Information • Case/client records • Documents of various types • Data files (paper or electronic) Individuals • Program participants • Others who know participants • Other individuals/general public Physical/environmental conditions • Physical environment Possible Methods of Collecting Program Outcome Data • Data file review • Document review • Record review • Questionnaire • Interview • Observation • Testing • Mechanical measurement Appendix A: Measuring Outcomes in the Independent Living Movement The following article was developed as part of IL NET’s 2007 Teleconference & Webcast: IL & Outcome Measures: What You Need to Know. That presentation focused on the basic principles of outcome measurement as well as the work of NCIL’s Outcome Measures Task Force. While that presentation and the work of the Task Force were geared towards outcome measures for Centers for Independent Living (CILs), we believe the following article still contains valuable information about the outcome measurement process as it relates to Statewide Independent Living Councils (SILCs). This article is also available at: http://www.ilru.org/html/publications/outcome_measures/ncil-website-blurb-for- ILNet-2.doc One of the most significant changes in the nonprofit world during the past few years has been the big emphasis on outcomes – outcomes being defined as changes in knowledge, skills, attitudes, values, behavior, condition, or status of the persons or communities we work with. We can hardly turn around these days without hearing about outcomes, results, impacts, improvement, or accountability. Many of our colleagues in human services and advocacy are already identifying the outcomes they want to achieve, regularly measuring their progress, and using what they learn to improve their programs and to document their successes. The outcome measurement train isn’t coming down the track – this train’s already here. Those of us in the IL movement are ready, even eager, to jump on this train. We know without a doubt that we help the people we serve, and we know we change communities for the better. For that reason, we look forward to showing ourselves, our funders, our partner agencies, the persons we serve, and the general public that we do an excellent job. But we also know in our hearts that we could probably accomplish even more. So we welcome the chance to better define our hoped-for outcomes, to find ways to measure how well we’re doing, and to make improvements where they’re needed. Someone described not measuring program outcomes as like managing a baseball team, but without being able to see the scoreboard. Or driving to a new destination, but without having a map. It’s simply not possible to manage well if we don’t know how we’re doing right now. Of course, it’s also important to use that information wisely when we have it, but first we have to know our outcomes. For that reason, NCIL has launched a 16-person Outcomes Task Force to move our IL community more firmly into measuring outcomes. This task force has been operating for several months, and we expect it to continue for the foreseeable future. There’s a lot of work ahead, but we’re making steady progress. In order to bring you up-to-date on our work and to seek your input, we’ve placed several products on this website: this brief introduction, a draft logic model of the CIL program , and a way you can help us decide what to measure in order to gauge our progress (our measurable indicators) . Below we will explain both logic models and measurable indicators. Measuring outcomes starts with three key steps: (1) developing a logic model of our program, (2) identifying the outcomes we want to measure, and (3) creating measurable indicators of each outcome. Step One: A Draft Logic Model of the CIL Program The phrase “logic model” is jargon, to be sure, but it’s also a phrase that’s becoming more common all the time. Perhaps you’ve heard it from a government agency, your local United Way, or a foundation, since more and more funders are routinely asking their grantees to develop logic models. But what exactly is a logic model? The answer is pretty simple: A logic model is a way to show visually, on one page, what a program does (its activities), what it’s trying to achieve (its desired outcomes), and in what order (the different levels of outcomes). The lines connecting the boxes show how each of these pieces fit together. The Outcomes Task Force has created a draft logic model for the CIL program, so let’s learn from that example. As you can see, the bottom row (yellow) shows the CIL program’s three main activities: IL services, I&R, and systems advocacy. The rest of the page shows the CIL program’s desired outcomes, divided into the levels of initial outcomes (orange), intermediate outcomes (red), and ultimate outcomes (blue). Working up the logic model from the bottom row, immediately above each activity is the initial outcome that each activity hopes to produce. For example, the initial outcome of systems advocacy is that “barriers and problems are identified”. But why? Why do we want to identify barriers and problems? The answers are in the two outcomes right above this box – we want to identify barriers and problems so that “A consumer agenda for change exists” and “Active coalitions exist around our issues”. But why do we want a consumer agenda for change to exist? Again, the answer is right above that box – we want a consumer agenda for change to exist so that “Decision-makers agree with our agenda”. Using slightly different words, but looking at the same boxes, we see that IF we do systems advocacy, THEN barriers and problems will be identified. And IF barriers and problems are identified, THEN a consumer agenda for change will exist AND active coalitions will exist around our issues. And IF a consumer agenda for change exists, THEN decisions-makers will agree with our agenda. In this same way, we can ask this why? question and draw these IF-THEN connections all over the logic model. Doing so shows us very clearly what the CIL program is all about. Step Two: Desired Outcomes of the CIL Program Our CIL logic model contains 20 different boxes. Three of these boxes represent our three different activities, and the remaining 17 boxes represent our 17 different desired outcomes. That is, the CIL program as a whole wants to achieve 17 different outcomes. Some of these outcomes are for individuals (the left side and middle of the logic model) and some of these outcomes are for communities (the right side of the logic model), but each of the 17 outcomes is important. Ideally we would like to measure our progress on all 17 desired outcomes, but in the real world we probably won’t have the time, money, or people to measure them all. Eventually we’ll need to pick and choose which ones to measure. It may be that all CILs measure the same outcomes, or this may vary from CIL to CIL, depending on the local interests. At this stage, our Task Force isn’t deciding which outcomes we should measure – for now, we’re looking at all 17. Step Three: Measurable Indicators of Desired Outcomes In measuring outcomes, the “rubber meets the road” when we wrestle with how to measure our progress on a desired outcome. Until we do this, we have warm, fuzzy concepts (“Communities are fully accessible”), but how do we measure that? How do we know the extent to which a community is fully accessible without measuring something? For example, maybe we measure “the number and percentage of multifamily housing properties in our community that meet fair housing requirements”. Or maybe we measure “the length of waiting lists for Section 8 vouchers or affordable, accessible housing”. Or maybe we measure something else – but we must measure something specific, and what we measure is called our measurable indicators. Our Task Force is now wrestling with this step of the process – we are creating a laundry list of possible measurable indicators for each of the 17 desired outcomes of our logic model. We have some first ideas, but we need more. Elsewhere on this website is an easy way for you to give us your ideas for how to measure CIL outcomes, and we hope you’ll give us lots of ideas. Don’t worry about the exact form or wording of your suggestions – we have experts to help us with that. What we need are your creative, real-world ideas of what might be good to measure. Please give us as many ideas as you possibly can, and we promise that the Task Force will consider each one seriously. Conclusion … For Now Measuring outcomes is important for the IL movement. We agree wholeheartedly with this emphasis, and our Task Force is working hard to develop ways for us to do this. We need your help in order to capture and reflect the reality of local CILs, so please do give us your best ideas. For our part, we promise to keep you fully up-to- date on our progress. Please watch this space for future postings. 7 i