>> SLIDE 1 IL-NET: CIL-NET + SILC-NET; A Project of ILRU Practical Measures: Telling Your Story through Outcome Measures November 12, 2014 Presenters: - Dan Kessler - Pat Puckett - Paul Spooner >> SLIDE 2 Objectives Upon completion of this webinar, participants will have knowledge and resources which will enable them to? - Describe the Outcome Measures Task Force tool that can be used by Centers to measure and understand outcome findings - Describe reporting and marketing opportunities that come from outcomes measurement - List examples that illustrate the value of outcome measures beyond Federal and/or funder requirements >> SLIDE 3 Why Might Your CIL be Interested in Outcome Measures? - Your findings will help you share your success story with your constituents, your community, funders and others - Outcome measures are a tool that can help you improve your own effectiveness -- Help you know if you’re really helping -- Help you focus on what’s important -- Help you know if your advocacy is effective >> SLIDE 4 Why Centers Began to Work on Outcomes - Centers were calling for a better way to capture their accomplishments—to tell our own story—in a consistent manner. - The Rehabilitation Services Administration had begun an increased focus on outcomes. - 2003—PART concluded “Results Not Demonstrated.” - Funders increased emphasis on outcomes and accountability. >> SLIDE 5 Background?NCIL Task Force on Outcome Measures - Extensive work between 2006 and 2011 - Bob Michaels Chaired the Task Force - Joint project of NCIL, ILRU, Univ. Kansas - Involved over 40 Centers - Final Report – http://www.ilru.org/sites/default/files/resources/outcome_measures/Focusing_on_Outcomes_in_the_CIL_Program.pdf >> SLIDE 6 What Centers Learned - First, and very importantly, it is possible to measure the outcomes of Center programs, and the benefits are worth the effort. - The Centers’ second message, however, was that focusing on Center outcomes is not simple. Not all Centers were ready. >> SLIDE 7 What Centers Learned, cont’d. 2 - Seventy-eight percent (78%) of CILs needed 10 or fewer hours to gather information about their own CIL’s outcomes - 93% needed 10 or fewer hours to enter all the outcome information into Survey Monkey. - Sixty-seven percent (67%) of CILs thought it was easy or very easy to ask questions of consumers. - 70% thought it was easy or very easy to ask questions of I&R callers. >> SLIDE 8 Indicators Define What We Mean by Outcomes - An indicator is a specific item of information that defines what we mean by the outcome - Sometimes we want to achieve things - Sometimes we want to prevent things >> SLIDE 9 Questions & Answers >> SLIDE 10 Logic Model for the CIL Program As an overview of the logic model, three types of CIL activities are represented by three separate boxes across the bottom of the page. These CIL activities are, from left to right: (1) IL services, (2) Information and Referral (I&R), and (3) Systems Advocacy. As a result of conducting these three activities, the CIL program achieves a total of 15 initial, intermediate, and ultimate outcomes. These different outcomes are represented by 15 different boxes that are arranged (“flow”) upward, on five different levels, from the three activity boxes. The combined effect of achieving these 15 outcomes eventually leads to the 16th and top-most outcome, which reads “People with disabilities are integrated into American Society.” In other words, 15 outcome boxes flow upward from the CIL’s three activities, and these 15 outcomes converge to achieve one ultimate outcome at the very top of the page. Delving into the details of the different outcomes as they flow upwards, we can use an “If – Then” logic to work systematically up the page from each activity to the ultimate outcomes. Although most outcomes flow from one set of activities directly upward, there is also some crossover and intermingling of these three streams. Beginning on the left side of the page, CILs conduct IL Services. Working upwards, because of these IL services, persons with disabilities have the skills, knowledge, and resources to support their choices. If they have these skills, knowledge, and resources, then they will make their own choices. If they make their own choices, then they will simultaneously regard themselves as more independent and also will, in fact, be more independent. If they both regard themselves as more independent and, in fact, are more independent, then persons with disabilities will participate in communities to the extent they wish. This outcome represents the top of the IL services stream. In the middle of the page is the Information and Referral stream. Because of I&R, persons with disabilities both get the information they need and simultaneously see different possibilities. These two outcomes are achieved mostly due to I&R, but IL services also contribute. If persons with disabilities see different possibilities and also get the information they need, then they will advocate for increased community supports. This outcome represents the top of the direct I&R stream, although this stream also merges with the final stream to be discussed. On the right side of the page is the Systems Advocacy stream. The first step in systems advocacy is to identify barriers and problems. If these barriers and problems are identified, then two different outcomes will be achieved -- a consumer agenda for change will exist, and active coalitions will exist around our issues. These two outcomes then diverge into two different directions. First, if a consumer agenda for change exists, this outcome merges with the I&R stream and helps persons with disabilities to advocate for increased community supports. At the same time, the existence of a consumer agenda, coupled with the existence of active coalitions, plus the advocacy efforts of persons with disabilities (from the previous stream) leads decision makers to act on our agenda. If decision makers act on our agenda, then two outcomes will result: communities will have more resources that support independence, and methods and practices will promote independence. If these two outcomes are achieved, then communities will be more accessible in terms of housing, transportation, information, employment, education, assistive technology, health care, etc. This outcome is the top of the systems advocacy stream. Finally, the top outcomes from these different streams combine to achieve the ultimate outcome. That is, if persons with disabilities participate in communities to the extent they wish, and if communities are more accessible, then persons with disabilities will be integrated into American Society. This stands alone as the top-most desired outcome of the CIL program. As an overview of the logic model, three types of CIL activities are represented by three separate boxes across the bottom of the page. These CIL activities are, from left to right: (1) IL services, (2) Information and Referral (I&R), and (3) Systems Advocacy. As a result of conducting these three activities, the CIL program achieves a total of 15 initial, intermediate, and ultimate outcomes. These different outcomes are represented by 15 different boxes that are arranged (“flow”) upward, on five different levels, from the three activity boxes. The combined effect of achieving these 15 outcomes eventually leads to the 16th and top-most outcome, which reads “People with disabilities are integrated into American Society.” In other words, 15 outcome boxes flow upward from the CIL’s three activities, and these 15 outcomes converge to achieve one ultimate outcome at the very top of the page. Delving into the details of the different outcomes as they flow upwards, we can use an “If – Then” logic to work systematically up the page from each activity to the ultimate outcomes. Although most outcomes flow from one set of activities directly upward, there is also some crossover and intermingling of these three streams. Beginning on the left side of the page, CILs conduct IL Services. Working upwards, because of these IL services, persons with disabilities have the skills, knowledge, and resources to support their choices. If they have these skills, knowledge, and resources, then they will make their own choices. If they make their own choices, then they will simultaneously regard themselves as more independent and also will, in fact, be more independent. If they both regard themselves as more independent and, in fact, are more independent, then persons with disabilities will participate in communities to the extent they wish. This outcome represents the top of the IL services stream. In the middle of the page is the Information and Referral stream. Because of I&R, persons with disabilities both get the information they need and simultaneously see different possibilities. These two outcomes are achieved mostly due to I&R, but IL services also contribute. If persons with disabilities see different possibilities and also get the information they need, then they will advocate for increased community supports. This outcome represents the top of the direct I&R stream, although this stream also merges with the final stream to be discussed. On the right side of the page is the Systems Advocacy stream. The first step in systems advocacy is to identify barriers and problems. If these barriers and problems are identified, then two different outcomes will be achieved -- a consumer agenda for change will exist, and active coalitions will exist around our issues. These two outcomes then diverge into two different directions. First, if a consumer agenda for change exists, this outcome merges with the I&R stream and helps persons with disabilities to advocate for increased community supports. At the same time, the existence of a consumer agenda, coupled with the existence of active coalitions, plus the advocacy efforts of persons with disabilities (from the previous stream) leads decision makers to act on our agenda. If decision makers act on our agenda, then two outcomes will result: communities will have more resources that support independence, and methods and practices will promote independence. If these two outcomes are achieved, then communities will be more accessible in terms of housing, transportation, information, employment, education, assistive technology, health care, etc. This outcome is the top of the systems advocacy stream. Finally, the top outcomes from these different streams combine to achieve the ultimate outcome. That is, if persons with disabilities participate in communities to the extent they wish, and if communities are more accessible, then persons with disabilities will be integrated into American Society. This stands alone as the top-most desired outcome of the CIL program. >> SLIDE 11 Desired Outcomes 1. PWD have skills/knowledge/resources to support their choices. 2. PWD are more independent. 3. PWD get the information they need. 4. PWD advocate for increased community supports. 5. Barriers and problems are identified. 6. A consumer agenda for change exists. 7. Decision Makers act on our agenda. 8. Methods and practices promote independence. >> SLIDE 12 Information and Referral Stream Outcome: Persons with disabilities get the information they need - Indicator: “Persons contacting the CIL report they have the information they requested.” - Indicator: “Persons contacting the CIL report they used a new resource they learned about from the CIL.” >> SLIDE 13 IL Services Stream Outcome: Persons with disabilities have skills/knowledge/resources to support their choices - Indicator: “People served by the CIL can list at least one (1) specific skill, type of knowledge, or resource they have now that they didn’t have before approaching the CIL.” >> SLIDE 14 Systems Advocacy Stream Outcome: Barriers, problems identified - Indicator: “Activities, such as surveys, public meetings, focus groups, polls, are conducted to identify or confirm the primary barriers/problems in the community that prevent persons with disabilities from leading more independent lives.” >> SLIDE 15 Systems Advocacy Stream, cont’d. Outcome: A consumer agenda for change exists - Indicator: “Presence within the CIL’s annual plan of a separate section containing an explicit systems advocacy work plan.” >> SLIDE 16 Systems Advocacy Stream, cont’d. 2 Outcome: Decision-makers act on our agenda - Indicator: “The number of positive changes achieved or negative changes prevented in legislation, policies, practices, or services at the local, state, or federal level that address the barriers/problems identified by the center’s consumers.” >> SLIDE 17 Systems Advocacy Stream, cont’d. 3 Outcome: Methods and practices promote independence - Indicator: “# and % of consumers served by the CIL within the past calendar year who moved out of an institution and into a self-directed, community-based setting.” - Indicator: “# and % of consumers served by the CIL within the past calendar year who remained in a self-directed, community-based setting on December 31 despite having been at risk of moving into an institution.” >> SLIDE 18 Resources http://www.ilru.org/resources-outcome-measures - Interview Guides (Consumers, I&R) Outcomes Management Worksheet http://www.ilru.org/training/outcome-measures-for-centers-for-independent-living - 2 ½ day training with Power Point, on-demand videos, handouts. >> SLIDE 19 Questions & Answers >> SLIDE 20 The Process CILs that beta tested the process, did so using three questionnaires. 1. I & R callers were told “our CIL wants to make sure that people are getting the information they need. May we call you back in a couple of weeks to check?” 2. CIL consumers were randomly selected until a statistically significant number of consumers were identified. 3. Overall CIL questionnaire. >> SLIDE 21 I & R Questionnaire WHEN YOU HAVE THE I&R CALLER ON THE PHONE, READ:   “Hi, my name is (your name) with (name of your CIL). On (date of the initial I&R call), you called our agency for information and referral assistance. In order to improve our services, we are calling back to some people like you who contacted us during this past year. I’d like to ask you two short questions about the service you got from our Center when you contacted us. All answers will be kept confidential. Do you have a moment to answer two short questions?” If answer is no, try to re-schedule FOR aNOTHER time. IF ANSWER IS STILL NO, DOCUMENT YOUR ATTEMPT AND THANK THE CONSUMER FOR HER/HIS TIME. >> SLIDE 22 I&R Questionnaire, cont’d. If answer is yes, continue: “Thank you. For each question, please be as honest as possible. There are no right or wrong answers, just whatever you feel is true. We want to know how you feel, so we can do the best possible job. Let’s begin: “Sometimes we’re able to help people get the information they need from us, and sometimes we’re not. For you personally, did you receive the information you needed from us?” I Don’t Remember: No: Yes: >> SLIDE 23 I & R Questionnaire, cont’d. 2 “Sometimes people use a new resource they learned about from us, and sometimes they don’t. By the word “resource,” I mean someplace you can call or visit to get more help or more information. For you personally, did you use a new resource you learned about from us?” I Don’t Remember: No: Yes: THANK YOU SCRIPT: “Those are my two questions. Thank you for your help. Your input will be useful to us and other Centers like us as we improve services for persons who contact us. And as I mentioned before, your answers will be kept completely confidential within our Center.” >> SLIDE 24 Examples from Report from the Outcomes Task Force Desired Outcome: PWD get the information they need - 72% of I&R callers received the information they needed; 15% did not and 13% could not remember. - 52% used the information they got, 32 % did not and 17% could not remember. Desired Outcome: PWD have skills/knowledge/resources to support their choices. - 70 % of CIL consumers reported that they learned new skills/knowledge or resources and were able to list one or more examples. >> SLIDE 25 Two Ways to Use Outcome Information 1. Outside our CIL – for PR value - Easily understood by many different audiences - A tool to enhance the fund-raising capacity of the CIL - Recruit talented staff and volunteers - Promote our CIL to potential clients and referral sources - Encourage other agencies to collaborate >> SLIDE 26 Two Ways to Use Outcome Information, cont’d. 2. Inside our CIL – for program improvement - Know how effective we’re being - Find ways to be even more effective - Help staff to focus on what’s important - Identify training needs - Support both short- and long-range planning >> SLIDE 27 Possible Next Steps - Review the training materials on ILRU website - Put “outcomes” on the agenda for the next staff and/or board meeting - Include a section on outcomes in your next annual report or on your website - Review your satisfaction survey, including whether it captures outcomes - Select several indicators to measure >> SLIDE 28 Questions & Answers >> SLIDE 29 Contact Information - Dan Kessler - dan.kessler@drradvocates.org - Patricia Puckett - PPuckett@silcga.org - Paul Spooner – pspooner@mwcil.org >> SLIDE 30 Wrap Up and Evaluation Please complete your evaluation of this program at: https://vovici.com/wsb.dll/s/12291g5712f >> SLIDE 31 CIL-NET Attribution - Support for development of this training was provided by the U.S. Department of Education, Rehabilitation Services Administration under grant number H132B120001. No official endorsement of the Department of Education should be inferred. Permission is granted for duplication of any portion of this PowerPoint presentation, providing that the following credit is given to the project: Developed as part of the CIL-NET, a project of the IL NET, an ILRU/NCIL/APRIL National Training and Technical Assistance Program. >> END PRESENTATION