ROUGH DRAFT 9-13-11, Outcome Measures for Centers for Independent Living – An IL NET Resource Presented by ILRU My clock was 1:45. Welcome back everybody here in person and everybody on line. We're getting ourselves settled, as you can see. Hope you all had as nice a lunch as I did. Let me mention to our technical people in the back, this is the session on choosing outcomes to measure. I presume we're all set up, ready to start on that. Online friends are ready to go too for that. Help me out. Remember this one we just did with the index cards? How many outcomes were there on it? >> Sixteen. >> Sixteen. >> MIKE HENDRICKS: This is the one with the index cards. >> Six. >> Bunches. >> Six. >> MIKE HENDRICKS: SIKS. You wreck the NCIL or CIL logic model? How many use comes were on that one? >> Sixteen. >> MIKE HENDRICKS: Right, that was the one with sixteen. Okay, up until now it's been nice thinking, oh, gee, what are my desired outcomes. That is nice, I can put down anything I want. How about my logic model? Let me think about how these all fit together and make sure they are logically and I'm doing something that has a chance of working. That is good too. Now we have to make some tough decisions. Before now we were just thinking, idealizing, planning, being smart. Now reality starts to intrude because we probably don't have the time, inclination, money to measure six much less 16 outcomes. You have to choose. You have to choose in real life and we have to choose today. That is the way it goes. Choosing outcomes to measure. Step three on our yellow brick road. Why is it necessary to choose? I think it's obvious. Don't have enough time or money to measure all the outcomes. Might be nice but probably aren't going to. Important to focus attention on what matters the most. Not everything can matter the most, right? Also it's necessary to identify where your CIL wants to be most effective. If there's something that really matter to you, we need to know that. Also make sure you don't overreach. You might try to measure them all but you will just burn yourself out. How to choose. I were I could tell you choose number 1, 7, 14. That is not going to work, is it. You can try it. Might work. What I'm going to suggest is a tip. You have to juggle three things at once. This is just the reality of what you have do. You have to juggle three things at once. First one, what outcomes capture the most meaningful benefits of your program. That is really the essence of what you're about. If someone were to say to you, what are you trying to achieve, what is the first thing you would say? You want to keep that ball in the air. That is obviously really important, right? You also need to juggle another ball, and that one is what outcomes would be most helpful to improve the effectiveness of your program. To improve the effectiveness. Look at your outcomes and say which ones would really help me improve if I knew how I was doing on that. That is another ball to keep up in the air. There's a third ball to keep up in the air too. As you can see, it's how can you best communicate the value of your program. What outcomes best communicate, what ones speak most to the outside audiences that are going to be caring. That is important too. Let's face it, it's a real world, and yes, I want you to focus on improving the effectiveness of your program, but also we all know that we do have to keep trying to tell a good story about what we're doing because we're going to be doing better and better all the time. We have to have that ball in the air, juggle the three things. Capture the most meaningful benefits of the program, what would be most helpful to improve effectiveness, and how would you best communicate. Problem is THOET those don't always come together. Doesn't always converge on the same outcomes. This is where the judgment call comes into it. Let's let Bob Michaels talk to us about how we made that decision when faced with 16 outcomes and knowing there's no way we could measure all 16 of them. . >> BOB MICHAELS: Thanks, Mike. Here we are. We have got this down now, identified desired outcomes, put them into logic model. Right away we realize, hey, we have way more than we can deal with here. We're going to have to pare this down. How do we go about doing that? Of course, being who we are, we said we wanted to go back in the field and find out what the field wants us to do. So we went through a number of different exercises we need to go through, realizing that whatever we came up to measure, we have to go back to the task force and say okay, this is what people are telling us out in the community that they want to do. So what we did is we went out there and said we're going to have to choose which outcome we want to measure. We wanted to make the process as inclusive as possible. We asked people to do was, we would like to you take a look at these 16 and pare it down to six. What are the six desired outcomes that you want to work on the most. What we did was put a link in the e-mail that we sent out and said when you go on this site, the outcome measures, there's a link there that says last chance, please give us feedback on the desired outcomes. You click on that and you have the survey. We'll will ask you to identify the six that you think are most important. What you did, identified eight. There were about eight clearly out ahead of everybody else. So we had eight instead of six. What we did, we took them back to the task force and said okay, we think that we should go with these eight rather than the six, because these all seem equally important to everybody else. So let's look at eight, and they agreed to do that. Here are the eight that we have. We picked the IL services, the first one, persons with disabilities have skills, knowledge, resources to support their choices. And the fourth one, people with disabilities are more independent. On the I&R stream, people with disabilities get the information they need and they picked people with disabilities advocate for increased community supports. On the systems advocacy, they picked four. Barriers and problems identified in the community. Consumer agenda for change exists. Decision-makers act on our agenda. And methods and practices promote independence. Now, the packet I gave you this morning, I referred to this morning, the one in color that also has the logic models, on page 8 of that packet you have a picture of all of the eight that they chose. The last three, all three service streams, the very top, none of those were picked. They did not pick people participate in communities to the extent they wish, communities are accessible, or that people with disabilities are integrated into American society. Didn't pick any of those. If you look at that chart, what you see is eight are having a line around them, indicating which ones we emphasized. You see there was at least one or two in each, at least two in each stream. And there was four in the final stream, having to do with systems advocacy. Okay? Any questions. This is what you said was most important. These are the ones that we, after you told us they were most important, then we began to work on them to identify ways to measure them. Okay? Everybody understand this? Any questions at all? Nothing on line? Okay. Now it's your turn. Mike? >> MIKE HENDRICKS: Bob, let me play devil's advocate if you don't mind. Bob and I work won't fully well together so he won't mind. Just to stir up conversation in the room. Just to stir up conversation in the room. Let me pretend that I'm a harsh critic of the effort. I'm looking at the one that you just showed us. Can you back up? >> Sure. >> MIKE HENDRICKS: Just to stir up controversy and see what you think. It's like what do you mean you didn't pick any of the top ones? That is really your ultimate outcomes. You labeled them yourself, your ultimate outcomes, now you're not even measuring them? You're just wasting your time. Or you might say, just seeing what you all think, or you might say wow, four of your eight have do with systems advocacy, hmm, you must really think that is the most important thing y'all do, more important than IL services and I&R. . >> BOB MICHAELS: Wasn't that interesting? That we did in fact pick most of them in systems advocacy. Really says something about who centers think they are, what centers think they do, what is important. >> MIKE HENDRICKS: Now that I have been that impolitic, anybody want to jump in more politic with comments about anything? >> Sit down, please. >> (Laughter.). >> MIKE HENDRICKS: This is one of the those path not taken things. You have left eight behind. That has more implications, which means we don't know anything about those other eight. At least in the yet . >> BOB MICHAELS: This is where centers work, I think. >> If there's an upside, at least by the way they are chosen --. >> Use the mike. >> I'm not loud enough. If there's an upside, at least you're going to be able to measure your success early on as you're progressing. I mean, four out of the eight are initial outcomes. The other four are at the next level. My question, though, wouldn't you maybe have just wanted to give them a list of intermediate outcomes and ultimate as their choices as opposed to the first round? If you really want people to measure the outcome, your ultimate outcomes, you think you might limit that list of 16 down to like top five, six, eight, whatever . >> BOB MICHAELS: Yes, I don't think it ever occurred to us do that because basically it was your model and they were your outcomes. You would be the ones, you would pick what was most important to you. We weren't going to try to tell you we think should be up here doing something harder. In fact, we knew most centers probably were in a place where they weren't doing the real sophisticated systems advocacy work. Does that answer your question? >> Sort of . >> BOB MICHAELS: Anything else? >> MIKE HENDRICKS: Let's make them work then. >> Hey Bob, I think one of the things that, to me, looking at that and making those choices, when you look at the intermediate outcomes that people with disabilities are more independent and on the other side, methods of practice promote independence, to me that sums up the top ultimate one of being integrated. I think maybe it's just a terminology issue that to me in my head, and this is just mow, it's repetitive. Once I have those key factors, I have that idea . >> BOB MICHAELS: If you do that, then you're under that. >> Right. That may be why people didn't pick those because they couldn't put a tangible to that as they could say methods of practices as opposed to just people are integrated. >> BOB MICHAELS: Right. >> MIKE HENDRICKS: Care. >> BOB MICHAELS: You devil you. Zak yeah. You all have your managements worksheet, right? How many desired outcomes are on it? >> Four. >> MIKE HENDRICKS: You don't have enough money do four or enough time to do four. You will burn out if you do four. We're only going to let you do two. >> Two? . >> MIKE HENDRICKS: Two. Not even three. Only two. Guess what your assignment is? You have to figure out which two it's going to be. Remember those three balls? Remember the ball of what is most important? The ball of what is most useful to improve your effectiveness? The ball of what you need to use for your audiences. You have to juggle three balls and look at four indicators and, or outcomes, sorry, and leave out two of them. Here's what I want you to do because I want you to feel the pain. I want you to the two you're not going to do, draw a line through them. Just draw a line through them and say we're leaving you behind right now. We're not going do anything with you right now. Leave two, draw lines through two. Okay, got to be consensus. Table has to agree. Go to it. You folks back on line, you have to do the same thing. Look at these four, leave out two of them. (Exercise) Any group still working? Nevermind, several of you still working. Keep going, not a problem. . What do you think, have we all made the painful decision of keeping two and losing two. Looks like we have. Okay, here is my question. Question isn't which one did you which ones did you pick and which did you leave out. The question is how did you decide. Somebody tell me. Sir, how did you side? Fred. >> One of our choices to eliminate was based on the idea that we could fold the concept into one of the surviving elements. . >> MIKE HENDRICKS: Oh. Okay, that probably worked well in your situation. Let me caution you against doing that too much in reality because what you will find yourself doing is taking your logic model, which made perfectly good sense, then starting to move it around because you want to start squishing things together and you lose the whole purpose of doing the model in the first place. >> That became part of our conversation around the table, how not to lose focus and keeping on eye on the concern colonel without going too far on the issue. . >> MIKE HENDRICKS: Very good of the logic model, remember, is supposed to capture your program, what you are about. By gosh, we don't ever want to lose that. We don't want to lose that. Yeah, that is good. Uh-huh. >> I think piggy backing on what was already said, the way we selected was based on what we collectively felt was the needs of the community, right? Like our personal passion in our centers is to meet the needs of the community. So with the particular topic of senior citizens, on a grander scale what are senior citizens requiring of, you know, society and how can we meet those needs directly. That is how we selected based on this. >> Let me see if I understand. I real like that. Let me see if I understand that. You basically said let me step outside myself and not just try to me figure out what makes most sense, but let me look from a bigger community wide perspective and what might the whole community say would be the key things. Did I miss is that? >> That is right. >> Sort of right? I like that personally, uh-huh. Otherwise people decided? Some of these tables down here that we have not been looking at? My mistake, I apologize. Any decisions, deciding ways that you thought of. >> Okay, we'll try the this out. I think for us we tried to look at it logically. >> MIKE HENDRICKS: I always like that. >> Yeah. Seemed like at first because these different indicators or outcomes were attached to the different services at the bottom, once we looked at what kind of, what was our focus, was it IL service, information referral, add voice case, then we could look at the logic model here and could find some that apply pretty well because of that. So just sort of flowed from, because as you look in the flowchart here, different kinds of measurements are attached to the different kind of services. But one of our team members pointed out there was some interchangeability. So we weren't trapped. Just gave us kind of a starting point from which we could get to the right ones. >> MIKE HENDRICKS: You look at the CIL logic model and that affected your thinking of which two to keep and which not? >> I think all the work that has gone into this has been very helpful and gave us, you know, I think we wanted to at least give it a solid chance and look and see how it related. It relates pretty well to, at least from the quick, if you have a time crunch, might as well not reenvent the wheel. We had a pretty good wheel the work from. >> MIKE HENDRICKS: Interesting. Okay. How about this way where I haven't been looking enough. How did you make the decision. How about my friends at this table. We'll volunteer my friends at this table, how did you decide which ones to keep and which to lose? >> We started with an observation, what we want to get rid of first. >> MIKE HENDRICKS: That is interesting. >> What we could afford, we're going to be advocating transportation in our case senior citizens over 65, transportation we're going advocate for, but something we can't afford, we divide the difference. We scratch is that. Other thing was healthcare. Issue for everybody. Realizing it's a goal of ours, a measurable outcome for us somewhere in the future. But at the same time it's something we can't afford. So we chose moving people, transiting people out of institutional care, which is something we would like to see happen, and to provide attendant care for these folks. >> You added a fourth ball in the air to juggle, the cost of hopefully getting some good information about these things. >> Right. >> MIKE HENDRICKS: You added a fourth. >> Yes, the cost. >> MIKE HENDRICKS: Sometime soon have you to add that ball, and you just added it now. I might have delayed a Tad, but no big deal. You added that in. Okay. You see the point though? You're going to have to make these decisions back in your CIL. Let's say you come out with ten major outcomes. You are going to have to decide just like we did just now, something you have to do. Any questions about that? Pretty simple process, conceptually. It's not a simple set of decisions once you start, but the process is pretty straightforward, don't you think? Any questions at all? If not we'll start sliding into the next session which gets into this measurement and indicators and things like that. Okay, we'll let that go. We have now figured out that we have this tough job of managing, choosing some outcomes to measure and some not. They need about 60 seconds to change something over for the online folks. I'll take that chance to change something here. Our friends will let us know when they are ready. By the way, you will notice there's a break built into at three clock. We will take that break at three. Don't have any fear. By the way, can I get a round of place applause for Deborah up here typing her fingers off. I can't help by notice she is all by herself with no one to spell her. She is really working hard on our behalf here. >> She is spelling just fine. >> MIKE HENDRICKS: She is doing great. >> You said nobody to spell her. I said she is spelling well. >> MIKE HENDRICKS: There's a pun. Nobody needs to spell her because she knows how to spell.