ROUGH DRAFT 9-13-11, Outcome Measures for Centers for Independent Living – An IL NET Resource Presented by ILRU I'm going to turn it over to Bob Michaels because he was the chair of that group and he is going to talk to you about how that worked out >> BOB MICHAELS: We need to wait 1 minute. I think I have to wait for a high sign. >> Is this logic models? . >> BOB MICHAELS: This is session one. >> MIKE HENDRICKS: Bob, how about those diamond backs while we're waiting? >> BOB MICHAELS: Aren't they something? Probably nobody else was watching. They were supposed to come in last place. They have been in first place by about eight and a half games. They are just having a wonderful year. Baseball fans in here. >> Bob, a pun came up at NICL conference. I have been waiting to tell you this. What do you call two NCIL members going to the Hill? A March of dimes. >> (Chuckles). >> It's funny, there's so many friends I have here I have known for 20, 30 years some of you. Amazing. I delivered, and I'm across the street. So I'm doing travel back and forth. It's killing me. But it's going to be delivered today by noon. Have my transportation Rigg. Okay. Thanks for your time. My name is Bob Michaels. I'm going to be talking to you about the NCIL outcome measures project. What we're going to have happen here, we went through a project NCIL did, but there's this whole way that you go about developing outcome measures and doing the outcome measurement that Mike is going to talk to you about. We took a look at 16 different outcome desired outcomes and we picked, using your help, we pick about eight, and using your outcomes we developed indicators for those eight. Some of you are going to look at those indicators that we came up with and say, this is exactly what we need at our center. And if you choose to do that, then what is going to happen is that we will have done a lot of the work for you. The field testing, the questions airs, we will give you a lot of information so if everything worked fine, perfect for you, you're going to have measures and indicators you will be able to use at your centers. However, what you are is going to find is that sooner or later you're going to be beyond that. Maybe you already have moved beyond that. You will be at place where you will be wanting to decide maybe instead of looking at the behavior, what happens when they come in, we're more interested in knowing in fact how many people are employed, people with disabilities in our area of the state. We have already done all of the work to try to set that up. Now we're going to try to look at our outcomes. We're hoping here that Mike is going to tell you about what you need to do if you need to develop other indicators, and I'm going to talk to you about happen in our project. So every time you get to this stage here it's going to be talking specifically about what it is that we did. Let me give you background here. Many of you who were around at that time, five, six, 7 years ago, understood that what happened was the office of management and budget started coming back down to RSA and to others and saying, we want you to start applying program assessment rating tool. What they said was that we took a preliminary look at the independent living program, and frankly, you guys did pretty bad. So we want you to start developing these outcome measures. At the same time there was a lot of pressure. Independent living centers were saying all of a sudden you had waste, for instance, starting to ask us, and other providers started to ask us. For about 20 years the centers have been coming to us at one point or another and saying, we have to tell our story in a more effective way. We have to get better about telling what it is that we do. In a 704 just does not do that. It doesn't really, it says what we're doing, but not whether or not what we are doing is successful. That is kind of what we started doing here. So we took a look at that. We said we're going to see, you have to understand what future is going to be and begin to take action to resolve that. So what happened was that now we're talking about, oh, 2006, and we had gotten this word from OMB. And the people at the NR say we are trying to come up with outcome measures. We said this is how it's working for us. I approached Kelly bok land who at that time was the chair or was, yeah, the chair of national council on independent living. Kelly is now the executive director. But Kelly, who is sitting back in the corner here, I went down and said boy, I think what we need to do, we need to start looking at these outcome measures and developing them ourselves rather than waiting for the rehab services administration to do it for us. We could be much more effective. So I talked Kelly into creating a task force, volunteered to chair it, that was specific to look at this problem. As it happens over the time, there actually have been two task forces. The first one was really CIL leaders. People around the country from centers for independent living that really have taken a lead. What we did, we went to each district and we said, tell us who it is in your district that is most actively interested in outcome measures. And they pointed out the people that they thought represented centers for independent living. That really became our task force. We hired Mike at that time to work with us, and we had a lot of involvement of people from department of education and from the office of management and budget who then participated and tried to give us assistance in getting started. And we had two meetings at that time to take a look at what we did. One in April of 2006 and another one in June, to begin developing some of the things that we think then came to you and asked for. Second one, second task force had experts in independent living like the last one had, but we just had a couple at that time. Then we tried to get was people who had background in research and in information gathering. So we would have people that were working with us that then really had a knowledge and understanding of what you need to be doing at this phase of our work. One of the things that Kelly really pushed me on and that we agreed to was really important from the beginning, we looked at these and said, these are yours. These are not ours, this is not NCILs, these outcome measures are yours. Every phase of the project we stopped and said okay, here is what we have done so far, and here is what we have come up with. What do you think? We developed a whole series of different ways of gathering information from you that would make what we do as effective, more effective and be more representative of what it was that you thought was important. So even the outcome measures that we had finally emphasized, the eight desired outcomes, they were ones that you chose. Maybe not some of you privately gave input on. Some of you did it very public way, made presentations around the country. But what we did, and you're going to see from us and from me, an is reflection of what you thought was important. I wanted to tell you that this has been very expensive project. We couldn't have done it without great support from, first of all, NCIL. They supported us all the way through this, made sure that we were able to give presentation at their conferences, that we had, they covered all the expenses for the projecting of things, andic making sure that everything worked -- and making sure everything worked. IL net that is actually covering a lot of this training, was also responsible in a big way and covered the costs that were needed, that were necessary when we were trying to put together what it is that we were doing. Then individual centers did a lot, contributed different money. When we would make appeal to them. Also University of Kansas through Glen White. Not only did he participate on the task force but he assisted us in making this a priority of theirs so that they could fund some money from other projects into what it was that we were doing. So really has been an effort that everybody has been involved in. One of the things that happened to us, and this is what you're going to find as you begin to develop your own, is that there's different ways. We know things work logicically. You know, this is how they should flow when you are developing outcome measures. But we found often is that we had to make adaptations and do things and bastardize a little bit. What we finally came up with in the process we used sometimes was a little bit different than what we would have originally recommended somebody. I think the first stage is a good example of that. We had, as I said, two meetings of the task force, and we had to get input that they, from them, and then provide that to the community. That made it very difficult for us. What we did the first time when we got together, we realized the first step in the process was identifying the desired outcomes. This is what we think we do. You know, not flowchart. Not what is it that work and didn't work. It was a logic model. This is what we want to accomplish and when things work in independent living, here is how they work. So we started developing that. Then you put that into a logic model. Because we were pressed for time, we started to blend those a bit and what we did was provide Mike with lots of information about independent living. Then he came up with the first time we meet, came up with the logic model and desired outcomes that he thought were reflection of what we wanted to do. Well, Mike likes to tell you he doesn't know anything about independent living, and that is the biggest joke. He knows. He's incredibly trustworthy here. But at this time, this was our first time together, and we hadn't even met, he didn't do a very good job. Right from the beginning we started, we tore apart his logic model and his desired outcomes. But it gave us a good starting point, really helped us along, and made it possible for us then after, in between the sessions of us meeting, in between teleconferences and that, to take something out to the community and say, this is what we have come up with so far, and this what we think works, and to gather your input on that. So what we're going to do here, I'm going to identify for you the desired outcomes that we came up with for independent living that came up out of our task force. We actually started these in three different areas. We decided we had IL services and they created certain outputs. We had IR, information referral, and they created some outcomes. Then we had the systems advocacy work that we did. Now, those of you who know independent living over many many years know that one of the problems we have always had is trying to identify how we do systems advocacy and how we measure what it is that we do. There is some really good way. If you take a look at centers and indicators that are developed nor systems advocacy, you will see that it's really not very sophisticated at all and really pretty easy to do what it is that you have do. Well, not if you do outcome measures. So here is is the IL services. The person with disabilities has skills, knowledge, resources to support their choices. If they have those skills, knowledge and resources, then they will make their own choices. If they make their own choices a couple things will happen. They will begin to regard themselves as more independent and in fact they will be more independent. So that is one thing from IL services. And out of information referral, if a person with disabilities gets the information they need, then they can see different possibilities. If they see different possibilities, then they are going to begin advocating for increased community supports. Okay? Again, this is reflection of what you felt happens in this process here. So we came back to you and asked, do you agree that these are the steps. On systems advocacy, you first tried to identify barriers or problems in the community. Then if you do that, you're going to start to come up with a consumer agenda. Then you have a consumer agenda, then give coalitions are going to exist around those issues. If you have active coalitions and decision makers will act on our agenda. And if that happens then two things will happen. The communities will have more resources to support independence, and there will be methods and practices that promote independence. Now, if those three things happen, what you're going see later is what we call initial outcomes and recommended outcomes. If those things happen, then you can have ultimate outcomes. You have three ultimate outcomes. People with disabilities participate in communities to the extent they wish. That communities are more accessible. And if you do those two things, then people with disabilities will be integrated into American society. Okay? Any questions?. >> Bob, can you tell me what you mean by consumer agenda. >> BOB MICHAELS: What we're talking about in here, what happens, we talked a lot about that. How it should work. You as independent living centers know that one of the things you're supposed to do is dedevelop advocacy that reflects what it is that your community wants. You're not supposed to think, well, we just know. You're supposed to take steps to find that out what it is. If you take steps to find out what it is that your community wants to do, then you will develop an agenda that usually we do that in the form of a plan, an advocacy plan. You have to say these are the three or four things that were identified by our community that we need to work on. Okay? Does that answer your question?. >> Yes. >> BOB MICHAELS: Anything else? Then I'll turn this back to Mike>> MIKE HENDRICKS: I was talking in general about outcomes and desired outcomes. Bob has shown you the 16 that we started with for the NCIL task force. Now it's your turn. In your packet, please find this page and pull it out. It's called at the top outcomes management worksheet. Every single one of you has one. Those of you here watching on line, this is part of the materials that are available to you. We need to ask you to print out a copy of this one sheet. Just this one sheet only. Print out a copy for each person who is sitting at your site. You will be using this for the next 2 days. This is a tool you will be needing. Find this. It's folded up actually. It's foldel over in your packet so it looks like a blank sheet of people, but it can be opened up. >> What is the title? >> MIKE HENDRICKS: It's the only sheet that is bigger than everything else. That helps. It's called outcomes management worksheet. I see a bunch of them appearing, so they are in there. That is good. If you will notice on this, it's got several different columns. The one that is relevant for us right now is this very first column on the left-hand side where it says desired outcomes of our hypothetical program. Now we thought about this a lot when we were putting together this course. We were thinking should we ask people to work with their own CIL program? That gets tricky because sitting with people from other CILs and centers and you get caught up in the details of your own programs. We thought let's do the conceptual learning caught outside our CI world and put our knowledge aside and work with concepts. I'm going to walk out of the screen. Those of you at home, I will k back. I'm going to walk over here and ask this table to work on a program for after school program for kids. And this table to work on a program for senior citizens. After school for kids, senior citizens. After school for kids, senior citizens. After school for kids, senior citizens. After school for kids, senior citizens. After school for kids, senior citizens. So what we're going to ask you to do, and those of you watching on line, pick whether you would rather work with an after school, all of you around the table should use the same program, after school for kids or senior citizens. Now, what we want you to do, as soon as I can retrieve from my good friend Bob the clicker. Thank you. We want you to write four desired outcomes for your hypothetical program and put them on your outcomes management worksheet, one desired outcome in each of the 4 boxes. Don't start yet please. We have two tips and they are pretty good important tips. First one is as you are trying to think of what these desired outcomes of this program might be, get outside the head of just the staff. This is a real common mistake. A lot of times people will gather three, four staff around a table and will say okay, what are our desired outcomes. Don't do that for this exercise. Don't do that in real life. When you go back home and you are thinking about your CIL's desired outcomes, don't get three staffers around the table and say come up with our desired outcomes. Look at it from multiple parens. Carol, you have a question?. >> Somebody asked the name. >> Someone on line wants to know the name. It's called outcomes management worksheet. Thank you. >> Mike, can you hold that just a second and let me zoom in so they can see it. >> This is great. This is like vana White. Hold this up. How we doing?. >> Can you see it on the screen? I think we got it. Thank you. >> This is what it looks like. Thank you. So at your tables now and back home in real life, remember that multiple, different perspectives will think about the desired outcomes differently. Program documents is one way to get some sources of ideas for outcomes. Program staff. Yes, absolutely. Volunteers look at things a little differently. They may have a different perception on the desired outcomes. The participants, the parents. Records of complaints. Complaints are a nice way of asking yourself what is the desired outcome. Obviously we didn't achieve it in this situation. If you work with a consumer who is going to eventually later on work with another agency, go to that agency and say to them, what outcomes do you think we ought to be achieving. Might have some interesting insights for you. Programs with mission, services, participants similar to yours, our simply outside observers of your program in action. There are multiple sources of ideas for outcomes. Here is the second tip, a really good one, and I strongly commends you this one. It's how to word an outcome statement. A whole bunch of different ways to word outcome statements. If you do it this way, I have a lot of years in this, this will help you. This will help you. Okay. Word them this way. Start with the target group, who it is you're trying to change, that is, who you're trying to benefit. Then follow with a present tense verb. Not a future tense. Not, you know, will act differently. Follow it present. They act differently. Whatever. Then whatever it is you want to happen. Let's look at some examples. Parents of preschool children, that is the target group. Not all parents. Parents of preschool children. Use, that is your present tense verb. Every day moments to encourage early learning. That is your desired outcome. Here is another one. Adults completing the literally program. Not all adults. Not the adults starting the literally program. The adults completing the literacy program, that is your target group. Read. There is your present tense verb. At the sixth grade level. Home bound seniors eat nutritionally balanced meals. Battered women who wish not to return home. Not all. The women who wish not to return home. Meet self defined objectives for rebuilding theirs lives. High school boys reported for fighting, the target group. Demonstrate, the verb. Skims at resolving conflicts verbally. Seems like a small thing but it will help you if you can stick in your head target group, present tense verb, what we want to happen. That is the tips. Remember the program. Talk amongst yourself. Maybe one of you say, I'll pretend to be a volunteer. Another one says I'll pretend to be a participant, I'll pretend to be another program. Maybe you wear a different hat. Take those different perspectives. Write it this way. We want you to put four very appropriate, legitimate, reasonable, desired outcomes, one in each of these 4 boxes. Any questions about the assignment? I want to be sure I'm semi clear at least. Any questions? We have a break coming up in 9 minutes. We will hold you to that. We will hold ourselves to getting you that. How is that? We'll take a break at that point. When you come back we'll see if you need a little more time to work on it or if you are ready to report back or whatever. Let's work on it for the 9 minutes. See how much we can get gone before the break. Thanks. (Groups working at tables). We're just about at break. If you want to work through, that is your business. We will start back at 10:15 because we have the good people on line expecting us to do that. We will start back at 10:45. It's officially break. >> 30 seconds. >> Sorry, I need your attention for one second, please. Richard Petty needs to make an announcement. I know it's important because he says it is and I love him. >> Very quickly, primarily for our video audience. We had a little bit of a snafu earlier and switched to a new segment incorrectly. So you will continue after break using the same segment titles. We'll be synced again properly after lunch. Just stay in the same title that you're using now, and that will get us back into appropriate order. We apologize for that. This is beta technology for us and that is why we have been offering it free to those of you online while we get our processes worked out. We appreciate your patience. We have had difficulty in implementing the onstreaming video PowerPoints for alternate format users. That process has not worked well up to this point. We are still working on it. We will try to implement that during the day. If not today, tomorrow we'll be trying our best on that. We're looking forward to sharing that with you. That is it, online folks. Stay on this same presentation that you're on now, and we will continue that and after lunch be back there sequence with the proper titles. Thanks very much. >> MIKE HENDRICKS: Thanks and we're on break until 10:45. (Break for 15 minutes). Folks, it's time to start back. Barely, I want to tell you Joan says hello. She is in fact here. Message I was asked to convey. Okay, you filled out presumably four desired outcomes. Simple, right? Piece of cake? No problems. Sometimes it gets harder when you actually have to start doing it. That is the purpose of the exercise. Let's hear from some of the tables. Let's just hear, who wants to tell at least one of the desired outcomes. Yes, sir, right here. Do use the microphones, thank you so much. Start off, tell us which kind of program you're working with. >> We have the kids for after school for kids program. >> Okay, after school program for kids, great. >> Yes. We have studies improve their ability to make ethical choices in social settings. >> Students, that is the target group. Improve. That is not will improve, but improve I which I love, present tense verb. Their ability to make?. >> To make ethical choices in social situations. >> I love it. What do you think? Sound like a desired outcome to you?. >> Yes. >> Think so too, a nice desired outcome. Good for you, for your table. Somebody else. Use the Mike. If anyone of the online folks want to send one to Carol we can read those if you want to. Speak lauder we'll see. >> Parents of children enrolled if the program regard their children's program for its safety and stimulation. >> This is very very interesting. Notice a different target group. This is also after school. >> Yes. >> This table, quite rightly, one of their desired outcomes was how it was going to affect the kids. This one, you want to have outcomes for parents. What was it? Parents of?. >> Parents of the children enrolled in the program. >> Parents of the children enrolled in the program. That is important. You're not going to be responsible for changing every parent in the school if you're not working with their kid. That is not fair. But you are willing to say parents of the kids enrolled in the program. Then what? The verb?. >> Regard. >> Present tense verb. >> Regard the program for its safety and stimulation. >> What of the seven magic words is that outcome?. >> I think it's value. Because they are going to pay for it. >> I like that. Yeah,question here. >> We have parents, dual is things here. Parents of students in program improve parenting schools. >> This is even different yet isn't it. A different target group. The parents. Again you have been also wise enough to say not all but just the ones of the kids we're working with. But your desired outcome is not something about the attitude or value. You actually have a what?. >> A skill. >> A skill. >> A skill. >> A skill. And read it again if you will. >> Parents of students in program improve parenting skills. >> Okay so the parents are actually changing in that way. I like that. Okay, good. These are good, great in fact. Other desired outcomes. Yes, sir. >> Student participants with and without disabilities greater ability to interact positively. >> Interesting. You have two target groups. You have, explain the two groups you have. >> We were thinking of the approach as being a CIL that we would probably be involving in this case envisioning children with and without disabilities. Participating together. Observing that they would demonstrate greater ability to interact positively. >> I really like that. You're thinking about what the program is going to be about, who is is going to be in it, and what we want coming out of it, what we want changing coming out of it. I like that. Let's stick with this after school program for a second then shift over to the seniors. Anybody else with after school program? Yeah. By the way, this is Joan by the way now talking. >> Teens demonstrate increased language skills. >> Teens demonstrate increased language skills. I assume by that you implicitly mean teens in the program. It's not a bad idea to say that, teens in the program. I like that. That is good. Anybody have the after school program with a target group different from either the kids or the parents? Curious. Maybe it's not possible. Just curious. Can you think of one? Anybody think of one? I can --. >> Yes. >> Maybe. Can you think of one? What might it be? >> If it's a school, the school could be operating the program so that more families see this as a community service. >> What would the target group be that we're trying to somehow change there? Benefit? >> School has more resources to offer the community. >> One way to put it. On the way might be the community supports the school. The community is seeing something good coming out of this. Just brainstorming different perspective. Let's shift over to senior program. Who had the senior programs? And what desired outcome? This is becoming increasingly relevant for me. I'm interested in what the desired outcomes are. Who has one? >> We have one. >> Go ahead. >> Service manage their health and wellness. >> Seniors manage their health and wellness. What kind of, what is the magic word, which one would that be? >> Skill. >> Maybe a behavior even if they are managing it. Not sure. Somewhere in there. I like that. Good. Excellent. Others. >> We have one, senior citizens attending our center classes incorporate exercise in their daily routines. >> I like that. You have that more specifically. Just the ones attending which they implicitly mean. You're spelling it out. That is good. What else? So far we have two target groups of the seniors themselves, which is fine. Others for the senior program. Come on, come on, some of you worked on it, I know. There you go, don't be shy. Individuals over 65 move into an independent living setting of their choice. >> MIKE HENDRICKS: Interesting. >> Home or family home, apartment. >> MIKE HENDRICKS: This is a desired out come from the senior program. >> We can't hear her. >> MIKE HENDRICKS: You need to use the mike a little closer. >> Hopefully it's on. >> It is. >> Individuals over 65 move into an independent living setting of their choice. >> MIKE HENDRICKS: We good with that? Anybody have a problem with that as desired outcome? >> Is the target group a little broad? >> More loudly please. >> Is the target group a little broad? >> The question is is the target group a little broad. >> The reason I say that, individuals over 65. Is that nationally? Is that in our community, in our state? >> Who knows. We're all learning. This is like the first time. We're at learning. >> That is why I ask the question. >> How would you maybe word it differently? >> I might say individuals over 65, you know, insert name of town or if you have a center. >> Okay. >> Over the age of 65 participating in our centers will --. >> Okay. >> Narrows the target group down a bit more so it's more specific. >> You know, there's nothing wrong with being specific about this stuff because you really don't lose anything. I'm not saying yours is not specific enough, don't get me wrong. I'm saying in general, the principle is there's nothing I don't think in being more specific because what do you have to loose. Whereas if you are less specific someone might not understand what you are talking about. >> The other reason I bring it up, sometimes if we are too vague, then we have another challenge with funders perhaps or something like that to overcome. We kind of present that challenge to ourselves and we don't need to. >> In a minute we're going to talk about that. I'll leave it there. How is that? >> Just feeding the lines. >> Thank you, I appreciate it. Anybody have a desired outcome they wrestled with just couldn't figure out how to put down. Sometimes that happens. Well, we knew the idea but just couldn't quite capture in writing. Anybody got one of those? You have one? >> I came in a little bit late. We kind of struggled with the action word. What did we say? >> Okay, so read what you struggled with. We'll see if we can help you. >> Latch key kids complete their home work. >> Latch key children in the program complete their home work on site. >> MIKE HENDRICKS: Okay, what was the struggle? Sounds good. >> Is complete an action word? Is complete a present tense verb? >> MIKE HENDRICKS: Anybody teach English? Is complete a present tense verb? >> Yes. >> MIKE HENDRICKS: Sounds good to me. >> In that context it is. >> MIKE HENDRICKS: I think it works fine. I think it is great. I'm pleased that you're all willing to at least humor me today and write them this way. I really do find it's useful and it will help you. It gives you this format which let's you really think through what you want to do. Any other questions? We're about, just to let you and our online friends know, we're about to wrap up this first session called outcomes and outcomes management and go into the second session. We don't want to do it too soon. I'm glad I asked. Here is a question. >> I think one of the things we struggle with at least when we started this is the outputs versus outcomes. >> Yeah. >> And I guess maybe as I was thinking about this when we took our break, in my head, think I have to write out what is the output going to be to figure out what the outcome from that will be. That was one of the things we kept talking about, you know, wait a minute, is this the activity they are going to be doing or is this what the outcome of that activity is going to be. You have, I know I'm asking this, it's probably going to be a no to my question, but do you have a magic formula to how you can avoid writing outputs as opposed to outcomes? >> I kind of do. >> Well, good. >> I don't know how magic it is. I'll let you decide that. How is that. I have a suggestion. Ask yourself, is it saying something about the program, or is it saying something about whom the program is trying to change. If it says something about the program, then it's an output. It's talking about how hard the program is working. If it's saying something about the people that you're trying to change, then it's an outcome. So for example, you give me an example. Give me one you were wrestling with. >> I don't even remember. Well, it was the one that we just did, which was the issue with the home work and completing the home work. >> MIKE HENDRICKS: As a group let's tackle this, let's help our friends out at this table. So the outcome, we're clear on what the outcome is. These kids are doing their home work, okay? What would be the output of the program's work that would lead up to that? Uh-huh. The mike should always be on. Maybe the wizard back here turns them on as you reach for it. >> We had the same struggle. I appreciate you bringing that up. I tell you what we came up with. Students in the program increased academic standard in at least one class, with the assistance of qualified staff in the after-school program. That forces us to have qualified staff helping the students, that we put an emphasis on assisting this emin their academics after school, and away measure that by knowing that they have increased at least 1 grade level, at least 1 grade, F to a D or D to C, in at least one class. >> I love your instinct but I'm going to give you an honest friendly warning, you're combining too much stuff in one. >> Okay. >> Keep the concepts separate and it will be much easier. I guarantee. I'm glad you brought this up. Let's talk about this a second. Let's play with it. >> One of the things I heard, what she said in my opinion, I'm hearing a goal as opposed to outcome. >> There is another question. Let's knock that one out right away. We don't use the word goal, purpose, objective. Give me another one. Any others? Don't use them at all. We don't use them at all. I don't use them at all. No need to use them at all. Outcomes covers it all. If you hold the thought for a second, I'll be able to show you. I don't talk about goals or purpose or objective. I know RSA does in some of their stuff, and in past trainings sessions we talked about how ours relates to theirs and how ours makes more sense than theirs, but that is another issue. Yeah. >> Mike, is it possible, another point of confusion might be created by the labeling of this column. The column is labeled desired outcomes. We don't know what the actual outcome is yet. We just know what our desired outcome is. >> Excellent point. >> We can't report on the actual outcome yet because we haven't performed, this is our desired outcome. >> And may I suggest --. >> Maybe that is throwing a wrench this there. I don't know. >> I think you raise a good point. Desired outcome is what you should be thinking about at this point, may I say. There are unintended outcomes sometimes, side outcomes. At this point ask yourself what is the desired outcome. You have a comment. >> In the case that Julia was talking about, wouldn't the qualified staff be the input? >> We're going do that right now. Thank you. >> Okay. >> MIKE HENDRICKS: If the outcome, the desired outcome, I hope everybody can see this. I need to get it in front of this thing too. If the desired outcome is kids do home work, which is kind of the essence of it, okay, let's ask ourselves, if that is the outcome, let's start at the bottom. Remember? What would be the inputs that we would have that we would want to use in order to make that happen? >> Qualified staff. >> Qualified staff. >> MIKE HENDRICKS: Some kind of staff. >> Or volunteers. >>. >> MIKE HENDRICKS: Volunteers. Don't forget the kids. Sometimes people forget the kids are an important input also. >> Academic environment. >> Some kind of an academic environment. Maybe some materials. Stuff like that. Pretty clear on that. Okay, now, we take these inputs. Remember what was next after inputs? >> Activities. >> MIKE HENDRICKS: Activities. What might be an activity that would lead to this? >> Tutoring sessions. >> MIKE HENDRICKS: What else? Maybe just tutoring sessions. >> Mentoring. >> MIKE HENDRICKS: Maybe some kind of mentoring to get them to pay attention to the tutoring sessions. Okay, now, these are inputs. We have that. Piece of cake so far. Activities. Now we have this empty box in here of outputs. What would outputs be? >> Numbers. >> MIKE HENDRICKS: What kind of numbers? >> Numbers of kids who complete home work sessions. >> MIKE HENDRICKS: Remember the the definition? Output is the volume of what? >> How hard we're working. >> Volume of work accomplished by the program. Volume of work accomplished by the program. How hard the program is working. Help me out here. What would be reasonable kinds of outputs? >> The number of sessions. >> The hours. >> Staff member hours. >> MIKE HENDRICKS: Of what? >> Tutoring. >> MIKE HENDRICKS: The number of hours tutored. How about number of kids tutored. That is another way to measure, right? Number of kids tutored. What else? >> Number of dollars spent. >> MIKE HENDRICKS: You can work hard for free or work hard for a lot of money. Know what I mean? Money might be down here as input, certainly down there as input. >> Materials used. >> MIKE HENDRICKS: Remember how hard the program is working. What would capture how hard the program is working when you're putting this stuff together, offering tutoring sessions and mentoring in the hopes that kids eventually will do their homework. What would capture. >> Hours. >> We have that already, the number of hours tutored. >> The number of kids completing their home work? >> MIKE HENDRICKS: Does that say something about the program or about the kids? >> That is the kids. >> MIKE HENDRICKS: This is about the program, right? This is about the program, the outputs. This talks about how hard you're working. This is about what you're achieving. This is about the kids. Let's write over here. This is about the kids. This is about you. How is that? This is about you. Okay. I'm glad you raised it. I can see we're not all a hundred percent clear on this difference between outputs and outcomes. Let's think of another question. Say if I were running a weight loss clinic. What would be a measure of how hard I'm working? I'm not in the weight loss clinic. I don't need it. I need the other. But what if I were running it? How hard would I be working? How would you capture? >> Number of fitness classes provided. >> MIKE HENDRICKS: Number of fitness classes I'm running. >> Hours you work. >> Number of hours I work on this. Number of? Number of clients I'm working with. Right. All these would capture how hard I'm working. Do these say anything at all about whether anybody has lost weight? No, not a think about whether anybody has lost weight. It says I'm working hard. It says I'm offering a lot of classes. It says I'm teaching a lot of people. Putting in a lot of hours. Doesn't say a thing about whether anybody is losing weight. Those are outputs, talking about me. What would be some outcomes? >> They lost weight. >> Number of pounds lost. >> Number of pounds. >> MIKE HENDRICKS: Pounds lost by people. Maybe the percentage of people who lost at least 10 pounds. Different ways to finesse but all the same thing, people losing weight. >> Number of referrals. >> MIKE HENDRICKS: That would be a nice outcome for my program but a different outcome, wouldn't it? But it would be an outcome. They refer other people to me. I like that, that is good. >> Decreased health risk. >> MIKE HENDRICKS: Probably a little higher, which feeds into our next topic, so I love you. Thank you. Does this help a little bit with the difference between output and outcome? I still sea some faces. I see Carol has some questions from online. >> Okay, someone has said seniors using personal assistance services help the employer learn effective communication which is a key to maintain independence. Is this okay for desired outcomes? >> MIKE HENDRICKS: Could you say it again please Carol. >> Seniors using personal assistant services. >> Stop there. That is the target. Seniors using personal assistant services. >> Help the employer learn effective communication, which is key to maintain independence. >> MIKE HENDRICKS: What do you all think? >> We don't know what they are learning. >> Too much. >>. >> MIKE HENDRICKS: Kind of a lot of words. Want to find way to cut the words down. The concept though, the seniors are learning something that can help their employers. Does that sound like a desired outcome of a program? To me it does. I don't know the program. I have to leave it to the experts who know the program. >> Mike, could you end that sentence after the word communicate? >> MIKE HENDRICKS: Probably could. Try it again and do that. >> Seniors using personal assistant services help the employer learn effective COMPLUN communication>> MIKE HENDRICKS: I'll give them a thumbs up on outcome. One more. >> Back to the chart, if I'm running the after school program, I'm not sure the only outcome that I care about is that kids do their home work. It may be an outcome and I do want them do that, but ultimately not unlike with your mission group in Washington, getting the job is one, but I wanted to improve their academics. >> MIKE HENDRICKS: Perfect segue into our next topic, thank you. So I need to say to the wizards in the back, we are now moving into the next session called logic models. You all also have a new set of hand-outs you might want to reach for. I will see if I can find what I need to find here. I'll have in the back let me know. We're ready to restart. Okay, great. A few minutes ago Bob used a word, a phrase logic models. Some of you probably know exactly what that is, and some of you probably have no idea what that is. That is fine, don't worry. It's jargon is what it is. The key thing, the tool is not jargony. The tool is a really handy thing for you. Really popular and really common. You have to know it. Something you need the know. We're going to learn it right now, we're going to learn about what logic models are. This is the second step on the yellow brick road. Now we all know what outcomes are. We all know why we're here for outcomes management, not outcomes measurement. We want to actually use this information to be able to be better. Let's move to logic models. A good way to think about it is that this is a lie. Even though we showed this to you a bit ago, this is the chart that has the 4 boxes of inputs, and if you have inputs then you do some activities with them, and because you're working with certain hardness, volume of work accomplished, you have your outputs. Then because you are effective, you're actually achieving something in your outcomes. Problem is just as you just said, my friend here, do all outcomes happen just like that, at the same time? No, what happens instead? Come on. >> They happen gradually over a period of time. >> Some happen gradually over a period of time. But what does that mean? Think about what that means. Say you have eight things you'd like to have happen in your program. Do all eight happen right away? >> Not at all. >> What happens instead? >> Some may happen at the end of the semester. >> A little closer to you. >> Some results may --. >> Outcomes. >> Outcomes may appear at the end of the semester in terms of academic standing. Another may show up 8 months later in that we have lost X amount of pounds and reached your outcome. Others may take, I don't know, a year or two in terms of developing a particular attitude or life-style so that you can improve your health, your academics, et cetera, et cetera, like going to college or maintaining different type of life-style takes a period of time to do that, a year or two, whatever. >> MIKE HENDRICKS: Absolutely. Only a fool would think all eight of your outcomes are going to happen day one. Just doesn't work that way, does it. Some happen right away, then they lead into some others happening, and they lead into other happening, and they build on top of each other and reinforce each other. We know that happens. That is the way life is. So this is wrong. If this is wrong, what is right? Well, this is closer to being right, isn't it? Sure we still have inputs. Absolutely we do something, we do some activities with those inputs. Because we worked hard, we have some outputs. But then there are different levels of outcomes, aren't there? Just think about it. I'm just capturing reality here. Nothing fancy about this. This is just what is real in the world. There's initial outcomes and sometimes they do happen right away, like learning. Sometimes you learn right away. That feeds into something else happening for independent immediate outcomes and just as you were saying, longer term outcomes. It's really a chain, isn't it, with different levels of outcomes. By the way, this is why I don't use the word goals or purposes or objectives because I guarantee you, you put ten evaluators, evaluation experts in a room, and you say goals, purpose, objective, order them in the proper order. You will not get consensus on what the proper order is. I have seen this so many times. You will not get consensus. So why worry about it? Let's just think of what is really real, which is some outcomes happen initially. They lead to some others, some others. This is what is real. This is the concept, the one we're going to be working with the next couple days, that yes, there are outcomes but they are not at happening at once. Now, if we had, I'm going to jump ahead actually. So this is a logic model. Jump ahead in your pages to get to this. This is the modified logic model in that it only has activities and outcomes. Only has activities and outcomes. It's up to you whether you put in more. I find this is pretty good. Let's look at this and see if we understand. What are the activities at the bottom? Mentors meet with at-risk teens for an hour each week. Mentors stress the importance of education, encourage school attendance, occasionally help with home work. This is another of those after-school programs, isn't it? They are working with these at-risk teens. These are the activities. First initial outcome they expect to happen, at risk teens complete home work regularly. If they do that, then the at-risk teens AERN better grades. If they do that, the at-risk teens achieve passing grades. They are saying it at the same time, it's not just home work that I care about. They also have to go to school. So another thing we want to have happen right away is that the at-risk teens attends school regularly. And if they do that, they will meet the district attendance requirements. And if they do both these things, if they achieve passing grades and meet district attendance requirements, then they will graduate from high school. You can see how they have some activities at the bottom, then different levels of outcomes working their way up in a logical manner. That is why it's called a logic model. It's a model of your program that captures your logic. You may not agree with this logic or you may have done it differently, but that is okay. This is their program and this is their logic model for the program. Let's go back. Hold this thought in your mind. Let's go back to right here. How could you use that 1 piece of paper? First of all it's a way to show visually on one page all the different things you might want to capture. If you want to put the inputs at the bottom, if you want to then have the activities right above it, if you want to then show how hard your program is working with your outputs, and what it's trying to achieve, the desired outcomes, and in what order. The different levels of outcomes. We just saw the different levels of outcomes. Key elements we marked here. Activities, desired outcomes and levels of outcomes. You heard me say if then a second ago. That is an awfully good way to think about a logic model, that it captures the if-then. So if we do these activity, and this is a good tip. I would circle this on the handout. This is a good way to think about these. If these activities, then this first outcome will happen. If that, then this next outcome. If that, then this next outcome. So it's the if-then working its way up. Here we saw it. If we do these things, then they will complete their home work. If they complete their homework, then they will earned better grades. If they earn better grades, then they will achieve passing grades. At the same time if they do this, they will attend school regularly. If they do that, then they meet district requirements. Here if they do both, then they graduate from high school. It's the if-then thinking that captures the logic. All right. If you had something like this for your program, for your CIL, something like this, and Bob is going to show you in a minute what they came up with for the CIL program in Nashville. If you had something like this, what would you be able to do with it? Well, frankly, a lot, is surprising lot. First thing is make the program design in theory very explicit. Stuff that was just in your head, stuff you had talked about with your staff, sort of had a sense of but hadn't put down on paper, now it's very explicit. Another thing you could do is to ensure a shared vision inside your program. Let's go back a second. What if not everybody on staff realized that this was your logic? Then they wouldn't be able to all be pulling in the same direction, would they? They have to all be pulling in the same direction if they are on the same program. Having something like this ensures a shared vision inside the program. It will be very interesting when Bob presents the NCIL logic model to see if that is your shared vision of what CILs are about. It also can communicate the program's rational to outsiders. You can say to one of your funders or to a potential partner or potential client, this is what we're about. Right here on this 1 piece of paper, this captures what we're about. That is a nice thing to be able to have. It orients workers to emphasize what is important. You say to them, look, guys, it's completing homework and attending school regularly. That is the essence. May be some drug problems, may be some sexual activity problems. Essence of what we're about is homework and school. It orients people to focusing on that. It also can help allocate resources properly to different activities. Down at the bottom you have different activities. Well, which ones are the most important? By looking at your outcomes, desired outcomes, you with pick out which ones are most important and help allocate resources to those. Also simulates a decision of how to define and measure success. Let's see. School regularly. Tell me, what is the successful attendance regularly? Tell me. >> Daily. >> MIKE HENDRICKS: Give me --. >> That was one of the reasons I was having difficulty with this. It doesn't seem real measurable. Some of these terms are a little subjective. >> MIKE HENDRICKS: I love it. You're exactly right. The comment was this doesn't seem at all measurable. It's not supposed to be. We're not there yet. Hold that thought. Yep. At this point what we're doing is asking yourself what do we want to achieve. Measurement will come. We're not there yet. So you tell me though. If we're now at this point of using it to stimulate a discussion of how to define and measure success, attend school regularly. You tell me, what is regular. How much? >> 5 days a week. >> 5 days a week is regular. You're a tough one. >> 80 percent. >> Four out of five, uh-huh. That is the only two candidates? >> 90 percent. >> Okay, there you go. >> Whatever the district's requirements are. >> Whatever the district requirements are, that is another way to think about it. If you don't have this and you don't know that attending school regularly is a desired outcome for you, you can't very well talk about what it means. Right? So you have to figure out what it means. Last way to use it is to help identify which desired outcome should be measured. How many desired outcomes are up here? Count them. >> Six. >> MIKE HENDRICKS: How many should be measured? >> Six. >> Five. >> MIKE HENDRICKS: Somebody says fix, somebody says five, somebody says one. See, already we're helping if I want which desired outcomes should be measured. You can't do that if you don't have something to talk from. So I know you're not quite convinced yet. I can see it on your face. We're going to keep working with this. I can tell you, these logic models will be very valuable tool for your local CIL. Keep an open mind about it for a little bit, okay? We'll keep working on it a little bit. This is another tip about thinking about a logic model. I would like to put a star or something on this page, if I were you. This is another one of those things just really really a good tool, good technique. What it says, when you start at an outcome on a logic model and you go up, you ask the question why. I'll demonstrate this in a second. Get this concept. When you start at out come, you go up and ask the question why. You go down and ask the question how. You go sideways and ask what else. This is a way of checking your logic and seeing is this logical, did I screw this up somewhere. Here we go. Let's try it out. Start right there. Why do we want at-risk teens to earn better grades. Right, the answer better be right above. The answer be right above it. Because we want them to achieve passing grades. Why do we want add-risk teens to attend school regularly? We want them to meet district attendance requirements. So far sounds logical. Let's work down. How are we doing get at-risk teens to earn better grades? There you go, should be right below it. At risk teens complete home work regularly. How are we going to get them to attend school regularly? Interesting, the thing right below are the activities. What we're saying is our activities are going to make this happen right here. First thing. That is a very very useful tip. Here you can see it again. Here is number two. How far up should your logic model go? What is the top of this logic model? Somebody read it for me. What is the top of the logic model. >> At risk teens graduate from high school. >> MIKE HENDRICKS: At risk teens graduate from high school. Why did we stop there? Why don't we have at risk teens enter college or technical training program? Why not? We probably want that, don't we? >> It's a separate goal. >> Outcome. >> Outcome. >> MIKE HENDRICKS: Thank you. I thought I heard someone say a TIRTy word over there. I didn't hear goal. I heard outcome. Why didn't we though? Why didn't we keep going? Yes>> MIKE HENDRICKS: It's outside the purview of what we have responsibility for? >> MIKE HENDRICKS: It might be. That is the kind of question that you want to ask yourself. Ask is it far enough up to show meaningful change for clients. You tell me, is that meaningful change, at risk teens graduate from high school? That is meaningful. >> Yes. >> MIKE HENDRICKS: That is good. Is it not so far out that your program can't reasonably influence the outcome. I think we could, if we ran this program, could we reasonably influence this outcome do you think? >> Yes. >> MIKE HENDRICKS: Graduating from high school? They seem to think so. Could rewe reasonably influence them going on to college or technical training? Maybe. They seem to think not. Maybe. That is a judgment call you're going to have to make. How far up to go, when you're working with your CIL and you have desired outcomes for your people, you're going to have ask yourself, how far up should my logic model go. I don't know the answer. You're going to have to decide. This is a tip to help you decide. Make sure it's meaningful. Make sure it's also within your reasonable influence. Now, Bob is is going to tell you and show you and walk you through the actual logic model for the CIL program . >> BOB MICHAELS: Thanks, Mike. Now we're going to see the connection between the outcome measures, the desired outcomes that we talked about the last time, and logic model that we put together now. Before I do that, we're going to see if you look through your slides there, you'll come upon a bunch of slides that show in black and white. What I want you to do is, I think it will be helpful for you, in your packets you have a thing called logic model for the CIL program. It's in color. We're going to use this throughout the training. First four or five pages are relevant for this discussion. I'll use the rest of it this afternoon and tomorrow morning. If you look in your packet, you'll see a colored packet of information. I really want to thank IL Net for going ahead and putting this together and NCIL for putting this packet together. Especially the colors. It's a very expensive project and difficult do. We wanted to be sure you had something in color. Now, I'm going to show you a knowledge lotic model. The reason I want to show you the entire model is just so you can get an idea of how it fits together. What we're going do then is take a look at the different streams that you talked about earlier, the one with IL services, and advocacy. What I want to do is have you see how these all fit together in one logical model. Those of you that cannot see the screen, you should feel assured nobody else will be able to either. I want you to see what it looks like, then we'll look at it more specifically. Also for you who are not able to see the screen or want it, pages two and three are the logic model in text. Kind of updated and we have written that out for you. Those of you that want to know how the logic model fits together can look at it in text as well. Here is what we came up with. Now, I'm here to tell you that we came up with a logic model, and I took it out in the field and said what do you think. We showed it. We did it at NCIL in April, we did it at CIL Congress. I did it in regions 10, 6, 4. We did it in several states. Arizona, Minnesota. Just got input from everybody. Every time we got input, we came back to the task force and said okay, does this make sense or not. Sometimes, very often we changed it. In fact, we made the last change about, oh, two and a half years ago just before we started the field testing. So this is something that is real dynamic and changes all the time. So here is what it looks like. Now, this is the proposed service model. Bottom, if you look at the one in your packet, the bottom of the activities, the IL services, INR and systems advocates. On the left side going up is the initial outcomes, the intermediate outcomes and ultimate outcomes. You notice that the ultimate outcomes then are up at the top in blue and the very top one is that people are disabilities are integrated into American society. If we did everything that we want to do as centers for independent living, then the people with disabilities would be integrated into American society. Here's a good example of something that changed over time. One of the very first changes we had to make was we had people with disabilities are main streamed into American society. We got all kind of comments from people who felt the mainstreaming was too tied to education and it was not a good word to use here. We had to talk about integration. People said well, you know, it's too tied into racial things and to changes that have happened in civil rights. But just as many people said, that is exactly what we want to do. So we ended up with people with disabilities being integrated into American society. Now, let me show you the streams because this is too much. If you look in your packet, you will see on page three of the little packet that I just gave you has the stream for IL services. >> MIKE HENDRICKS: Box it's page 4 . >> BOB MICHAELS: That is right. We have one also for the ultimate outcomes. This stream, the first one for IL services. This is all the pieces that have to do with any services that are provided by independent living that are not INR and are not systems advocacy. So it says just about everything else. Peer supports, skills training, transition services, transition assistance, individual advocacy, all kind of things. If people have those services from the independent living centers, if they get that, then they have the skills and knowledge and resources to support their choices. If they get that, then they make their own choices. If they get that, two things will happen. They will regard themselves as more independent. In fact, they will be more independent. Okay? Again, this is what you felt, what the membership in all centers for independent living, people in independent living around the country felt should be the desired outcomes. Here is the I and R stream. If you get I and R from a center for independent living, people with disabilities see different possibilities, and they get the information they need. If those two things happen,, if people get the information they need, then they begin to advocate for increased community supports. Notice how the arrow goes out to the right side. We'll have people who are advocates forcommunitir community support, they are does systems advocacy. This is the systems advocacy stream. If you identify the barriers and problems in your community, then the community agenda for change exists. Active coalitions exist around our issues. If those two things happen, then decision makers will act on our agenda. If they do that, then two things will happen. Communities will have more resources that support independence. And methods and practices will be felt that promote independence. Now you have the three streams. What happens when we start off with IL services, I and R, systems advocacy, and built them up. What you're looking at here are initial and intermediate outcomes. Here is the ultimate outcomes. People with disabilities participate in the communities to the extent they wish. That comments up from the IL services stream. Communities are more accessible in housing, transportation, information, employment, education, AT, healthcare, and so forth. That KOOMS up out of systems advocacy and I and R. And if those two things happen, then people with disabilities are integrated into American society. >> Bob, I hope I'm not splitting hairs. Would you go back a screen for a question. >> Sure. >> This has always been a pet peeve of mine. Maybe I be splitting hairs but maybe it is just semantics. When we use the word independence promoting independence resources that support independence, are we really talking about interdependence? I don't think anybody is really truly independent . >> BOB MICHAELS: We have lots of discussion about this, about interdependence. >> Okay . >> BOB MICHAELS: And people just decided that independence was such a known word within independent living that it was better to stay with that and not worry about interdependence. I know what you're saying. In many ways I agree with what you are saying. >> Okay. >> BOB MICHAELS: Yeah. Here is one of the interesting things as we started putting this together. You notice this whole thing, this is all tied to, sorry, communities that are accessible in housing, transportation, so forth. That is an ultimate outcome. However, that is where we started when we started, when RSA started coming up with what it is that we report on for independent living. Instead of doing, looking at your community and trying to decide what it is that the community needs, this is starting to put together a plan to address that, then we're here with the decision makers to get that changed, having them act and all that kind of business. Instead of doing those things, they were going to communities that are more accessible for housing and transportation. They said we want you to start reporting on transportation, AT and health. They started doing that. Well, our problem with that was, we don't know if that is what the community wants us to do. >> That is right >> BOB MICHAELS: You as a center may be in a place where you have gone to your community, worked with your decision makers and all that business and now are to the stage where you know exactly what the areas are that you want to emphasize. You wouldn't start off by saying this is what we want to emphasize, you know, transportation, AT and healthcare. So you know, what we want you to do is try to work from the bottom up. Now, you may, again, as I said, you may have already passed through and been through and done that within your center. When we ask centers to prioritize these, as you will see later, that is not where they were. The majority of centers were still trying to decide how is it we organize within our community and respond to their needs. Not the bigger picture. Okay, again, this is how the whole thing looks. See, the activities and the outcomes on left side, activities on the bottom, the three different streams. Then you have the ultimate streams where they all get together and work together. You notice things, people are doing good work in I and R, this is connected with the two streams. That really is the picture. So another word on I and R. If you are like my centers, we didn't really track I and R very well. We had long discussion about this on whether it should be reported on this, including the outcome measure process. You came back to us and said no, we should. It's important. The task force and the membership all felt that these are core services. And because I&R is core service, we really need for it to be part of what it is that we report on. Okay, any questions? >> Bob, I have a question. I do remember when you brought this to region 4, I think it was 1840 or something . >> BOB MICHAELS: It has been a long time. >> It has . >> BOB MICHAELS: This is one of the amazing things. I can remember us sitting down in 2006 and sitting down with Mike and saying, okay, how will we do this. You know, he started talking about the different steps that you need to go through it. Although if you are in a field test, which is an incredibly critical part of what we do here, and we put time lines on it. I said boy, this is going to be like 2011 before we're done here. He said yeah, well, it will be. And who would have thought, you know, here we are. I'm sorry. >> To back up a slide if you don't mind. I will say that it's come a long way and streamlined a lot since the plate of spaghetti that you showed us in region 4. One of my concerns is that those are not activities. In a logic model as I understand it, activities are verbs, and we don't have too many verbs. These are programs that we have, but one of the things that would help me a lot as a center is to see what sort of people think their activities are according to what Mike was teaching us this morning. That is one of the things that I wanted to ask you about, this doesn't, you know, I know what my programs are, I don't know what my verbs are. The second thing that concerned me then and concerns me now is what kind of happens if I think there's a flawed logic in the logic model. I'll tell you where I'm going with it. Let's just take systems advocacy, for example. I'm from Charlotte North Carolina. We do have for example a transportation program where we have identified the barriers. We have identified the agenda that needs to be done. We have a coalition of people with disabilities and community service providers. And we have gone to our decision makers with proof, and they laughed at us. So the only way that we're getting the decision makers to act on our agenda is not having an agenda, and telling them what the agenda is, but it's beating them over the head with a federal complaint. So I think there's a step in there, you know, I'm not sure it actually flows. With people with disabilities, they get more information, they will become advocates, that is a leap I'm not seeing in my community. >> BOB MICHAELS: With regard to the activities, we tried to look and say this is what a center provides, the activity the center does systems advocacy. If they do systems advocacy, so it just seemed to fit in. People wanted early on for us to have four core services. That was a model we had. We just didn't feel that it fit in correctly when we took a look at it. What you need to understand is that this logic model is a logic model that we tried to make reflective of what all centers in the country or most centers in the country would follow. We understand and understood then that this was not ready for everybody. The things we emphasized were not right for everybody. Some centers, in different places than we, you know, than the task force and people that responded to us. But it was clear, you know, that is really the point of what we want to try to get here, this is not ours. This is not NCIL's. This is the whole community's. You do, you know, you can use it or not use like you want. If you feel it doesn't fit, knock something out, put something else in. You know, there's nothing that says that this has to be the logic model that you use. You should use that. >> Continuing on the example that was given in terms of getting to the decision maker point and having them laugh at you or dismiss your suggestion or whatever, that is really a flaw in the logic model as I see it. What that means is that perhaps that step in that model, we maybe need to look at some different activities that will bring us to this output. We may need the look at some different strategies or activities. But it's not really, yeah, not really a function. We do that and have a very very active, Minneapolis St. Paul, in the eight centers in the state we have an a very active collaborative system that does the activity. In addition to serving as executive director of one of the centers I'm also the lead lobbyist for the association. We engage in a lot of those kind of activities and we get to that point where we get to the decision makers and they are not buying in. So we take a step back. We don't change our logic model but we change some of the activities that will brief us to those desired outcomes. I would suggest to maybe look at that as not really a challenge in the logic model but a different step in the process. Does that make sense? >> BOB MICHAELS: Make, anything you want to? >> MIKE HENDRICKS: If I can, a couple comments from my perspective. I like the idea actually of putting the ing words down in the activities. I think we originally didn't because it's already kind of a busy chart and we were trying to keep it somewhat pretty. I think you raise a really good point and perhaps that would be a nice thing for us to try. You know, give it a shot and see. I have to defer to those who know more about exactly how change happens out there in your field, whether you need to have another box in there of a desired outcome or whether it is an activity or not. But this is, Bob is exactly right. What this is doing is stimulating exactly the kind of discussion the task force wanted to stimulate, is to have you, from your CIL's perspective, look and say does this work for me. Is this close to what I'm doing. Trying to achieve. Or does this not work. If it doesn't work, fine. Change it into what does work for you. I think the task force feeling is you need to think this through for yourself, your own CIL, something like this. Would you say, Bob? >> BOB MICHAELS: Yes. >> MIKE HENDRICKS: I think you really need to attend to this and focus on this and ask yourself does this work for me or does some variation on that work for me. . >> BOB MICHAELS: Any other questions Carol? Karla. >> Can you talk about under information and referral the outcome that people with disabilities advocate for increased community supports and why it is in that position there? That if you provide, if people get the information they need, that they are going to advocate for increased community supports. >> Now, this is a logic model, it is what you want. You know, desired outcomes. So we're talking about that we felt very much that if people, if an independent living center gave the information out, that they would get the information they need, but that if they got that kind of information, some of them, the ultimate thing you want to happen is that people would with disabilities would then get the information so much they would want to become advocates. Some wouldn't, but somewhat. And the feeling of the group is that is what we want to see it go. >> MIKE HENDRICKS: Bob, can I ask you to go to the big picture. Bob very nicely thought it would be and it is helpful to separate the streams out so you can see the separate streams. But remember they all fit together. Let's look for just one second if we can. The one you're talking about, advocating for, try that again, advocating for increased community supports. Remember what we talked about about how sometimes things come together. If you look here, you can see that three things together make that happen. Seeing different possibilities, getting the information they need, and a consumer agenda for change existing. So maybe it's more palatable to you when you put that in, then you see that it would lead,en I don't, I'm just pointing that out. There is that third outcome that also feeds into it. I want to make sure we see that. >> Okay, that does make it clearer. I thought you had missed a step. I thought we haven't identified the barrier, you don't have your agenda. I wasn't looking at the whole big picture. Thank you. >> This helps. >> I think in a CIL atmosphere too it's based on the presume shun that if they get the information they need, it will be looking. Well, there is no transportation in your area. There is no healthcare clinic within 30 miles. You know what I mean. That will therefore prompt the advocacy for those necessary services . >> BOB MICHAELS: Exactly. It's desired outcomes. Anything else? Other comments? Mike. >> MIKE HENDRICKS: Except now it's your turn. It's one thing to talk about a logic model. It's thing to look at somebody else's logic model. Let's see how good you are at it. On your table there's a set of index cards, somewhere in the middle. Different color. Pull those out if you would, please. One of you reach for them. Are there nine? Let's see. Nine. Nine index cards. Here is the trick. If you arrange these index cards in a pretty reasonably intelligent order on your table, there will be inputs, activities, outputs, and different levels of outcomes. And those levels of outcomes fit together nicely. That was an if. That is your task. This box here, is this the top ultimate outcome? Or is this the input? I don't know. This next one, is this an outcome, this is an activity? I don't know. Work on these index cards. Arrange them into a very small logic model, please. (Card exercise). >> MIKE HENDRICKS: Two more minutes then we will see what we have got. All right, everybody has solved this conundrum? You want to turn over the pieces of paper I put on your table and see what you think? You don't have to agree. One thing about logic models, there is no right or wrong answer. Different people can have different opinions just like Bob was saying. Look for a second and then we'll go over it. >> We're not as clueless as we look. >> MIKE HENDRICKS: There you go. Let's say we go over it together and see if we can understand their logic. Remember, let's remember what a logic model is. Okay. It's a model of the logic of the program that is why they call it logic model. These people had a certain logic. Let's follow along and first at least understand their logic. Down at the bottom, inputs. Right? Lots of things we all know those are the inputs. What do you do with those inputs? Activities. Program manager, doing this, doing that. Activities. Outputs. Right from the activities. What are the outputs? Outputs are the. >> Number. >>. >> MIKE HENDRICKS: A particular phrases. >> Volume of work. >> Work accomplished. >> MIKE HENDRICKS: Volume of? >> Volume of work accomplished by the program. >> MIKE HENDRICKS: Volume of work accomplished by the program. Those are the outputs. What have we got? Number of teens served. Number of hours of instruction provided. Number of hours of counseling provided. Outputs, how hard the program is working. Does this say anything at all about whether anybody has changed? >> No. >> MIKE HENDRICKS: No. All right. Outcomes, let's look and see what their outcomes were. Notice they have two completely different streams. That is interesting. What are the two streams about? How would you describe the difference between these two streams or columns or whatever? What is the difference? >> One is pregnant and one is not. >> One is pregnant and the other ones already had their babies. >> MIKE HENDRICKS: One stream has to do with before the baby is born. The other stream has to do with what you're going to do after the baby is born. So prenatal, post natal. They call it that, prenatal care classes, counseling, then infant care parenting, classes, counseling. That is interesting, isn't it? Let me ask you this question, why wasn't their program just these 3 boxes right here? That would have been a fine program, wouldn't it? Pregnant teens, prenatal nutrition and health guidelines. If they know them they follow prenatal nutrition and health guidelines. If they follow those, the pregnant teens deliver healthy babies. That is not a bad program just by itself, is it? Right? Why didn't they stop there? >> They wanted to go further. >> MIKE HENDRICKS: That would have been a fine program. Let's get healthy babies born. Gosh knows there are some unhealthy babies born. Use the mike please. >> Some might come in as teen mother. Might not be pregnant. >> MIKE HENDRICKS: Could be some would come in as teen mother. Good point. What else? >> Ultimate outcome was babies achieve appropriate 12-month milestones. >> A wonderful disembodied voice. I have no idea where that is coming from. Thank you. Would you say that again. >> Ultimate outcome is babies achieve appropriate 12-month milestones. >> Why is that their ultimate outcome? Why isn't it pregnant teens deliver healthy babies? That would have been fine. >> You're looking for sustainability. >> MIKE HENDRICKS: Who says this? >> Me. >> Say again? >> You're looking for sustainability. >> MIKE HENDRICKS: You think they were looking for sustainability. Okay. Over here. >> The program as funding follows the child through the first 12 months. >> MIKE HENDRICKS: Okay, maybe. I don't know. Quite possibly, yeah. Another one. >> Because they established why their agencies exists. >> MIKE HENDRICKS: Ah, because this is their way to define why they exist. They, the answer is because. It's this way because. Because they looked what the they do, why they go to work each morning, and they said, we go to work for more than getting the healthy babies delivered. We go to work so that in 12 months, those babies are in good shape. Now, if given the top, given that is the top ultimate outcome, would these three have been fully sufficient? These three outcomes on the left-hand side. Sir, you're shaking your head no. Why not? Use the mike for us please. >> You're talking about their motor skills, verbal skills, social development. All the things that happen to them after they are one. Get them here is one thing. It's like the printout we have here, it says why, how, and what else. This is the what else. >> MIKE HENDRICKS: Oh, that is fantastic. Thank you so much. You're exactly right. I'm so glad you caught that slide. This is exactly the what else. The stuff over here is important. These outcomes are really important. Getting the baby born healthy is very very important. I'm so glad, you obviously caught that, is you need something else. You need what else if you're going to get up here to the top ultimate outcomes. They decided that they had to have two completely different streams of outcomes leading to the top. Now, that was a decision they made and that is the thing about logic models. Nobody can tell you how yours should look. There is no right or wrong way. You have to, at your CIL, you have to ask yourself, what makes sense for us. Okay? Come up with something. Other comments about this? Other people who maybe had something quite different from this and see a different logic. There's different ways of looking at stuff. Anybody have something different? Does this seem to make some sense? Seem to make reasonable sense to you when you look at it this way? Now, big question. Be honest with me on this. Does this logic model stuff make any sense to you at all or seem like research jargon and have nothing to with your CIL. I'm perfectly fine to have someone say that. >> No, I think it makes good sense. Our group turned a bit with regard to the outcomes because so many are intertwined. I think how that plays out, particularly in this scenario applied to our own CILs in our own cities and situations, those outcomes I think are going to look different. >> MIKE HENDRICKS: So you will have different outcomes, obviously, from this. Do you think you will have different outcomes from this? This is the proposed NCIL one. I shouldn't say that. That is not fair. Proposed logic model for the CIL program. You think you will have different ones from this? >> A little bit. >> MIKE HENDRICKS: Someone says a little bit. That is fair. As Bob said, this was an effort to try to capture something generic, a generic CIL program. How impossible is that? You know. But go ahead, please. >> I was going to say, I think that is just it with this model, is that it gives us kind of an infrastructure to work off of. It isn't committing us to any one thing. It's giving us a very visual and organized description of a way that it could go and could be successful. I think within that we can really, you know, apply it to our own environments and it would be very effective. >> MIKE HENDRICKS: We're not trying to, I hope it's clear we're not trying to force this on you. We're just showing you one possibility for you to do with as makes sense for you. >> Yeah, Mike, going through this logic model, it looks like they have something pretty much every step of the way that can somehow be measured, which, in some form or fashion, I guess. That is certainly, would you say that would be a criteria for including something in a logic model, that it can somehow be measured? >> Dan, do you want to introduce yourself to people who don't know and your role in this? You have a special role. >> I'm Dan Kessler. I'm the President of NCIL this year and I have been the president of the outcome tack force for a couple of years. >> So Dan has been --. >> I'm also Executive Director of a Center in Birmingham Alabama. >> Your title is President of NCIL. >> Yes. >> Dan has been very involved in this since the beginning. And I think you raise a really good point about that. >> Do you subscribe to the theory if you can measure, it's not worth doing? >> MIKE HENDRICKS: No. What you say is that some of these things look measurable and some maybe do not look measurable. Is that what you're saying? >> Right. >> MIKE HENDRICKS: What I would say, we talked about this early, I would say don't even think about measurement yet. Because if you think about measurement, you'll design your program to achieve things you can measure easily. That is not what we ought to be about in the world. We ought to be about designing our program to achieve what we need to be achieving and worry about measurement later. A lot more stuff can be measured than you might think. A whole lot more stuff can be measured than you might think. There are people who spend their whole lives, their whole careers, figuring out ways to measure really tough stuff. They write books about it, they give conferences about it. There's a whole group of people who spend a lot of time thinking about this stuff. And we can steal ideas from them. But if we start off by limiting ourselves by saying let's only try the eight outcomes that we know we can measure easily, oh, what a disaster that is. Then why are we here. I would say, Dan, no. I would say let this be what you're trying to achieve in the world. And even this afternoon and tomorrow, we'll deal with measurement issues. But take step by step. That is why it's the yellow brick road. Take one step on the road at a time. We will get to that. I would is a no, don't be limited by that. Excellent question. >> I think there's a lot of freedom in not having to subscribe just to what is measurable. I think when we can dream and be free to write down what makes sense in time and space for our independent living center, then from that like you said, I'm glad that you submit to stealing. I think there's a lot of wisdom in gleaning from the wisdom of others. >> MIKE HENDRICKS: Of course. I'm glad to hear you say that. I don't want you to dream pie in the sky stuff, but I want you to ask yourself as we talked earlier, reasonably influence, up at the top, reasonably influence, what are you there for. Why are you there? What are you trying do. Put that on your logic model. Absolutely. Other comments about logic model or any of this? Yeah, sir. >> Looking at the chart with the arrows going up with inputs, activities, outputs, initial, intermediate, long term outcomes. >> MIKE HENDRICKS: Hang on so I can get back to you. This one also? >> Well, yes, that one. >> Okay. >> Looking at that makes it look like a real clean line starting with inputs, activities, outputs, initial outcomes. I think when you get past that initial outcome, sometimes you have to go back to the inputs, activities and outputs again and you have to go back again and again and again before you get to that longer term outcome. >> MIKE HENDRICKS: You're raising the question of this cyclical maybe nature of some of these and this is too simple. Again, it's called a model. It is a model, a simplified model. I'm glad you said that. Let's see if we can get back to this. The idea is to try to capture the essence of it. You're certainly right there would certainly be feedback things at various spots. We could try drawing those arrows if you wants. Starts to get messy, but we could. You raise an excellent point and I'm glad you did. Yep, thank you. One more, please. I'm glad you're engaging with this. This is a tool for you. >> I would add to that. I think it goes back to the previous point that you have to personalize it and it's not something that is set in stone. You might have to rework it. >> I like that. I like to say every object model should always be marked draft. We're always learning and rethinking and finding something. Like you were talking about earlier, political pressure or threat of a lawsuit. For us we all, we ought to always be rethinking our logic. I'm glad you said that. Anything else? Let's let you take an early break, but let's catch what we did today. You are ready. We have already thrown a ton of stuff at you and not even noon yet. You already clarified what outcomes is, difference between measurement and management, what a knowledge lotic model is. You practiced your own. Bob has shown you a bunch of stuff. Now here is the point where we are right now. Let me give you example. How many outcomes are up on here? Someone says bunches. That is a qualitatively oriented person. Anybody counting them? Sixteen. After lunch we're going to ask you two important questions. One, you have the money to measure all 16? That will be one question. The other question will be of the ones you do have the money for, how are you going to do it. That is this afternoon. Okay, any reason, yes, I'm supposed to tell our online friends that we are all going to lunch because it's lunchtime out here in the Pacific northwest. We will start back precisely or pretty darn close to 1:45 Pacific time. Pacific time. Please join us, folks. Folks here, the lunch is outside there or will be in a bit. We will start at 1:45. Thank you so much.