IL and Outcome Measurement: What You Need to Know. March 12, 2007 Presenter: Mike Hendricks. OPERATOR: Your National Council of Independent Living conference call. Today's first speaker will be Mr. Tim Fuchs. And throughout the discussion, all participants will be muted, you will have the opportunity to ask your questions by pressing 01 on your telephone key pad. As a reminder, your conference call is being recorded and now without further delay, I will turn your conference call over to Mr. Fuchs. TIM: Thank you, Dawn. Good afternoon and thanks for joining us today for the conference and webcast. Independent living and outcome measurement, what you need to know. I'm Tim Fuchs, operations director at the National Council on Independent Living. And as part of this position, I oversee NCIL's training and technical assistance activities for the IL NET. If you're not familiar with the IL NET, it's a collaborative project of ILRU which is the Independent Living Research Utilization program in Houston, Texas. And the National Council on Independent Living in Washington D. C. I will L. net is a national training and technical assistance project funded by the Rehabilitation Services Administration or RSA, which serves to strength en the independent living movement by supporting centers and statewide independent living councils. Through national, regional and distance learning activities, IL NET offers workshops, teleconferences, technical assistance, online courses, training materials and other resources specifically for CIL's and SILC. Our trainer today is Mike Hendricks. She a program evaluation guru providing assistance to government at all levels, foundations, nonprofits and other community organizations, what's most important to us today is Mike's role as a facilitator for NCIL's outcome measures task force. I want to thank Mike for being here today for preparing their these two webcast. Also on line with us today is Bob Michaels and my thanks goes to Bob for joining us today and for his help preparing for the calls. And now before I turn it over to Mike, I wanted to direct everyone to our training manual and evaluation form for today's call which is located on our website at www.ncil.org/outcomes.html. I'll be giving that repeatedly throughout the call. The manual and evaluation form that are located on that site, the manual is vital to following along with today's presentation and the evaluation form is equally vital to us as we prepare for upcoming calls. So if you would please take a moment at the end of the calls to fill it out and let us know what you think, there is one evaluation form for both calls so if you're participating in both calls I'd appreciate it if you'd wait until Wednesday and respond to both training activities. Another quick note about the materials, if you've received accessible formats such as Braille or large print, they have the same page numbers of the manual that Mike will be referencing during the call. In other words f the information on Page 10 of the ink print manual that Mike is references takes up three Braille pages, each of those Braille pages will be labeled page 10. I hope this helps you to follow along and if you think there is a better way to do that, please let us know in the evaluation form. So thanks for listening to our housekeeping bit and without further ado I'd like to turn it over to Mike. MIKE: Thank you, Tim. I appreciate that very much. I'm not sure I'm a guru but I'll take the compliment any way. Thanks everybody for being on the call. I understand that there are actually quite a few of us -- quite a few of us, so I'm not going to take that as a personal compliment, but I am going to take that as an indication that outcomes measures is something we're all interested in and that makes me feel good. There is a couple of reasons I'm really, really happy to be in this position today. First one is I'm really flat erred obviously. Tim and Bob could have invited somebody one else to do this. I'm honored they asked me and I do appreciate that. the second one though is that I am absolutely a true believer in outcome measurement. I have -- I work with a lot of local organizations for 20 some years now and I've just seen with my own eyes that outcomes measurement is something that can help you. In my opinion, this is not something to be avoided. It's not something to be feared. It's not something to be done reluctantly. I'm guessing that none of you woke up this morning and said, oh, boy, today I get to learn how to measure outcomes. I'm just betting that didn't happen. But I understand that, but I do hope that by the end of the call, you'll have a feeling at least in your mind that you know this is a tool that can help me. This is something I can use myself to make my CIL be as effective as possible. I'll give you a couple of examples as we go on, but I know that it's possible because I've seen that happen. So that's what I hope we can take away from today's call along with some specific information, but I do hope we can take away and attitude of this is something that can help me and if I do it well, it will help me. So having said that, if you would, please -- and you have a book that Tim mentioned -- turn to page vi, which is Roman No. rule 6 and it has the agenda on it today and we're going to try to accomplish three things, three main topics we're going to cover. One of them is going to be the reasons that outcome measurement is so important to us these days, us in the CIL world and there is a couple of different motivations for that as you'll see, including something from OMB which we'll stalk more about. The second thing I'd like us to try to do today is to make sure we're all very aware of what's happening, what NCIL is doing in response to this -- these pressures, if you will, or these forces. I think there is a lot going on and you should know that I think. And the third thing I'd like to do is let's begin to learn ourselves a little bit more about outcomes, let's learn something about the difference between outcomes and outputs, let's learn about this jargon term logic model. You may have heard this phrase before. Let's learn exactly what that is, and then let's learn about indicators. This will give us a very good grounding of what we're talking about when we mean outcomes measurement and then on Wednesday, we'll go into very specific information about how all of those things apply to the independent living world. So that's what we'll cover. As Tim mentioned we'll have lots of time for questions and answers after each section and then at the end. So please let's make this a nice interactive day. Why don't we, if we can, turn to what I think would be page ix and let's talk about what I think are the two forces that are actually pushing all of us to be on this call today, and I think one is internal and one is external. Internally, and this is history that I'm guessing all of you know better than I do. I've been working with Tim and Bob for about the past year but I'm guessing many of you have been working for many years. So you know this. Over time, the CILs as I understand it have felt that they've been doing very good work, but they haven't always felt that that work is being captured, especially perhaps not being captured by The 704 Report. In other words, I think you've felt that you've changed people's lives for the better, you've improved community conditions, you've done good system advocacy, but it's not been captured in any way that you document and feel good about and show to other people. So my understanding is that several CILs on their own on the local develop worked in this area and developed good materials to capture these outcomes and I've seen some of those. I've seen Michigan, I've seen Georgia. I think there are several others, forgive me for not mentioning them and then those CILs said we think this is good work. How come NCIL -- maybe NCIL ought to get together and do something at a national level. So there last been this internal pressure basically you'll say from the bottom up pushing NCIL to do something about outcomes. At the same time, and many of you probably know this -- at the same time at the federal level the Office of Management and Budget, OMB, has been assessing all federal programs. And let's look, if we can, at Pages 1 and 2 of our book because here we have some pages taken out of the OMB website, www dot expects more.gov and these pages relate to the program assessment weighting tool or the part or generally called the part process. The part tool. And this is the technique that OMB has been using for the past five years to assess every single federal program and if you look on Page 2 at the bottom, you'll see that by now in the past five years they've just about covered the full range of all federal programs. They've courted about 96 percent of them and they've given them a rating and you can even see here what the different ratings are. For instance, you can see at the bottom of page 2 the number of programs assessed now have been 977. And you can see that 17 percent were rated effective, 30 percent were rated moderately effective, 28 percent adequate, 3 percent ineffective and 22 percent results not demonstrated. Now, let me just mention that these ratings are important in Washington these days. Every program knows what its PART rating is and every program is responding to what its PART rating is. This is an important government exercise. Well, as the CIL program been part, yes, it has been parted. On Page 3 you can see the results again from ExpectMORE.gov. You can see the results of what happened when the CIL program was parted. It happened, by the way in 2003. And it was scored by OMB and education together. They do this together. And the rating given was results not demonstrated. Now, results not demonstrated is an interesting score, an interesting rating. It doesn't mean you're ineffective, but it means we don't have any evidence of whether you're effective or not. And in OMB's way of thinking, that's not a good thing. It's not good to be a results not demonstrated. In fact, results not demonstrated and ineffective are both included under a category called not performing. I'm not sure that's fair as a methodologies I'm not sure that's fair, but that's the way it is. So currently, in OMB's eyes, the CIL program has results not demonstrated and therefore is considered not performing. Well, why is that the case? If you look on Page 5 and 6, you'll see the particular scores that OMB gave to the CIL program. Now, this is a little bit of a shortened version -- I shortened some of the particular wording so it all fit, but this is certainly the essence of the different questions asked and you can go to the ExpectMORE.gov website if you want to and see the exact questions and you can look at this and see where the CIL program was scored well and scored not so well. You'll notice there are four parts. That's a pun I guess. There are four sections to the PART. Section 1, you did quite well. We did quite well, 100 percent. Section 2, strategic planning, this is on Page 5. Section 2, strategic planning, you can see we didn't do so well, only 12 percent. Section 3, program management, now looking on Page 6. You can see we did middle, 40 percent. And section 4, program results/accountability, not so good, 8 percent. Overall rating at the bottom there on Page 6, results not demonstrated, included under not performing. Well, why? Why did we get this score? The bottom line basically is captured a lot by questions 2.1 and 2.3, if you can find those. In 2.1 does the program have specific long term performance measures that focus on outcomes? No was the answer. And 2.3, are there annual performance measures that practice progress toward the long term goals? Again, no. Well, the way this is constructed, if you get a no on 2.1 and 2.3, you're in big trouble because a lot of others cascade from that one. So I think it's pretty clear reading pages 5 and 6 you can see that not having outcomes, not having measurable outcomes hurt the CIL program when OMB PARTed it the last time around. So let's put it in context though. Because it's not the only program -- CIL is not the only program in the Department of Education that received this score of results not demonstrated. In fact, not by any means. So if you turn to pages 7 and 8, and also 9 and 10, actually all four of those pages, here we've done -- we wanted to give you as much knowledge as we could so you could understand the CIL situation in the proper perspective. And here we've listed all 88 Department of Education programs with their PART scores for each section and also with their PART rating. So you'll notice actually the very bottom line on Page 7, which also carries over of course to be the bottom line on Page 8, that's us. That's the independent living for people with disabilities, there you can see our different scores, and there you can see our results not demonstrated. But notice all the other results not demonstrated also on Page 8 in that third column on Page 8. Look at all the results not demonstrated and then you can go to page 10 and look at all the results not demonstrated on that page, too. In fact, I calculated these up being the nerd analyst that I am. I calculated these up and of these 88 programs, 47 of them in the Department of Education were given a rating of results not demonstrated and that's 53 percent. Remember, earlier we said on Page 2 that across the entire government, 22 percent were results not demonstrated. Here in the Department of Education, you can see it's 53 percent of results not demonstrated. So in a way, the Department of Education hasn't quite been doing some things perhaps though it wants it to be doing and has its work cut out for it and I say that only to let you know that the CIL program is definitely not alone. In fact, a majority of the programs in the Department of Education are also results not demonstrated. That's some comfort, right? But not a lot because it's not good to fall into that not performing category as we mentioned earlier. And RSA knows this and to their credit they responding to this. So look if you would at Page 111. On Page 11 was just sent out this last December by RSA. It's some frequently asked questions about the revised 704 annual performance report, and let's, if we can, take a second and just look at this third and last paragraph on Page 11 because I think it really is why we're here today, isn't it, in a lot of ways. Let me read a bit of it. The revised 704 report incorporates newly established performance goals and measures for the independent living IL program. These measures aim to better reflect the IL impact on individuals and the community. The primary difference in the new 704 report is that increased focus on outcomes in comparison without puts -- well, there is our call right there -- the primary difference in the new report is its increased focus on outcomes in comparison without puts. It goes on to say what -- and then it says outcomes represents the impact that the IL services and activities have on the consumer. That is, the extent to which consumers achieved greater independence and community integration, outcomes are the means, where -- I'm sorry, my mistake -- outcomes are the ends whereas outputs are the means and here is a very important line also, both are important, yet ultimately the independent living program will be evaluated based on its outcomes. The independent living program will be evaluated based on its outcomes. So we are clearly, clearly in an outcome environment in the CIL program and we clearly need to be paying a lot of attention to outcomes as you know. Well, let's get specific for one more second. Look at Page 12 because on Page 11 RSA says that they are now putting in new measures of outcomes, and on Page 12, they list exactly what those new measures are. Now, I won't go over them in detail. You'll have plenty of time to read them. If you sigh the word indented where it says measure, at least with my accent it's called measure. It may be a different word in your accent, but in my accent, measure and if you read the italicized part, the first one is the percentage of CIL consumers who report having access to previously unavailable types of situations. The next measure is the percentage of CIL consumers who move out of institutions. The next measure, the percentage of CILs with CIL staff, board members and/or consumers creating or participating in various things in the community. And then the last measure, the number of months from due date to the release of certain data. So these four are the new four measures that RSA has included in the 704 report in order to respond to the part that we saw that says we don't have those kind of measures right now. Okay, now, I hope this little background has been interesting and useful to you. Why don't we stop here and see if we have any questions. OPERATOR: Thank you. If anyone does have a question at this time, please press 01 on your telephone key pad. Once again, if you do have a question at this time, you may press the 0 key followed by the 1 key on your telephone. The first question comes from Peter. CALLER: Thank you, and you may be getting to this, but one of the things you mentioned in the previous page talking about the OMB website, or the rating tool for PART, you talked about long term outcomes and also the next part was annual indicators. One of the difficulties I find -- I'm a database oh cord Nate tore -- when we look at some of the individuals that we provide assistance to and independent living skills training to, those outcomes actually are much better seen when you look at them over three or four years as opposed to one year. And I'm wondering if there is any way that when we look at outcomes -- because the 704 is an annual report from October 1st, you know, through September 30th and it's sometimes very difficult to show an outcome in that short a range in terms of some of the kinds of work that we're doing. And so I'd just like some feedback on that. Thank you. MIKE: Peter, that is an excellent question. I couldn't have planted it better myself. I hope to address that a little later in the call when we talk about different levels of outcomes. When we talk about logic models, we'll be talking about how some outcomes are achieved right away or hopefully right way I, other outcomes need to be achieved later or further down the road. Stop me at that point we haven't covered it, but I think what we might be able to say then is, yes, Peter is exactly right. A number of our outcomes won't be achieved for several years, but there are others that perhaps we can watch on an annual basis, that would be leading toward -- the leading indicators, if you will, leading towards those other outcomes. If we don't cover that correctly, Peter, at the time, please ask again. CALLER: Thank you. OPERATOR: We have four our audio questions and the next question comes from Brenda Little. CALLER: Before you said when we talked about outcome measurements, it says long term goals. Are we reaching our long term goals? What is long term? MIKE: Well, that's a good question, too, and that depends on who you ask. Many people will say five years. If you could go to the OMB website and see detail about this. Many people will say five years, what are you trying to achieve then, and then annual within of course one year. But there is variation by program and since what we'll be eventually producing is something for the CIL program as a whole, that's the kind of decision we'll be having to make. We'll be talking more on Wednesday about that, but that's the kind of decision we'll be having to make. CALLER: Thank you. OPERATOR: Thank you. The next question comes from John Headinger. CALLER: Yes, actually the question had already been answered, but I was hoping to get a little more clarification on the tying in of this information to The 704 Reports for centers for independent living. MIKE: We might be able to ask if Tim or Bob Michaels has something to say about that. They would certainly be more knowledgeable about the specifics, although my guess is what they might say is that that might be a more appropriate detailed discussion for another call. I'm not sure. Let's see what they have to say. BOB: This is Bob Michaels. Can you hear me? CALLER: Yes. BOB: Our intention is that the primary reason we're doing this outcome measures work is from NCIL's perspective is because people from centers have been asking for it. They are saying -- they have been saying we want a better way to measure what it is that we're doing. Secondary to that is getting rid of the 704 report that hasn't worked very well. And what we've been doing is throughout this process is talking to RSA about this, and then trying to get some commitment from them that when we do develop these measures, that they will replace the measures that are in there now and replace the information you are already being asked. Not supplementing it, but supplanting it and then trying to replace it. Now, whether or not we're successful in doing that will remain to be seen, but our primary focus is on getting you some documents that you can use that would better tell your story. Secondary to that is the 704 report. CALLER: Okay. OPERATOR: Thank you. Before we take another audio question, are there any E- mail questions? TIM: This is Tim. Thanks, Dawn, there are no E-mail questions at this time. I do want to remind our webcast participants you are able to ask live questions. Simply E-mail webcast@ilru.org. And I'll ask them live on the call. OPERATOR: Thank you. And you have three other audio questions. The theeks question comes from Charlotte Stewart. CALLER: I have two questions, my first one is how long is OMB giving the Department of Education to correct the programs that have results not demonstrated ratings? MIKE: Well, it survey ease a little bit, but typically they are reparting programs after about five years and since ours was PARTed in 2003, that means that next year it will be PARTed again. That's a little misleading because after each part OMB works with the program to develop what they call an improvement plan. So RSA has actually developed a plan and these measures are part of it, and OMB is monitoring that on a regular basis, but the actual PART should happen next year. CALLER: Thank you. My other question is on Page 12, when you were just reviewing those indicators, objectives and indicators, why are we being asked about increasing the number of people moving out of institutions when that currently is not a service that centers are required to provide? MIKE: Well, let's ask Bob Michaels to see if he can address that. BOB: Well, I think that's an excellent question, Charlotte. I think you're exactly right, but that is an area -- and it's been in the 704 report for some time and I think that one of the reasons they put it in there is because it's something that's very easy for them to identify and to measure and to say this is an outcome that we want to accomplish. We haven't argued about it too much because nationally we're trying to get people out of institutions, but you're exactly right. It's not a requirement. CALLER: Thank you. MIKE: Operator, let me mention something to the operator if I can. I think we were going to have five minutes worth of questions and answer after each section. So I'm not sure if we've reached five minutes, but I think we may have. OPERATOR: at this time we'll conclude the question and answer session. We'll go on with the presentation. MIKE: Thanks very much. There will be plenty of time for questions later. I want to get through all the material. Let's go to the next section, and stop me if we didn't succeed, but that hopefully gave you an overview perspective of why this matters to us, both the internal forces coming from CILs and the external coming from the OMB PART process. Now, let's ask ourselves how has NCIL responded to this? And if you look on Page 13, 14 and 15, you'll see a letter that Kelly Buckland, the president of NCIL, has out. Now, this resulted from a task force and forgive me if you all know this, but I will assume not. And give me 60 seconds to give background. About a year ago, in fact, April of last year, Kelly Buckland and other people formed a task force on outcomes. You can see the composition of the task force, by the way, on Page -- well, it's coming later on. But there are about 12 people on this task force, perhaps some of you are on this task force, and from the local level, people who are very, very knowledgeable about the local level reality of CILs. This task force and I have the honor of working with them, and we've been working for about a year now to tackle exactly this issue we're talking about. One of the first things the task force did and I think it was very wise, was decided that they needed to put a marker in the sand or they needed to make pint and they needed to have it in writing and they needed people to receive it. And this is the that serves to do that. This is a letter that Kelly send out. You can see that last September. It went to the head of RSA and I think higher up this education, if I'm not mistaken, it may have gone to the Hill and OMB. I'm not sure of all the people. First of all, it said that we are happily collaborating with RSA and OMB, which the task force is. They are working very collaboratively with RSA and OMB and there are good terms. This is an amicable type of collaboration, at least as I can see, but -- but the NCIL wanted to make it very clear that they have some concerns about the way RSA responded to the PART. Remember, they responded with those four measures we talked about, plus some other changes. And the task force wanted to make clear that they weren't entirely in agreement with the way RSA responded. So you can see for yourself in this letter, that they talked about some of the specific problems that NCIL has with the 704 measures. And it also talked about some of the specific concerns that NCIL has with how this data might eventually be used, in particular, whether one CIL is going to be compared against another one and things like that. so this was a good letter, in my opinion, that went out on your behalf. And it received a response from RSA that you see on Page 16, 17 and 18. And the response basically came back from Ed Anthony to John Lancaster, and you can again read it at your convenience, but it also said we're very happy to collaborate, too. We like working together on this. We don't agree, we RSA, don't agree with everything you, NCIL, said in your letter, but that's fair. But we are open to ideas. And that is the next to last paragraph. If you see on Page 18, again, humor me, if you will, while I look at this and read it. In on collusion, let me take this opportunity to reaffirm RSA's willingness to consider improvements to the new measures and the revised 704 report. Willingness to consider improvements, that's good. This has been perhaps RSA's consistent theme in engaging our partners and I have to say from our experience that is had a been true. They have consistently said we're very open to better ideas, and then it says RSA well comes the outcomes task force's participation and a waits its recommendation. So the ball -- the ball in a way now is in the task force's court to work collaboratively, but also hard, and to come up with some suggestions for making the 704 better. So Bob Michaels is the chair of the task force. On Page 19 and 20, this is Bob's most recent update dated December 21st, not too long ago at all and here on Page 20 you can see the composition of the work group there. And in this, Bob talks about the two factors we were just talking about, and he talks about what the task force has done to date and what it's planning to do and he also talks about how everyone in the CIL and the IL movement can help out. As we will talk more about on Wednesday, but mention now, the task force needs some help right now. It's at a critical point and it needs your ideas and it's very, very open to your ideas. So in addition to the update that you have on Page 19 and 20, there are some things on the website on the IL NET. And on Page 21, 22 and 23 is also a part of the website. It has a little background on measuring outcomes in the independent living movement, talks a little bit about many of the things we'll talk about today, a little bit about logic model, a little bit about desired outcome, measurable indicators and then where do we go from here. So that I hope has been useful to you in giving you a sense of how NCIL has responded to these two encouragements, if you will, both by creating the task force and then by doing various things with the task force. So, again, let's stop there and see if -- let's take five minutes if we can and see if there are any questions about NCIL's response. OPERATOR: Thank you. If anyone does have a question at this time, please press the 01 on your telephone key pad. Once again, you may press 01 on your telephone key pad at this time. We do not have any audio questions. Do we have any E-mail questions that have come in? TIM: Yes, thank you, Dawn. We have two questions by E-mail, however, one of them was answered in the last session about when OMB. [ INAUDIBLE ]. The second question is from Eric Reed from El Paso, Texas. Eric has a question about measuring CIL's activities in regards to or vis-a-vis the hurricanes that hit the gulf region in 2006. I don't know if this is a question for Mike, so Eric, I just want you to know I'll respond to you myself by E-mail and if I don't answer your question then, and feel free to chime back in and we'll forward you to Mike. There are no further questions by E-mail. Thank you, Dawn. OPERATOR: Thank you. The next audio question comes from Dana Jackson. CALLER: I'm sorry, mine is sort of like a response to what we were talking about before, and as far as measuring the outcomes forgetting people out of institutions, now in Virginia we've been told that that is now our fifth core service. Instead of four core services, we are now five. MIKE: I don't know if Bob or Tim -- BOB: Mike, I should probably answer that. You know, it's our intention to have a fifth core service and we've been asking Congress to go ahead and include that in the next reauthorization of the Rehab Act. So that is in there and there will be a fifth core service added and right now they are working on the language and that, and we still don't have a bill. So it hasn't been done. Now, if Virginia has already decided to do that, that's between you and Virginia, but it's not a national thing yet. CALLER: Okay, I was just wanting to input that we're actually doing it as a fifth core service. Because I think of what it said on the letter. OPERATOR: Thank you. The next question comes from Brenda Little. CALLER: Yes, I was wondering if there is any way that. [ INAUDIBLE ] practices for business could be applied to the IL to better help document what we're doing to prove that we're doing it? Is. MIKE: Well, I'm glad you said that. The situation at the current moment is that the task force -- and again we'll talk more detail on Wednesday -- but the situation at the current moment is that the task force has developed what we would say would be a logic model. We'll talk more about that in a bit. A logic model for the program and the task force is now wrestling with the really, really important issue of what specific data items are we going to measure to capture our outcomes? And we're very -- we're at the moment of casting the net very widely. Net has been cast. We want to cast it even wider. So what you're suggesting I think is that's an area we should look at as possibilities that might have relevance for our work, is that correct? CALLER: That's correct. MIKE: That's exactly -- we need suggestions exactly like that. So thank you. And let me say if others -- if others have suggestions, again, we'll talk more on Wednesday, but if others have suggestions exactly like that, things we need to look at, please, we need those. OPERATOR: Thank you. We have no more phone questions at this time. Do we have any web questions? TIM: We do not have any webcast questions at this time. Thank you, Dawn. OPERATOR: Thank you. That will concludes this portion of the questions and we'll go on with the presentation. MIKE: Good, so we've talked about why we care about this area and we've talked about what NCIL is doing about it. Now, let's if we can, let's get a little technical, if we will. Let's make sure we understand exactly what we're talking about when we talk about outcomes and outcome measurements and things like that. So if you would, I'd like us to start now with page 24 in our handout. And can you see it says why measure outcomes? I think this is really, really an important page. It probably doesn't look like it to you. It looks like just a place holder to transfer from one section to another, but it real is I a very important page because it lists in my opinion at least two reasons to be doing this and it lists them in a particular order. Increase effectiveness is listed first and it's not a coincidence. A communicate value is listed second. Now, why that order? Well, communicate value, what's that mean? The way I mean that is this is our P. R. function. This is where we show RSA and OMB and your local United Way and whoever else you might need to show all the good works you're doing that you in fact are doing good works. That's what I mean by the communicate value here. Communicate the value of your CIL, the wonder actual outcomes your CIL has been accomplishing. And I know that's important. I realize we all live in a real world. We need to do that. But I put it second and I put it second on purpose because I think if we focus on communicating value, we're going to be hurting ourselves and hurting our clients. In my opinion, our emphasis really needs to be on increasing effectiveness. (captions will resume momentarily). MIKE: Again, you can ignore that or pay attention to it. The principal decided that he would have -- begin a classroom competition, the third graders against the fourth graders and they are competing now to see who can walk more. He also prepared information for each classroom teacher to show them there is a very interesting relationship between how many steps the teachers are walking and how many steps are students are walking. I'm guessing the teacher didn't realize they were actually having that big an effect on the kids. So we'll see how that pans out. So they are doing something bit. Again, we don't know how that one is going to turn out yet, but I love what they are doing because they ask themselves what they care about. That's walking. They've measured it. And they are intervening to increase their effectiveness. So that's what I mean not Jess measuring outcomes to show something to RSA unde OMB. That for ourselves and our own purpose to make ourselves as effective as possible. Once we do that, in my opinion, once we do that everything else takes care of itself automatically because in fact we have a good story to tell. So forgive me for being on my soap box there, but I think it's important to ask ourselves why do we care about outcomes. I think it's for us. If you look on Page 25, who is focusing on outcomes, I don't think we need to spend any time on this at all. I think we all know this. Basically, everybody is focusing on outcomes. You can't turn around without hearing more and more about the outcome world we live in. And I wanted to include page 26 particularly because I think if I'm not mistaken that many CILs receive United Way money or at least many do. And I don't know if you realize this, but the United Way in the nonprofit world, the United Way is clearly the leaders in outcome measurement and they've been working in this for about ten years. I wanted you to know this good web page they have, so it's available on here, and you can see it, and there is a lot of good information on there. Now, one caveat, if I can, well, United Ways have a lot of local flexibility. Some are heavy into outcome measurement and some are resisting it and some are in between. I don't know how your local United Way is feeling about outcome measurement, but you may find you have some supportive, experienced, talented people there and you have this national website at United Way of America. Now, page 27 and 28, if you will, this captures what I think is the way the world used to be and the way the world is now. And page 27, if I would say in the nonprofit world is the way our world used to be. We used to have inputs, of course we still do, but we used to have inputs, resources dedicated to or consumed by the program. That would be things like money, staff, volunteers, whatever, and then with those inputs we did things with them, right? Activities, we still did. What the program does with inputs to fulfill its mission, that would be feeding and sheltering or providing job training or educating or counseling, and then in the past, basically we all know this, but out puts. The volume of work accomplished but the program and if I could say -- actually. [ INAUDIBLE ] I'm getting static on my end. [ INAUDIBLE ]. TIM: This is Tim at NCIL. Is there any way we can alleviate the problem? OPERATOR: Mr. Hendrix, are you on a speaker phone? MIKE: Me? I am. OPERATOR: Can you pick up the hand set, sir? MIKE: I can. Does that help? OPERATOR: a little bit. The static is still there, but I'm able to hear you now. MIKE: It's been fine before. OPERATOR: We are able to hear you all now. MIKE: So let's get back then to page 27. So outputs is what we used to care about right? The volume of work accomplished by the program and as I was mentioning, I think that's not a bad way at all if you can fix in your mind that that's a good way to think about outputs, the volume of work accomplished by the program. Or how hard the program is working, the number of classes taught, the number of counseling sessions conducted, number of services delivered, number of participants. That's what the 7804 report used to cover. That's what a lot of reports used to cover. That's what we used to care about. That is still necessary. All of that is still necessary on Page 27, but it's not sufficient anymore. Now we have to add page 28. Or what actually some people kind of irreverently call the so what question. So what you're working this hard, oh teaching that many classes or conducting this many counseling sessions, so what if you're serving this many participants. What are the outcomes? What are the benefits or changes for participants in program activities? And below this are some of the kinds of benefits or changes you might see, new knowledge, increased skills, changed attitudes or values, modified behavior, improved conditions, altered status. So page 28 -- 27 and 28 together, in my mind at least, represents the way the world is now for us. We live in an outcomes world. And on Page 29, just a little cheat sheet, if you will, to try to help you remember the differences between the four key concepts. And on Page 30, let's look at some specific examples of what do we mean by outcomes? Because it's this outcomes as you read this, is the outcomes that we're going to be focusing on from now into the future really. Let's look at examples. What if you were running a comprehensive child care program and a possible outcome could be children exhibit age appropriate mental and verbal skills. That's what you want to happen. I actually often call these desired outcomes because it's what you want to see happening. Or children are school ready for kindergarten. Well, we'll skip down a couple of lines here. What if you're running an outpatient treatment for adolescent substance abusers. Well, a possible outcome could be adolescents increase their knowledge about the effects of substance abuse and addiction. Another possible outcome would be adolescents change their attitude towards substance abuse. And another possible outcome, actually a quite different one, is graduates remain free of substance abuse six months after program completion. So you can see the outcomes are what we want to see the changes, the benefits for the people participating. Here is the last one, overnight camping for some boys. Boys learn outdoor survival skills. Boys feel more confident. Boys develop and maintain positive relationships with their peers. And by the way this is not a bad way to word outcome. You start off with who you are trying to change and follow it with a verb and then finish it off. So those are some examples of different outcomes for different programs. All right, one more then, let's look on Page 31. Now, the hard part sometimes for all of us, frankly, is to make sure that we keep the difference between outputs and outcomes clear in our minds. Now, remember outputs, there is a volume of work accomplished. That's how hard we're working. Doesn't say anything about whether we've changed anyone or not, it's just the volume of work we've been doing. The outcomes, that's the changes in people. So let's look at a couple of examples and if we can let's compare outputs to outcomes. Say that comprehensive child care program we talked about, what would capture the volume of work accomplished? Well, things like the number of children cared for. Yes, it would, wouldn't it? Or the number of child days cared. That's a good one, too. Doesn't say whether the children have been helped. So that we go over to the outcomes and here we saw before, children exhibit age appropriate physical, mental and verbal skills. There is an outcome. Children are school ready for kindergarten. There is an outcome. Let's go down to the last one there, what if wooer ee running a congregant meals for senior citizens. Well, the number of seniors served lunch, yes, makes sense, right, the number of seniors led in exercise programs. Okay. The number of tables of seniors organized for after lunch games. Okay, that makes sense, too. But notice none of those really say if the seniors have benefited or changed for the positive. For that we go to the outcome. Seniors interact socially with peers. There is a positive change. They interact socially with peers. They are not homebound. They eat a nutritious and survey ed diet. They exhibit fewer health problems. That sounds good to me. So you can see the difference here between the outputs and outcomes. All right, let's stop here and see if we have any questions. What I've tried to do -- let's see what we've tried to cover here. What do we mean by outcome measurement and why would we do it? What do we mean by outcomes and outputs? What's the difference between them? Let's stop here and see if we have any questions. OPERATOR: if anyone does have an audio question, you may press 01 on your telephone key pad at this time. We do have two questions. And the first one comes from Lou. CALLER: in looking at these outputs and outcomes, when you're saying you're keeping all your data on the number of seniors served lunch and the number of exercising and so on, but how are you measuring those outcomes? How are you knowing whether -- I mean, just because you had tabled organized for games, is that your outcome that you interacted socially with peers? MIKE: No, and that's a great question. What we're talking about now are what are the desired outcomes. In a little bit here we're going to point out that you don't measure outcomes directly. And you picked right up on it. You're very good on it. You said, well, seniors interact socially with peers. What do I do with that? That doesn't give me anything. I'm fond of saying that you cannot measure and outcome. I know that's strange to say since the whole topic of this call is outcome measurement, but in fact I sometimes say no one ever has measured and outcome. No one is measuring an outcome today. And no one ever will measure an outcome. And that's because we have to develop some indicators of those outcomes and that's what we measure. We then measure the indicators. So you're about 10 or 15 minutes ahead of us but you're right on target. We don't measure outcomes directly. We have to develop some indicators of those. And by the way, that's exactly the task that the task force is working on right now is developing those indicators of outcomes. CALLER: Thank you. OPERATOR: Thank you. The next audio question comes from Ms. [ INAUDIBLE ]. CALLER: Yes, I just had a question. Why aren't we utilizing some of those consumer goal met -- goals met and that were set by the consumers without comes and some of that data as well as some of those community goals met? Why didn't we focus in on that? Why weren't those looked at by the PART's as well as some indicators of what was going on in the community and some things to measure by in terms of how many consumer goals were met and how many community goals were met? And then why don't we utilize that information as the possible outcomes or the changes in people? And go by our very consumer independent living plans and what kind of goals that they are setting for themselves and what outcomes they want from that? What kind of goals to meet that they want? Couldn't that be used for the outcomes? MIKE: Well, I think the short answer to that is we might do that. This is a very open process. We're in the middle of this process right now. Again, as you'll see on Wednesday, we have some ideas for specific indicators for specific ways to measure outcomes, but they are just ideas at this point. We're soliciting all kinds of ideas and we may end up doing something like what you just said. It hasn't been decided yet. The only thing that has been decide sed that RSA has decided which four mess ice they are putting in the 704 report, but remember the whole task force assignment is to see if we can't come up with something we're more comfortable with. So at this point everything is possibly on the table. CALLER: and why wasn't it that the OMB process didn't look at the goals section of goals met and see that as indicators of whether a program was working or not? MIKE: I'm sorry, I have no idea. I wasn't there. CALLER: Okay. Thank you. OPERATOR: Thank you. The next question comes from Mike Brown. CALLER: Hi, this is Doris Ray at independent center of northern Virginia. I have two questions, one is when you're discussing desired outcomes, is there anyplace here for desired outcomes to be not just moving from one thing to another, but maintaining something, maintaining independence, for example, or maintaining one's self in the community? And the second question is I understand the need for coming up with outcomes and desired outcomes and such and what RSA is trying to do. The question is how are we dealing with that in terms of measuring the philosophical and basis of centers for independent living and how we're different from the RSA programs, the VR programs, I mean, are we going towards outcome 26 for all CIL consumers? MIKE: Doris, let me -- if you're willing to, let he ask if you're willing to defer the second question until Wednesday because that's the very specific in terms of IL situation and we will be having a lot of time to talk about that on Wednesday, if that's okay? CALLER: Okay. MIKE: Let me answer your first one though now, does every outcome has to be a change for someone? They are moving from one situation to another. Can we have an outcome of someone maintaining? And the importance to that I think is I think we can and should do whatever makes sense. And if what makes sense for you is to maintain someone in a certain situation, then by gosh, that's what it will be and that's what we'll measure. And let me give you an example. I'm working with a program that works with young pregnant -- teenage mothers and one of their very cleared desired outcomes is that they don't get pregnant again for two years. So they are trying to very much maintaining them as not pregnant. So it sounds like a very similar kind of thing, but again the most important thing to keep in mind there is whatever makes sense is what we need to do, yes. CALLER: and how are you measuring outcomes when it comes to the community activities that were referred to earlier? For example, right now I think they've been measuring well. So how many curb cuts have you gotten installed in your city or how many buildings are being made accessible? Something like that; but that doesn't -- what it sound like you're saying is we should be looking at so how many people benefit from that? MIKE: No, not necessarily. You remember one of the types of outcomes can be changed or improved condition or status. It can be the condition of a street as well. It can be the condition of a human being, but again, if we can, since that's specific to the IL situation, we'll have plenty of time to talk about that on Wednesday, that's okay. CALLER: Okay. OPERATOR: Thank you. The next question comes from Amy Mays. CALLER: Yes, my question was actually the first question. So it's all covered. OPERATOR: Before taking any more audio, do we have any more E-mail questions? TIM: Not at this time. OPERATOR: Our five minutes is up for questions at this time. We will take questions at a later time. MIKE: Okay, great. Now we've been talking about outcomes. We know what outcomes are. Let's ask ourselves what the heck is this nerdy thing called logic models that we've been talking about. Let me ask you if I can to flip back to page 27 and 28. This is where we had the program outcome model. You remember I said that these two pages together I think capture the world we live in now with the input, activities, outputs and then outcomes. It does, but it also doesn't. Really in a way page 28 is misleading, because if you Reid it, what it looks like is would take our input, do something with them in our activities, work at a nice hearted level, and then bingo, all our outcomes happen automatic all once. There is only one person on this figure. Everything happens all at once. So every change we want to see is going to occur simultaneously. We know that's not the case. That's just not the way the world works, is it? Generally you achieve something and that leads to something else and really the world is much more like page 32, don't you think? Inputs and then we can do activities and then we work hard that leads to outcomes, but it's more different levels of outcomes. There are? That happen initially. We mentioned this earlier on the call. There are some that happen initially and because those have happened, then that will lead us in intermediate outcomes and because that's happened that will lead to some longer term outcomes and by the way there is nothing magical about we have three levels of outcomes here. That's just because what fit on the page, the concept here is that there are different levels of outcomes, aren't there? And this page 32 is the conceptual framework for a logic model. It's a model that displays the logic of a program. Now, look on Page 33, if you will. A good way to think about this is if then -- in fact, an excellent way to think about a logic model is if then. If we have these activities, then this outcome will occur. And if we have this outcome, then working our way up, then this outcome will occur and if we achieve this outcome, then this one. Look at Page 34 to put more meat on those bones. If we have these activities, possibly, possibly then this change in knowledge occurs. Kids learn something, say, and then if they've learned this knowledge, then this change in behavior. It's led to something he do differently. Maybe they don't do drugs anymore. If they change this behavior, then this condition and this change in condition status, maybe they are actually healthier or get better grades. It's this connection that really displays what's going on with a program. Now, let's look at real examples, page 35 and 36. These are real examples, in fact, these are real programs, the one on Page 36 I was talking to you about earlier, actually. These are real programs and let's just look and see what they've done here just briefly. Down at the bottom you can see inputs and here is the various kinds of things the people who work, whatever. And then certain activities are done. You can see I'm looking on Page 35 now. There are classes on prenatal and then there is classes on infant care and then also individual work with counselors. Okay. Good. Now, how hard are we working, what's our volume of work accomplished? We're going to keep track of the number of teens served in the classes, the number of hours of instruction, not just the number of teenagers but also the number of instruction hours and the number of counseling. So these are our inputs activities. Now, let's see if we can't display our outcomes and how they relate to each other. Notice on the left side, what we're saying is that the prenatal classes, before the birth, first we think the pregnant teens will know about prenatal nutrition and health guidelines and if they know about that, working our way up, then we think they'll follow those guidelines. And we think if they follow those guidelines, then they'll deliver a healthy baby. Okay, we have a healthy baby here on the left side. Now, on the right side, we've also been giving them classes on parenting skills after the baby is born. So we think that if we do these classes, then teen mothers will know the infant nutrition, all that guidelines, and then if they know it, then they'll provide all that proper care to their babies and now as we're getting close to the top on the left we have them delivering a healthy baby and if they also provide good care to that baby, then the very top, the babies will achieve appropriate 12 month milestones for physical, motor, verbal and social development. So here is a case you can see we have four different levels of outcomes because that's just what made sense to us. No other reason. It's what made sense. The top one is 12 months later, and our friend asked earlier about long term for this program that might be viewed long term, but we don't have to wait 12 months to see if we're being effective because our initial outcomes are some initial knowledges that people and then some behaviors. So we have different levels of outcomes. This is a logic model. This is what many of us call a logic model. Page 36 shows another one. I won't take too much time on it, but you can see it's the same thing. Inputs and various activities, various classes. This is in fact just a variation on the one I told you about earlier. This is about alcoholic men and you can see that different kinds of outcomes were being achieved. You notice this part on the right-hand side, the graduate that identified work- related problems and none of that existed when the program first started. That whole right side didn't exist and nor did the activity on the follow up. So they added this when they realized they had something missing. You can see now that the plan is as you get near the top, if graduates obtain full-time employment appropriate for their skills, and if the graduates resolve work related problems, then graduates remain employed six months after program. So these are two examples of logic models. Now, you might be thinking -- I don't know what you might be thinking, but you might be thinking this was pretty nerdy to you, but if you would, trust me on this. This may seem strange to you now, but I just worked with too many programs that have developed these and they've come to realize about six months later how wonderful it is that they've done it. In fact, I sometimes think of it as duct tape. Everybody knows duct tape. You can hold a roll of duck tape and you have no idea what you're going to use it for, but you'll find a whole bunch of uses for it. So I think of logic model as a little bit of duct tape and on Page 37, here we have some of the ways that a logic model is useful by itself. No data whatsoever now. We haven't collected a single piece of data, okay, just the piece of paper, just the logic model by itself. It creates a sheer vision of the program, it makes the theory very exist plight. It helps you recruit and train staff, allocate resources properly, communicates the programs to other stakeholders, explain it to potential clients. This is what we're B. if you want to be about this, come work with us. Negotiate fair accountability, guide a systematic review of outcome data in order to identify improvement opportunity. That's what we're focusing on today and we'll be focusing more on Wednesday, that last reason for it. It guides a systematic review of outcome data in order to identify improvement opportunities. So let's stop here. What we've tried to do on Page 32, we talked about the concept of different levels of outcomes and then we talked about the if/then connection. We looked at a couple of examples of logic models and then how they are useful by themselves. Let's see if we have any questions hear. OPERATOR: the first question comes from Rachel. CALLER: Hi, can you hear me? Hello? MIKE: Yes, absolutely. CALLER: I'm sorry, this is the wrong number. Yes, this is Rachel and I had a question for you regarding -- can you hear me? MIKE: Yes. CALLER: I have a question for you regarding the different levels of outcomes. I was wondering if you had any specific examples of the different -- of an effective model of different levels of outcomes from the independent living movement itself instead of outside movements? MIKE: We absolutely do, and if fact that's a lot of what we'll be doing on Wednesday. CALLER: Could you give us a little preview today if you have time? MIKE: Why don't you turn to page 41. And 41 is a logic model that the task force has created, been working on a lot over the past six months I guess, and again we'll be focusing in detail on that once. CALLER: Thank you. This is Mike at the center that Ann is at. It seems to me that what we're doing here is still going to the same problem where we asked our question a little while ago in terms of the length of time. CIL programs for the most part do not show anything happening in the year they are first fund. Often in the year after they are first funded. Frequently they start showing them the third, fourth and fifth year later. Why are we planning a reporting process that will hand tie us and can only report what we're hoping to accomplish four years if now but we have to report it this year. MIKE: I'll let Bob Michaels answer more specifically. Nothing has been decided yet. I don't think anything is set in concrete yet. Let's see what Bob has to say. (captioning will resume momentarily) MIKE: I guess what I'm asking is -- it seems like if we're giving input to RSA, RSA isn't really listening for input about next year or the year after, they want it one year at a time. And the simple fact is we can't give that any more than, you know, the president can tell us how long it's going to be before we're done whatever we're doing. I don't understand how somebody can expect a one year input on a four year project. It won't happen. BOB: I really think -- I think you're exactly right. We've never thought that the -- CALLER: but it's never going to be reportable this way. Frankly, I agree that we need to be able to gather this information so we can see where we're going, for one, and number two, so we prove to the people that we're asking funding from that we actually deliver one hell of a job. And we can't report it because the format they wanted it in is one year at a time. BOB: You're right. You're right. And one of the problems we're always faced with, one of the things I said early on in this process is that we're doing this not -- primarily not because people with the 704 are telling us -- not because we're trying to change the 704, because members of NCIL have said to us for 20 years now, we need to find a better way to tell our story. And we need to -- we really need to show our outcomes and The 704 Report doesn't do that. And what we've been trying to do in this whole process is include them along the waned say, look, now, we would look to see these measures included in The 704 Report and replace the ones you have and I think you're exactly right. They'll have to take a look at these as long term. You can't look at what centers do and expect them to be responding in one year increments. You're exactly right. CALLER: Okay. MIKE: Let me put one caveat on that if I can. When I look -- again we'll talk about this much more on Wednesday, when I look at Page 41 and I look at the logic models of the CIL programs -- [ INAUDIBLE ]. MIKE: -- have skills, knowledge, choices, that barriers and problems in the community are identified, I'm not an expect in independent living nearly like all of you are, it seems to me that many of those will be accomplished and some of them maybe even before a year. So I'd just like to leave open this possibility that there are initial outcomes that I think. [ INAUDIBLE ]. Much more quickly than other outcomes. OPERATOR: Thank you. Are there any web questions at this time? TIM: There are not, thank you, Dawn. OPERATOR: We'll take one last question from Charlotte Stewart. CALLER: Mike, can you give us an example of going back to your example on Page 35, under the outcome section, how do you find out if those things had happened? Give us some examples. MIKE: I will, and you've led right into the next section, our last section for today in fact. CALLER: That's great. Thank you. MIKE: Let's look at Page 38 because there is one more thing we want to touch on today before we finish. Yes, we've talked about there being desired outcomes and now we've talked about let's think about how they all fit together and that's very, very important. Trust me on this, knowing which ones come first and it helps you think through the logic of your program and see gaps, and see redundancy. There is all kind of good advantages to creating a logic model, but what it doesn't do is get you any data yet and to get that data, we have to do what I mentioned earlier, is we have to decide which of the outcomes are important enough to spend the time measuring them. We have to pick out which outcomes the measure and then developed some indicators. And there is a difference between indicators and outcomes. And just like between outputs and outcomes, we need to really make sure we're clear on the difference between indicators and outcomes. So on Page 38, indicators are the specific -- emphasis on the word specific here -- the specific things we measure in order to show how fully the desired outcome is being achieved. It's different from the outcome. It's the specific thing we measure that reflects the outcome, if you will. Let's look at a couple of examples on Page 39, if you will. Now, here is outcomes and indicators. Remember we talked about our comprehensive child care, where the children are school ready. What does it mean to be school ready? Well, one possible way is that what it means is that they have passed a test, a particular test that the school has to determine whether they are school ready. So then we would want to do, we'd want to know the number and percent of full year children, now, that's interesting, isn't it? If someone walked in the door the day we gave the test and we hadn't seep them all year, that's probably not an appropriate indicator. We want to work with kids we've worked with all year long who have pasted the test to determine school readiness. The third one, outpatient treatment for adolescent substance abusers. Desired outcome -- adolescents increase their knowledge. They know more. Well, what's the indicator? How are we going to know if they know more? The indicator is -- the one possible one, and this is where Bob Michaels just mentioned, this is where the art really comes in. Number and percent of adolescents attending sessions at least three days a week because, again, its if someone comes only one day a week it's not a fair test. We're going to limit it to people that tending sessions at least three days a week. And they have to be able to describe five harmful effects of using drugs. Well, why five? Why not four? Why not six? No reason, it just seemed to us that five made sense. So what we'd be measuring here is we'd be taking all the kids who attended sessions at least three days a week, we'd be asking them, tell us five harmful effects of using drugs and if they can, then we say, yes, this person has increased their knowledge about the effects of substance abuse and addiction. Now, you might think that's a wonderful indicator. You mute think that's a terrible indicator. That's okay, but it's a possible indicator and that's the art of this. We cannot measure the outcomes directly, we have to come up with indicators. Now just a tease for Wednesday, for instance, one of the desired outcomes on the NCIL IL logic model is that an agenda for change exists. That's a desired outcome. An agenda for change exists. How will we know? How will we know if an agenda for change exists? What will be the indicator that tells us that an agenda for change exists? That's the task we're in right now. That's what Bob was talking about. That's the task we're in right now. In the interest of time I won't go over anymore on Page 39. And I won't go over page 40, but you can see what 40 does. It shows you key traits of a good indicator, the clever little way that someone thought up, SMART, specific, measurable, ambitious, realistic and timely. Let me stop and see if there are questions on anything at all now. OPERATOR: You may press 01 on your telephone if you do have a question. Thank you. The first question today comes from Ms. Mitchell. CALLER: Hi, I am a little bit lost and I'm going to tell you what I think I heard and you tell me if I'm okay. What I think I heard you say is that our activities are meant to do certain things, to impart certain things that go toward an over arching goal and that goal is really kind of the outcome that we're going for. Whereas, what the activities are tended to accomplish in a smaller way are the indicators, is that correct? MIKE: No, not really. The program activities are the things we do in order to achieve all of our outcomes. Some of those outcomes will happen first and some of them will happen later on. For instance, if you look at Page 35, you can see the different activities we have here of the we have classes for prenatal, classes on infant care, and then counsel or meets individual ry. So we do thee kinds of things with these girls. We run two kinds of classes and meet with them individually. Those are our three activities. That's all we do with them and as a result of those three activities, we want all of those outcomes at the top to be achieved. How many are there -- I see six different outcomes at the top. All of those outcomes should result from us having done these activities. CALLER: Okay. I'll spend some time with it and ponder on it and see if I can get it by Wednesday. Thanks. OPERATOR: Thank you. The next question comes from Donna. CALLER: Yes, this is Ken with Donna's group. You had mentioned earlier that you know most centers have United Way grants and I don't know if that's totally true. There are a lot of centers that don't have it and they've done that purposely, but I guess one of my questions is -- and as well as concern -- in trying to develop outcome measures, is it really -- is it important to you in the process to try to make sure that there is consistency of understanding of the process because even with the current 704 and the 704 before that, one of the problems that was really inherent from my perspective is that oftentimes when people reported information, people had different understanding of that information. So it was never apples and apples. It was apples and oranges because people didn't have the same understanding of terminology and so on and so forth. Even in this report, regardless of whether you like the content of the current report or not, I think there is a lot of confusion about the content and even if we develop a new measures for outcome, I think it's really important for us to have consistency of understanding and also some economy of the process. For example, one of the things we're doing right now, we're utilizing -- it seems to me we're utilizing the United Way logic model, but one of the things about that model is it is extremely work intensive and I guess one of my questions relative to that is what additional man power will be needed to do that volume of work if we don't think outside the box if we don't think of a logic outside of the United Way model. MIKE: Let me see if I can answer two of your questions. The second one is easier. No one, at least I am certainly not or Tim or Bob is going to, no one is asking you right now to create a logic model. What the task force has done on Page 41 is they've done that work you talked about. They've created a logic model for the CIL program on a national level. If you were over OMB and somebody said to you what's the CIL program about? They would hand you page 41 and say that's what the CIL program is about. Now, it may not apply to every single CIL. In fact, it may not apply to anyone CIL, but look across all 600, that's the CIL program. So you don't need to get nervous quite yet because we're not going to be asking you to do a logic model. I don't agree more that the instructions, whatever we end up with, whatever outcomes we decide are important enough to measure, whatever indicators we decide to use, those must be so clear, so specific, if you look -- in fact, on Page 40, you notice where it says the key traits of a good indicator? A good indicator is SMART -- the S. stands for specific. The indicator is clear enough that different people would measure the same thing in the same way. I couldn't agree with you more. You are exactly right. So what we know we're going to be doing already in the task force is we're going to be very clear on the indicators exactly what we mean. We're going to be very clear on the procedures and we will pilot test them in various CILs to make sure that they are in fact clear. Good point. OPERATOR: Thank you. The next question comes from Rachel. CALLER: All right, I was wondering how can the individual centers for independent living become involved in the national movement here that you guys have been referring to today to help make these standards and also how can individual CILs have a voice at the nation level like at Congress? MIKE: I'll let other people answer the second part, but let me say the first part is what we'll be talking about on Wednesday. We'll do page 41 and on, and there aren't too many pages past that. We will be talking about how can you help the process? We very much need the local CILs to help. Does someone want to talk about the second one on you they can have a voice at the national level? BOB: I think what's going to happen, Mike, is once we get these -- once the membership tells us what indicators they want to have for these outcomes, then we're going to be going to the people in Washington and we're going to say, you know, even though what we've been saying all along here is this is string primarily for centers, we're going to continue to see this in the 704. We don't want to be doing -- we don't want to continue to do a report that doesn't make any sense. So some point we're going to come back to membership and say, oh say, we're going to go to RSA and say we want you to put these into the 704 report and get rid of all the junk stuff and we'll be looking to members to back us up on that and it will become important for all of our members then to step forward and say these are outcome measures and these are the things we think you should be evaluating us on and I think if we do that we'll get these through. CALLER: So, in other words, we can -- so things like writing the Congress people or calling our Congress people that are representing our specific state and regions would be an appropriate way of getting involved or would you prefer for us to just go through the national CIL office? BOB: No, I think calling your individual congressman is always more effective, but again, it's something that's done -- with regard to this stuff, it's down the road a bit. We may not have to go that far. We may go right to RSA and say we would really like to see these measures put in place and it may not be necessary to go to Congress. If enough members come forward and say this is what we want, there is a. [ INAUDIBLE ] and we're going to get it. CALLER: Okay, thank you. BOB: One of the really important things here is that you be -- well, whether you're going to be on Wednesday or not, that you go up on the IL NET website and provide us input as we're developing these measures, what you're suggesting and wanting to do is exactly what we want. We don't want you to feel that these are our measures. These are your measures. CALLER: Thank you very much. OPERATOR: Thank you. The next question comes from Mike Brown. CALLER: Hi, this is Doris Ray. I wanted to ask to try to clarify what I'm hearing. That I think what you're saying is that the output measures are -- you set goals for things that you're going to do so in so many workshops or programs and so and so many units of peer counseling or whatever. And the output measures indicate whether you've done what you promised, but the other half of it is so you did what you promised, but how did people or communities or systems benefit from it? And that's the part that RSA is -- Rand OMB is asking from us? Is that right? MIKE: Well, you're certainly right that the outcomes, the part that happens because of the volume of work in the outputs, the outcomes, the changes in people, you're certainly right that that's what people are caring more and more about all the time. As I mentioned earlier, page 27 and 28, 27 is the world we used to live in where we stopped at out puts. That just doesn't cut it anymore. Now outcomes are an important part of what we have to care about and not just OMB and RSA, but as you heard me say for ourselves, outcomes for ourselves. So, yes, you're exactly right. Now I guess I would take a different point of view. You mentioned that for the outputs we ought to lift a certain number of classes we're going to teach, a certain number of people we ought to work with. Let me say when you say things like that, you're talking now about setting targets. You're setting targets. You're saying I don't. [ INAUDIBLE ] and I haven't done well enough. You can set targets on output. [ INAUDIBLE ]. CALLER: I can't hear you. MIKE: I'm hearing the static also. Let's try this. You can set targets on outputs. You can also set targets on outcomes, but I would strongly suggest let's not even think about targets at this point. Let's not set standards that we have to achieve in order to consider ourselves successor failure. Let's just decide what would be important to know. What would be important to know about ourselves and that's real what I we're wrestling with right now is what would reflect good performance in a CIL. Does that help at all. CALLER: We didn't hear everything you said. MIKE: I'm sorry. Maybe we can take that up again on Wednesday. How is that? CALLER: Sure. OPERATOR: Thank you. The time is up at this time. Do you wish to continue with all the rest of the questions or do you have more presentation. We have we have no more presentation. TIM: I'm afraid we're going to have to stop there. What I can offer with Mike's blessing is that any questions that are still remaining can be forwarded to me at Tim@ncil.org. And Mike and I will work to have those questions answered by Wednesday and we'll either respond by E-mail or we'll answer them on the call when we review today's content. Mike, does that sound fair? MIKE: That sounds great and let me before we have to hang up let me thank everybody. It's been really great and I hope you've gotten something out of the call today. TIM: Wonderful. Thanks, Mike. The other thing I'd like to say is I apologize for the audio problems and I hope we're not causing them. We'll work with our teleconference provider today and tomorrow to troubleshoot them so that they don't happen on Wednesday's call as well. And with that, I want to thank everyone for joining the call. I especially want to thank Mike and Bob for their time. And all of our participants for your participation and wonderful questions. If you could, again, I'd remind you to visit NCIL's website. I'll remind everyone if you're attending both calls, both today and Wednesday's if you could please just complete one evaluation form for both calls and there are instructions on the form for returning them to the NCIL office. They are extremely valuable to us and we'd appreciate your time. My thanks again to Mike, to Bob, and all of you for your time and we'll talk to you again at 3:00 p.m. eastern this coming Wednesday. Thanks, everyone. OPERATOR: Thank you, that does conclude today's conference.