ROUGH DRAFT 9-13-11, Outcome Measures for Centers for 
Independent Living – An IL NET Resource Presented by ILRU
 >>  (Applause)>> MIKE HENDRICKS: We 
do have to wait a moment while they 
do something fancy.  I don't know 
what it is.
We have to wait a minute.
They will give us a high sign. 
 >> 1 minute?  Anybody know a joke?
We're off-line.
Your boss can't hear you.
I'll say this while we're waiting.
Thanks for the introductions, 
Richard.
Equally nice things could have been 
said about you if you weren't too 
modest to say them.
It's nice to be here.  I have had a 
chance to work with a lot of you 
before.
I see a lot of familiar faces.
In person and also maybe on various 
webcasts.
I want to welcome you to Oregon.
Anybody else here live in Oregon?
Yes, where? 
 >> Eugene. 
 >>  Eugene, I'm in Hood River.
Yeah.  In addition to welcoming you 
to Portland, welcome to Oregon.  It's 
a wonderful state, a friendly people 
who aren't pretentious, get outdoors 
a lot, work hard, try to be nice to 
each other.
Gosh, it's beautiful.
If you have a chance to stick around, 
go to our ocean or mountains or 
streams or deserts, it's a wonderful 
place.
You're lucky.
The people on line only get to think 
about it.
You're actually here.
If you can take a day or two and take 
a tour, it's awfully nice. 
 >>  Starting the feed now. 
 >> This will be like TV.
We'll pretend none of that just 
happened.
I'll even stand over here. 
 >> You’re rolling. 
 >> Okay, thank you Richard for that 
very nice introduction.
That was really very nice.
Hello everybody on line.
We have been joking about you while 
you were not here and we're saying 
it's wonderful to be in Oregon and 
you should come 1 day.
So please do.
Yes, I'm Mike Hendricks, the part of 
the team who knows one hundredth of 
what Bob Mike else knows.  I have 
been working with outcomes for a 
while.
It's been a treat working with people 
over the past few years.  I think 
this is a wonderful session because 
we're at an interesting point.
Up until like say a year or so ago we 
were like just saying this is what 
would be nice to do.
You know, theoretically nice to do.  
This is what RSA would like us to do 
and things like that.
Now we have actually been out there 
and tried this stuff out.
We have had 2 years of field testing.
You're going to hear about that.
That is what for me so different 
about this training, that you're 
going to hear about what worked and 
what didn't work in the field.
We found some stuff that didn't work.
Our ideas just didn't quite pan out.
So now we'll try to find something 
better.
It's going to be a combination of 
general training, if you will.
Let's all make sure we're on the same 
wavelength.
Also real world feedback about what 
we learned.
Let's jump right into it if we can.
It's called outcome measures for CILs 
and that is exactly what it's going 
to be about.  Our first session is 
outcomes and outcomes management.
Let's talk about our objectives.  
Talk about what we're trying to 
achieve.  An outcomes focus.
First of all we're going to learn 
basic outcomes and practices of 
outcomes and improving outcomes.
We all have to know that.
We know how important this field is.
You wouldn't be here if this weren't 
an important topic.
Then we're going to learn in detail 
how the NCIL task force on outcome 
measures did its work including its 
two field tests.
Bobbie the way was the chair of that 
-- Bob, by the way, was the chair of 
that task force.  He ran that and 
still does.
We're in the just going to tell you 
what we learned.
You're going to get all the materials 
we developed so far.
You can use them back at home or in 
the use them back at home.
That is up to you but you are going 
to get them.
You will receive and understand all 
the operational materials from the 
more recent NCIL field test.
We have done this twice.  Second year 
was better because we learned.
You will get the stuff from the 
second year.
Also though Bob and I are big 
believers you have to practice 
something to learn it.
We're going to have you practice 
making the kind of outcome decisions 
that you're going to need to make 
once you get back home.
We hope you will make those decisions 
back home.  Okay?
We're going to try to answer 
questions anyone might have.
By the way, don't feel like you have 
to hold your questions.
Sometimes people say, you know, don't 
say anything for 45 minutes and then 
put your hand up.
We're not that way.
If you have a question, just stick 
your hand up or whatever.
If we don't have time to deal with it 
right then, we will be polite and say 
hold this just a little bit.
Probably we will be able to deal with 
it right then so don't hold your 
questions.
Carol, same for you.
If you see something you think we 
ought to dress for our good friends 
on line, just let us know right then.
Finally, let's have some fun.
Gosh, we're going to be together two 
and a half days.
Let's not be dour all the time.
I'm one of those people; I believe we 
can do serious business in a light 
hearted manner.
I don't know see why not.
I love terrible puns.
Feel free.
Let's have some fun together.
Now, this is Carol Eubanks' creation.
I give her great credit for it.
Nine steps we will try to cover.
By the way, thank you.
I see someone turning the handouts.
Yes, you have in your packet every 
single one of these PowerPoints.
Feel free to follow along, make 
notes, whatever you like.
Carol very cleverly said let's not 
just make boring nine steps.
So we have the yellow brick road.
We're going to follow this yellow 
brick road.
What is the doing's name? 
 >> Toto. 
 >>  Right, exactly.  Lots of nice 
dogs around the room by the way, too.
I love dogs.
We're going to follow the yellow 
brick road.
We're now at step one.
Outcomes and outcomes management.
The reason this is important is 
because if we're not all on the same 
wavelength of what exactly we're even 
talking about when we say outcomes, 
then we need to be.
We all need to have the same thing in 
our head of what we mean by an 
outcome.
So let's try.
You may have seen this before.  
Pretty common way of thinking about a 
program.  Essential components.
You have the inputs.
Those would be the resources 
dedicated or consumed by the program.
Things like money, staff and staff 
time, volunteers and volunteer team, 
facilities, equipment.
I like to say if you are thinking 
about baking, all the stuff out on 
the counter, everything that you will 
be using.
Those are the inputs.
We do something with those inputs, 
right?
Those are the activities.
What the program does with the inputs 
to fulfill its mission.
Things like feeding and sheltering 
homeless families, providing job 
training, educating teachers about 
signs of child abuse, counseling 
pregnant women.
These are the things we do.
Now here is where it gets interesting 
or tricky depending on point of view.
Now, the next one is the outputs.
Outputs.
Here is a really really good way, in 
my opinion, my experience at least, 
to think about outputs.
It's the volume of work accomplished 
by the program.
Boy, if you can stick that in your 
head, outputs is essentially how hard 
we're working, okay, the volume of 
work accomplished by a program.
Notice it doesn't say anything yet 
about what you have done, what you 
have achieved, what you have changed.
It just says the volume of work 
accomplished by the program.
The number of classes taught, the 
number of counseling sessions 
conducted, the number of educational 
materials distributed, the number of 
hours of services delivered, the 
number of participants served.
Let's take a more personal.
The number of INR calls you answer.
What else.
The number of persons you do 
something with in your center.
That is the volume of work 
accomplished.
Okay.
This is the way the world used to be.  
I like to say the way the world used 
to be for nonprofits.
This was perfectly sufficient.
This is what people cared about, what 
people asked us about, what they 
wanted us to keep track of.
Tell us about your inputs, about your 
activities, about your outputs.
We all know this don't cut it 
anymore.  Right?  This is not good 
enough anymore.
You can deny that all you want.
You can ignore that all you want.
You can pretend it's not true, but 
you're wrong because simply this 
doesn't cut it anymore.
We're in a new world where outcomes 
matter a lot.
Here is that new world.
Yes, inputs matter.
Yes, activities matter.
Yes, outputs matter.
But also, not anymore, but also 
mattering is the outcomes.
Benefits or changes for participants 
during or after program activities.
I like to say these are the seven 
magic words.
In fact, if I were you, on your 
handout, I would underline these 
seven magic words.
It can be new knowledge, increased 
skills, changed attitudes or values, 
modified behavior, improved 
condition, altered status.
If you have changed one of those 
seven things, then you have affected 
an outcome for someone.
This is the world we live in now.  
This is the world RSA is in we live 
in, and rightly show.
They should be insisting we live in 
this world.
You see this all the time.  You hear 
them talk about results.
One thing, my results, outcomes.
I want to point out down here at the 
bottom, I want to give credit, you 
see this united way symbol down here.
We have used, I have been fortunate 
enough to work for about the last 
10 years or so with the united way 
system and they have been kind enough 
to let us use a fair number of their 
slides.
I want to give credit to them, make 
sure we all recognize their 
contribution to this training.
A big thanks to United Way worldwide 
for letting us use some of their 
excellent slides.
It's not always marked.  Each slide 
you let us use isn't always mark 
because of space constraints.
I want you to know they have been 
very valuable and helpful to us.
Let's go over these one more time.
Then it's your time.
I'm going to give you a quiz.
Remember inputs.
It's often expressed as a noun.
You know the name of a person, place, 
thing or concept like staff, 
volunteers, things we talked about.
Activity, the way I like to think 
about activity, and I think it's 
right, it's a gerund, right, an ing 
word?  Like feeding, training, 
counseling.
It's often expressed as an ing word.  
That is the activity.
Output, remember, again, if you can 
stick it in your head, output, the 
volume of work accomplished by the 
program.
Not what the program changed in the 
world.
Just the volume of work accomplished.
Number of classes taught.
Hey, maybe people slept through.
Number of materials handed out.
Hey, maybe people threw them away.
Number of counseling sessions given.
Maybe nobody paid attention.
It's the volume of work accomplished.
It's the outcome that we care about 
here.
Describe a change in persons you 
worked with.
Let's just take one more slide on the 
difference between outputs and 
outcomes.
That is the one I see people missing 
up a lot.
That is the tricky one.
If you can get that down you're in 
great shape.
What is the difference between an 
output and an outcome?
Here is some examples.
Comprehensive childcare, an output.
Number of child days of care given,
 Outcome.
Children are school ready for 
kindergarten.
GED preparation.  Number of tutoring 
sessions.  The volume of work 
accomplished.
Outcome, participants obtained their 
GED certificates.
Independent living services.  How 
about this.  Number of consumers 
served.
Consumers have skills to support 
choices.
More examples of outcomes.
Clients at high risk for becoming 
parents demonstrate healthy socially 
appropriate attitudes towards teenage 
parent hood.
I won't go down all these.
Let's take the last one.
CIL consumers obtain independent 
housing.  That is a change in status, 
an outcome.
Anybody not got it?  Good, it's your 
turn.
I'm going to give you a pop quiz.
Your job is to tell me the outcome.
Spot the outcomes, if you can.
Parenting education programs.
Spot the outcomes.  Going to make it 
harder are you.
Tell me which the seven magic words 
it is.
Okay.
Spot the outcomes.
This is a parenting education 
program.
I need a hand so I can, and I'll 
repeat.
We don't have to wear a microphone.
I'll repeat it again.
Anybody spot an outcome.
You have a hand up. 
 >> Parents understanding. 
 >> Say again? 
 >> Parents’ understanding of 
children's developmental issues 
increases. 
 >> Understanding of children's 
developmental issues increases.
Why do you think that is an outcome? 
 >> It shows change. 
 >> In what, which seven magic words 
is this. 
 >> In understanding. 
 >> Or knowledge? 
 >> Yes. 
 >> I agree with you.
Any other outcomes up here?
No other outcomes? 
 >> I didn't hear people mention, but 
parents provide more age appropriate 
guidance for children is behavior 
change. 
 >> You think that is an outcome.  
Which magic word? 
 >> Behavior. 
 >> You have changed the behavior of 
the parent.  Absolutely.  Ding ding 
ding.  Very good.
Any other outcomes up here anybody 
think? 
 >> Yes, the last one.
>> MIKE HENDRICKS: Someone says the 
last one.
Parents participate in role place and 
group discussions.
What you do think?  I see someone 
shaking their head no.
Why do you say no?
We have a disagreement.
I love it already. 
 >> It’s part of the service being 
provided.
>> MIKE HENDRICKS: Someone says it's 
part of the service being provided 
which makes it what? 
 >> An activity. 
 >> Activity, something going on 
inside the program.
This is a good learning moment.  
Let's talk a second and think about 
this.
It sound like a behavior, doesn't it?  
They are participating, doing 
something.
Here is the question you want to ask 
yourself.
Is it something they are doing inside 
the program as part of the program?  
If so it's an activity.
Or is it something they are taking 
away from the program and doing 
outside.
Then it's a behavior change, as you 
pointed out, and it becomes an 
outcome.
This one I would say, if you don't 
mind, I think we agree this one is 
probably something they do inside the 
program.
So it's an activity.
Pretty straightforward.
Yeah.
Another quiz.
Tutoring program.
Any outcomes, we're spotting the 
outcomes.
Any outcomes here? 
 >> Number two. 
 >> Someone says number two.  Why 
number two? 
 >> Knowledge increases, performance 
increases. 
 >> Knowledge or skill.  The line 
gets fuzzy in my mind at least.
Yes, something definitely gets better 
there doesn't it for the youngsters.
I'll agree with that.
Ding ding.
Other outcomes. 
 >> Third one. 
 >> Someone says the third one.  
Interesting.  Youngsters indicate 
increased belief in their abilities 
to learn new subjects.
Which of the seven magic words is 
that? 
 >> Knowledge. 
 >> Skills? 
 >> Attitude. 
 >> Attitude.  I think its attitude.  
They indicate an increased belief in 
their abilities.
That is an attitude change, right?
Attitude changes are fine outcomes to 
achieve.
No problem.
A good one.
Anything else?
I don't see anything else.
Okay, ho about this one?
After school program.
Any outcomes here for this after 
school program.
We have a hand here, please.
Yes.
Hi Pat.
Uh-huh. 
 >> First one, children mastering 
individual and group activities. 
 >> Why do you think the first one is 
an outcome of the program? 
 >> The change in behavior.  They 
have mastered.  Yes, they mastered. 
 >> For me, again, these lines get 
fuzzy in my mind.
I don't know if this is a new skill 
or a new behavior, but it's somewhere 
in there, isn't it.
The kids are definitely doing 
something different than they did 
before, so I'm willing to say yes, 
agree to that one.
Other outcomes for here?
Here we have one. 
 >> Social skills have improved. 
 >> She says the children's social 
skills improve.
Which of the seven magic words is 
that? 
 >> Skills or training. 
 >> Skills, bingo.  Good for you.
How about anymore or no more? 
 >>  Last one. 
 >>  Someone says the last one.  
Children make more positive use of 
free time outside the program.
What kind of a change in outcome is 
that?. 
 >>  Behavior. 
 >>  Exactly.  Agree completely.
You're getting this clearly.
One more.
Conflict management program.
Any outcomes you see up here.
Yes. 
 >>  First one. 
 >>  Someone says the first one.
Youth are involved in fewer physical 
conflicts.
What makes that an outcome?. 
 >>  Behavioral change. 
 >>  Behavioral change, thank you.
Excellent.
Very good.
Others?. 
 >>  Third one. 
 >>  Third one, youth display greater 
tolerance of differing points of 
view.
Why is that an outcome?. 
 >>  Attitude. 
 >>  Attitude, I would say.
Yes, I would agree.
Any others?
Someone said the fourth one.
Did I hear the fourth one?
Youth practice communication and 
negotiation skills.
What do you think?. 
 >>  Depends whether they had any 
before. 
 >>  Behavior. 
 >>  Someone says depends on whether 
they had any before. 
 >>  Or if it's in the program or 
outside the program. 
 >>  Gosh, thank you my darling.
I love you. 
 >>  Make sure to use the microphones 
please. 
 >>  She says it depends on whether 
it's in the program or outside the 
program.
I'm so happy because we were just 
talking about that difference and how 
important it is.
What would you guess on this one?  
Youth program communication and 
negotiation skills?. 
 >>  In the program. 
 >>  Sorry, you would guess?. 
 >>  I would guess in the program as 
part of the activities. 
 >>  And what makes you think that?  
Just reading it?. 
 >>  Doesn't say some of the other 
things where they report or things 
like that.  I think it's part of the 
activity.
Depending upon how the program is set 
up, it could be either one. 
 >>  For me there's a key word in the 
line that makes me think it's inside 
the program. 
 >>  Practice. 
 >>  For me that is the word that 
tells me it's maybe inside the 
program.
Any others?. 
 >>  Last one. 
 >>  Somebody said the last one.  Who 
said the last one?
Tell me why they think the last one 
might be.
Grab a mic. 
 >>  Last one. 
 >>  Tim wants us to use the mike if 
we can.  Thanks, appreciate it. 
 >>  Last one seems to be the 
measurement. 
 >>  Measurement. 
 >>  A measurement of the outcome. 
 >>  Okay, what to you mean by a 
measurement of the outcome?. 
 >>  It's a tool for measuring.
If I were going to find out, if I 
were going to measure my outcome, I 
would perhaps field survey, and 
survey might include questioning 
which the youth would report back how 
it has changed their behavior. 
 >>  That is an interesting point.
Glad you brought it up.
We're going to be talking about 
measuring outcomes a lot, the 
difference between outcomes and 
indicators of outcomes and also 
method.
We'll talk about all that and sort 
that all out.
Don't worry.
You're raising a good point.
I'm going to say that there's 
elements of that in here, but I would 
probably accept this as an outcome.
But if it is an outcome, which of the 
seven magic words makes it one?. 
 >>  Attitude. 
 >>  Attitudes?. 
 >>  I would say attitudes. 
 >>  Or maybe even. 
 >>  Behavior?. 
 >>.
>> MIKE HENDRICKS: Not yet.  Doesn't 
say they have more friends.
Just says they have willingness.
Who said something?. 
 >>  Values. 
 >>  Somewhere between attitudes and 
values.  Let's not forget, values are 
important outcomes we with work on.
Very good.
You now see what an outcome is.
Here is the question in my mind.
So what.
Irreferantly.
So what.
So what if we figured out now we know 
what outcomes are.
Turns out big what.
Really big what.
People are focusing on outcomes like 
crazy in all sorts of different 
sectors.
Governments at all levels.
Someone here from RSA?
I think there's going to be sometime 
along the line.
Governments at all levels including 
RSA.
You all read that.  You know that.
Accrediting bodies, national 
nonprofits associations, local 
nonprofit agencies, private 
foundations, international agencies.
Earlier this year I was in Bangkok 
and bogate, our foreign aid 
department, international government, 
very big on wanting people to 
determine what it is you want to 
change and how would you know it.  
Very big on outcomes.
I don't think these are the reasons 
you should.
If I were in your shoes, this would 
not compel me enough to do it.
Let me try to convince you, if I can, 
that you should do it back on the 
your own CIL.
This is a true story because I was 
involved in it.
In Washington DC there's a local 
nonprofit, faith based called 
samaritan ministry.
They do a lot of good things.
Churches get together and fund it.
They do a lot of good work.
One of the things that is the hardest 
they try to do, you can picture how 
hard, is an employment program for 
homeless alcoholic men.
Boy, tough population to work with 
and get jobs, right?
They work hard with them.
They give them some interviewing 
skills, they help them find leads, 
they help them learn how to set up 
appointments, they give them bus 
tokens to get to the interview.
Dog gone, men are being placed in 
jobs.
Not every single man by a pretty dog 
gone good rate.
They are pretty happy and comfortable 
and proud of that.
A friend of mine was running this 
place and we were talking 1 day and I 
said well, that is really great.
Is that the outcome you care about?
Well, yeah, of course it is, getting 
men in jobs.
Is that the only outcome you care 
about?
Well, no.
I mean, we want them to keep the 
jobs.
Is that the only outcome you care 
about? ?. 
 >>  Well, no, hope is after a while 
they will get a better job.  These 
are all entry level jobs.  You can 
imagine. 
 >>  Okay, great.  How you doing?. 
 >>  Oh, how am I doing.
Well, men are being placed in jobs.
Yeah, by how are you doing on the 
other two?  Are they staying there?  
Are they getting promoted?
You know, I just don't know.
Good questions.
Well, thin we could find out?
Cost too much money.
Why would it cost too much money?
You have volunteers at your 
organization that kind of are always 
saying there's something I can do to 
help?
Well, yeah, faith based churches, 
people volunteer and say help you out 
if we can.
Why don't we get won of them to make 
some phone calls.
Oh, okay.
We go e this guy, he is willing to 
make some calls, he calls like, I 
can't remember, 25 employers.
How is it going with that man we 
placed in the job for you?
Not so good as it turns out in 
reality.
Not so good.
Not only were they being fired, they 
were being fired fast.
I did a little chart on it once.
Dropped off the earth.
It was like after 3 weeks only like 
20 percent of them were still there.
Why were they losing their jobs?  
They were fired for generally getting 
into some kind of argument with the 
boss, theory physical or verb -- 
either physical or verbal argument, 
they just weren't there.
Okay, what did the program do?  Did 
it hide its head and say my gosh, 
it's terrible, aurful, don't tell 
anybody?  No, they didn't do that at 
all.
They learned from that and changed 
their activities.
Here is at least three things I 
remember they did.
First, they realized these men were 
not used to authority, people telling 
them what to do, when to show up.
They built into their work with the 
men role play sections, quite a few 
of them, where the boss would tell 
them something.
Maybe the boss would be nice, maybe 
the boss would be a jerk.
They tried a lot of different role 
playing, put them in that situation 
in role playing before reality.
Second thing they did, they had for 
the first month after getting the 
job, the men had to come in every 
Saturday, little group session, and 
talk about how it was going, what 
problems you got, how do you solve 
it, da da da.
Third thing, they started calling the 
employers, think for the first week 
they called every other day.
Any problems, any problems, any 
problems?
They tapered it down, but they kept 
calling pretty regularly, how is it 
going.
They changed their program based on 
some bad outcome information.
My gosh, that is what we want.
That is all what we want.
The outcomes were much improved.
I didn't actually see these numbers, 
but they told me later about a year 
or so that they went back and did a 
follow-up and people were much better 
at staying in the jobs and getting 
raced.
And that is the reason to focus on 
outcomes.
You're going to get tired of hearing 
me say this, I guarantee.
For the next two and a half days I'm 
going to keep saying, it's not about 
looking good to RSA.
It's not about looking good to RSA.
It's about doing the best job you can 
for those wonderful people you're 
working with.
It's about you, it's about your CIL, 
it's about making yourselves be as 
effective as possible.
Here is the good part.
If you do the second way, if you make 
yourself as effective as possible, if 
you're continually improving your 
services, the first one takes care of 
itself.
You have the story to tell RSA.
Right?
If you do the second one, the first 
one automatically takes cares of 
itself.
Let me ask you this.
If you do the first one, look good to 
RSA, does the second one 
automatically take care of itself?  
No.
In fact, you're liable to screw 
yourself up, to be honest, because 
you're liable to pick outcomes and 
ways of measuring, as you're talking 
about, that are easy and make 
yourself look good to RSA, but aren't 
going to tell you very much about 
where you're warts are and what you 
have to work on to get better.
So you're going to hear me say that 
again.
Do this for yourself.
Do it for your effectiveness.
Do it for your clients.
The story for RSA will right itself.
That is the difference between what 
some people call outcomes 
measurement, which is simply the 
nerdy research activity of measuring 
this and putting it into a report, 
and outcomes management.
Outcomes management, the systematic 
use of outcome information to improve 
the effectiveness of your  -- 
parameters, that is first, right?
First you improve the effectiveness 
of your programs.
Then second, communicate their value.
Takes care of itself automatically.
You will hear me say this again too.
Here it is.  Why do outcomes 
management?
Increase program effectiveness.
That is job one.  Used to be an ad 
like that.
Job one, increase program 
effectiveness.
Then you will have a great story to 
tell.
Oh, by the way, I forgot this.
In the group in Washington DC, you 
might think having that bad outcome 
data about the men losing their jobs 
would have hurt them.
Actually a little local foundation in 
town was so happy that they had had 
the guts to look and see and find out 
and improve, they gave them another 
$-10,000.
They said we just think you guys are 
managing right.
That is true.  I'm not making that 
up.
That is true.
So you increase your program 
effectiveness and focus on that's 
correct you will have a story to 
tell.
One more time, just to make sure 
we're real clear on this difference 
here.
What is the difference, how does 
outcomes management differ from 
outcomes measurement.
If you notice this course is on 
outcomes management.
The measurement helps to measure 
program performance and report 
findings.
What is happening.
Don't get me wrong.
That is nice.
It's just not enough.
Outcomes management takes the next 
step.
It encourages an agency to 
systematically use that performance 
information to learn about its 
services and improve them.
Just like samaritan ministry did.
Why is it happening.
How can we make things better.
Again, a thing for our brain for the 
next two and a half days, I'm not 
hear to learn about outcomes, sorry, 
not hear to learn about outcomes 
measurements.
I'm hear to learn about outcomes 
management.
I'm going to make my CIL as effective 
as possible.
All right now, these desired outcomes 
we're talking about earlier, here is 
the list we had up here.
I won't go that far back.
The NCIL outcome measures project had 
to figure this out themselves.
This is the real world work we have 
been doing.
What are the desired outcomes of a 
center.