RICHARD PETTY: Let's, 
we're going to cover 
outcome measures this

afternoon, and we'll spend 
several minutes doing that.

But let me just go back for 
a moment and remind you,

I know a lot of you would 
love it if we would,

if we would produce a model 
plan that we could drop in

your lap before you left, 
and the good thing is,

is that even though we don't 
think we, we don't think that

will serve your purposes, 
because circumstances

from one center to another 
differ so widely,

that producing a model would 
give you a document that just

wouldn't be very meaningful,
but already you've had

a couple of frameworks 
that you could build a

quality initiative around,
and you've seen the 
RSA review tool.

We've talked about what's going 
on in Wisconsin, and there's 
some excellent material there.

For services, I'm going to 
suggest that you could use the

outcome measures that were 
developed by the NCIL taskforce

as a framework for your quality 
initiative around services.

And we'll talk about that as we 
go along, and before we finish,

we'll go through all eight 
of the outcomes and the

eleven indicators,
and we'll explain what

all those are before 
we're finished.

And when it comes time to go 
through the, through the

indicators and outcomes 
one by one, Paula is going 
to help me do that.

And we have a couple of 
videos that are going to be 
part of this presentation.

Paula is going to 
help with that also.

So the background 
is that in 2006,

the National Council on 
Independent Living formed 
a task force on outcome measures.

And that taskforce has operated 
for these several years,

and before we're finished,
we're also going to have an

update from some of the folks 
who are involved now,

Tammy Hoar is involved.

Were any of you 
on the original taskforce?

I don't think any, 
I'm not sure anyone 
was on the original

taskforce, but some of you
participated in the test.

Tammy's center did.

Did anyone else participate 
in one of the two tests?

PAULA MCELWEE: Kansas.

RICHARD PETTY:  Okay, great.

All right.

Well then, when we talk 
about the test, if either

of you have, any of you have 
points that you'd like to add

to this, please do because 
that will add a lot.

Certainly, you, 
we'll go over these.

You may want to adopt 
them at your center as the 
taskforce developed them.

You may want to 
develop your own.

You may want to do nothing.

And, you know, for the most 
part, I think all those are

okay with us, those of 
us who are trainers.

There is a lot to recommend 
using one common set of

outcomes and 
indicators nationally.

We're still on slide one,

so a lot to recommend using 
that common set of outcomes

and indicators because 
all centers will be doing 
very similar things.

It's a way of demonstrating 
a success story nationally

for the independent living 
movement, for all of the

centers for independent 
living, and we're at a point 
where that's really crucial.

Okay.

Let's review for a minute the 
points that Paula talked about

yesterday on slide 2 for 
quality initiative because

there's going to be some 
commonality here, and this 
is a good point

just to review why 
we engage in 
quality activities.

First, it helps us do 
our work well, it helps 
us do it effectively,

it helps us offer the right 
services, do the right things,

and remember the quote 
that's a Peter Drucker quote,

nothing is so useless 
as doing the right,

the wrong things effectively.

So it's really truly important 
that we're engaged in doing 
the right things.

This or the quality 
initiative can help us

to create transformational 
change within our 
organizations,

within our communities,
for the people that we serve.

And we talked about what was 
transformational change.

You know, if we look 
back at what centers

have always wanted to 
do within communities,

it was to break
down barriers,

it was to make 
communities 
fully accessible

it was to transform places, 
communities into places 
that were accepting

and open and into which 
people with disabilities 
could be fully integrated.

And it was also to help people 
in their own lives make

transformations from being 
isolated, segregated,

unemployed, disempowered,
to being people who were

people of power, people who 
were fully part of their

communities, and for many,
many people, employment.

Now, from looking back at the 
thinking of the people who

were involved in the initial 
work of the taskforce,

they also had a vision for why 
we would do outcome measures,

and it was to improve 
effectiveness of services,

it would be to make sure that 
we were offering the right

services, to do the things that 
were really important,

that were really effective at 
creating change for people with

disabilities, and then also,
and notice the crosswalk here,

from what we were talking 
about in terms of quality,

the purposes of quality,
and many of the purposes

of being engaged in outcome 
measures are the very same.

In addition to that, in 
addition, another reason that

the community wanted to be 
engaged in outcome measures

was to tell the success story of 
centers for independent living.

And then, finally, continuous 
quality improvement, which,

of course, is just exactly 
to connect with quality.

Just looking back 
historically also,

about why there was such an 
interest in creating the

taskforce, and there were 
key independent living leaders

in the community saw the value 
of doing an outcome measures

initiative, but in addition 
to that, RSA was interested

in outcome measures, and the 
reason that RSA was interested

in outcome measures was that 
the Office of Management and

Budget had initiated something 
called the Program Assessment

Review Tool, or PART, in the 
1990s, and that was how OMB was

looking at the effectiveness of 
government programs and trying

to demonstrate to taxpayers and 
other people in the community

that the government was doing 
valuable and worthwhile things.

And that's why RSA became 
interested in it, was because 
OMB was interested in it.

And the troubling thing was 
that when it came time to

look at the centers for 
independent living program,

in the PART report, centers for 
independent living were shown 
as results not demonstrated.

It didn't say that the 
program was ineffective,

but it did say that they 
could not demonstrate the 
results of the program.

And that, in a time 
of funding difficulties,

was a very difficult 
circumstance for the 
centers to be in.

And things aren't that 
much different right now 
in terms of funding.

We still have a great need for,
to be able to demonstrate the

effectiveness of the centers 
for independent living program.

So we're still looking for ways 
of doing that, and outcome

measures continue to be a 
worthwhile approach.

Now, also, there have 
been some state initiatives

for outcome measures,
and those have proven

successful, so there were 
enough national leaders

who had seen some success 
at the state level that they

really wanted to move 
forward with an outcome 
measures taskforce.

The taskforce, as I mentioned,
was formed in 2006.

Bob Michaels was appointed 
as chair of the taskforce.

We've already talked about Bob.

If you don't know Bob,
Bob is the former director

of two of the largest centers 
in the country.

He's worked with ILRU as a 
consultant for 15 years.

He's done many valuable and 
worthwhile things in the 
independent living field,

and he was the natural choice 
to lead the taskforce.

There were ten members of the 
taskforce representing

centers for independent living, 
statewide independent living

councils, ILRU, of course
NCIL, and the University of 
Kansas.

The taskforce reached out to 
find a national consultant to 
lead the taskforce,

and that person was 
Mike Hendricks.

ILRU had worked with Mike 
previously in the Real Choice

Systems Change Initiative,
and Mike was working with

grantees in that arena to help 
them with outcome measures

and on program evaluation,
and we felt Mike would be

a good person to work with,
a good person for this work,

and indeed, he, indeed he was.

He proved to be invaluable 
to the taskforce.

Just a second.

As a strategy, the taskforce 
saw that it was really

important to engage in a 
strategy of outcome management,

not just outcome measurement,
telling the success,

I can't say that word,
telling the success story

of centers was important,
but the taskforce also

recognized that 
outcome measures 
could be an important

part of an overall 
quality improvement 
program for centers.

So they saw it as 
a real opportunity to

improve the work of the entire 
independent living field

through the work 
of the taskforce.

We talked a little bit about 
some scientific managment

strategies yesterday and 
one of the strategies that

Deming saw as so valuable 
is called the Shewhart

method and that's the 
name of a person.

Shewhart suggested that 
a system of planning and

doing and checking and 
adjusting, plan, do, check,

adjust, was a really useful,
overall strategy and the

members of the task force 
believed that by putting

outcomes in place,
checking, monitoring,

measuring, and adjusting 
would lead to a results based

management approach for the 
field or managing for results

and that is something that 
brings outcome measures

in a direct connection 
to a quality initiative.

The task force conducted 
two field tests, one in 2009

and one in 2010 and there 
were 21 centers that

participated in the first test.

Thirty-two in the second.

Twelve centers actually 
repeated the test.

They both wanted to support 
the field and were seeing

results within their 
centers that led them to

want to continue to work 
and to get better at it,

so they, indeed repeated.

What did they say they 
learned from having gone 
through the tests?

Well, the first thing 
that the center said that

they learned is that the 
work was worthwhile.

That engaging in outcome 
measures was worth

the effort, and 66% 
of the centers that 
participated said it was

worthwhile and interestingly,
even more of the centers

that participated in the 
test said that they would,

they would do it again.

And several did, in fact,
repeat and said that they 
would continue the work.

So that's telling, I think.

And what they also said 
was, it's not easy.

It involves staff time.

It involves new systems.

It involves new approaches,
and it means that you will

be doing things in a very 
different, different way.

Some of the centers that 
participated in the tests 
weren't able to finish.

They couldn't collect the 
data that needed to be

collected, so I think it's,
as important as it is and

as good as it is, as useful as 
it is, we also need to be very

clear that it's something 
that's going to require

effort and resources.

When the task force 
did its work, it first

developed a set of outcome 
measures and as you heard

yesterday, they actually 
initially developed 16.

They went through a process 
by which the field, reviewed the

measures and rated or scored 
them and they had hoped to

choose 6 and they actually 
ended up choosing 8.

And as Bob said, 
this is independent living,

and people make their 
own choices and the field

made their choices,
so there were 8,

8 measures that we have,
and the next thing that was

done was that the task force 
developed a logic model,

and logic models describe 
the process by which 
objectives operate.

They allow you to choose 
an overall objective

they require you 
to choose overall

objectives and then the 
streams of processes or

the sub-objectives that 
flow through to those

higher level objectives.

This is the logic model,
and you also have a larger,

full color version of it in 
front of you there are

alternate format text 
versions on your tables

also, and what we will do,
though, is I don't think

anyone can describe the logic 
model better than Mike Hendrix,

so we are going to let Mike 
shed some light on the model

you have in front of you,
sorry, not Mike, but this is

Bob's presentation on the model 
and how it works, how it was,

what the thinking was 
behind it, so let's hear Bob's 
presentation on this.

(Video plays) 
BOB MICHAELS: So here 
is what it looks like.

RICHARD PETTY: This is 
the model that you have.

BOB MICHAELS: Now, 
this is the proposed service model.

On the bottom, if you look 
at the one in your packet,

on the bottom are activities, 
the IL services, I&R and 
systems advocacy and

on the left side going up is 
the initial outcomes,

the intermediate outcomes 
and the ultimate outcomes.

You will notice that the 
ultimate outcomes then,

are up at the top in blue 
and at the very top one is

that people with 
disabilies are 
integrated into 
American society.

So if we did everything 
that we want to do as centers

for independent living,
then the people with

disabilities would be 
integrated into American

society and there is a 
good example of some 
thing that changed over time.

One of the very first changes 
we had to make was we had

people with disabilities 
are mainstreamed 
into American society.

And we got all kinds of 
comments from people who

felt  the mainstream
was too tied to education,

and that it was not 
a good word to use here.

And we had to talk about 
integration and people said, 
well, it's too tied into

racial things and changes 
that have happened in 
the civil rights,

but just as many people 
said, that is exactly what 
we want to do.

So we ended up with 
people with disabilities

are integrated into 
American society.

Now, let me show you 
the streams because 
this is too much.

If you look in your packet,
you will see on page 3 of

the little packet that 
I just gave you,

sent to you, it has 
a stream for IL services.

Now, oh, gosh, that's right,
that's right.

We have one also for 
the ultimate outcomes.

But this stream,
this first one starts 
with the IL services.

Now, this is all of the pieces 
that have to do with any

services that are provided 
by an independent living

that are not I&R and 
not systems advocacy.

So it's just about 
everything else.

Peer support,
skills training,

transition services,
transition assistance,

individual advocacy,
all kinds of things.

If people get those services 
from an independent living

center, if they get that,
then they have the skills,

knowledge, and resources 
to support their choices.

And if they get that,
then they make their 
own choices.

If they get that,
two things will happen.

They will regard themselves 
as more independent.

In fact, they will be 
more independent.

Okay.

Again, this is what you felt with 
the membership and all the

centers from independent 
living and people in

independent living around 
the country felt should be 
the desired outcomes.

Okay.

Here is the I&R stream.

If you get I&R from the 
center for independent

living, that people with 
disabilities see different

possibilities and they get
the information they need.

If those two things happen,
then, well, no, wait.

If people get the 
information they need,

then they will begin to 
advocate for increased 
community supports.

Notice how the arrow 
goes out to the right 
side of that?

Well, if  people are 
advocating for increased

community supports, then they 
are doing systems advocacy,

and this ties them into the 
systems advocacy stream.

Okay.

In systems advocacy, if you 
identify the barriers and

problems in your community,
then the community agenda

for change exists and 
active coalitions exist 
around our issues.

If those two things happen,
then decision-makers will 
act on our agenda.

If they do that, then 
two things will happen.

Communities will have more 
resources that support

independence and 
methods and practices will

be developed that 
promote independence.

Okay.

So now you have 
these three streams.

What happens when we start 
off with independent living services,

I&R, and systems advocacy and 
you build them up and what

you are looking at 
here are initial and 
intermediate outcomes.

And here is the 
ultimate outcomes.

That people with disabilities 
participate in the community

to the extent they wish,
that comes up from the IL 
services stream.

Communities are more 
accessible in housing,

transportation,
information,

employment,
education,

aging and healthcare 
and so forth.

And that comes up out of 
systems advocacy and I&R.

And then if those 
two things happen,

then people with 
disabilities are 
integrated into 
American society.

(End of video).

RICHARD PETTY:  We are 
going to talk about 
now indicators and all we

have been talking, we have 
been talking about outcome 
measurement.

That's what the task force was 
named and every time we

mention it up to now in this 
training we have said 
outcome measures.

Well, actually you can't 
measure outcomes.

Outcomes themselves 
are not measurable.

So let's let Mike Hendrix 
shed some light on this

and what sounds like 
a contradiction.

So let's play that video.

[Video plays]
MIKE HENDRIX:  Okay.

What makes it nice?  
Anybody.

You have a roll-in shower.

You have a comfortable bed.

And you have an 
accessible room.

I'm sorry, 
I couldn't hear that.

It's clean?  
Someone said it's clean.

Wireless internet.

Air conditioning.

No bed bugs.

Where do they normally 
make you stay?

[Laughter].

You need a, well, you need 
a CIL that's not so cheap.

Yeah.
You have cable?

Nice.
Okay.

So if I counted correctly,
that was seven different things

that made a room nice.

And I bet we could have 
gotten 17 if we kept going.

When you go to look at a room,
do you look at the room,

or do you look at these seven 
different kinds of things we 
were just now talking about?

Or these 17 different kinds 
of things, or whatever it is?

What is it you look at?

Do you look at the room,

or do you look at these 
different aspects of the room?

Yeah, if you can identify 
what it is, then you look 
at those aspects.

So here's what I want to say.

Nobody is measuring 
outcomes today.

Nobody ever has 
measured an outcome.

Nobody ever will 
measure an outcome.

That's pretty radical 
for a session on outcome 
measures and CILs.

Right?

You do not measure outcomes.

You measure 
indicators 
of outcomes.

You don't look at a hotel room, 
you see if it has a roll-in

shower, if it's clean,
if it's accessible,
if it has bed bugs.

You look at different 
aspects of it.

And those things define what 
you mean by a nice hotel room.

Add them up, and that's 
a nice hotel room for you.

Okay?

Same thing with outcomes.

You will not be 
measuring your 
outcomes.

You will be measuring 
your indicators.

Pat, uh-huh?

AUDIENCE MEMBER: I think 
the bed bug issue,
it really does bring up

something that I have 
often wondered about,

is that the prevention of bad 
things from happening to people

can actually be a statement 
about the power of,

the power and the ability of 
the activities that you do.

But it puts you in a position 
of knowing what the bad things

are and having some baseline,
and then showing how your

interventions can stop bad 
things from happening.

And I know we do it at centers 
all the time because we talk

about how many people did 
not go to nursing homes on 
account of what we do.

But how do, how does the larger 
world of program evaluation

think about measuring the 
absence of bad things,

as opposed to 
the presence of 
good things?

MIKE HENDRIX: Good 
question, Pat.

And may I suggest you're 
kind of talking two different 
concepts here.

One is the measurement of it,

and the second one 
is the analysis of it.

And those are two 
different things.

So the measurement,
I think you probably

would agree with me, is 
pretty straightforward,

like are there any 
bed bugs there or not.

Or, where is the person living.

So the measurement is not the 
tricky part, it's the analysis

of it that, where you 
have to be careful.

And we'll get into that later 
in the week if that's okay,
if you don't mind.

But you raised 
a really good point.

Yes, sometimes we 
want to achieve things,

and sometimes we 
want to prevent things.

Both of those are desired 
outcomes, or could be.

It's up to you.

Both of those could 
be desired outcomes.

Entirely your choice,
completely.

Yeah.
Okay.

So I've just said you will 
not be able to measure your

outcomes at all, you have to 
measure your indicators.

That's the next step on 
our yellow brick road,

step four, 
is measurable 
indicators.

So what the heck is a 
measurable indicator?

I think you know this in 
your heart, you know this

in your gut, you know 
this intuitively,

let's just see if we 
can't be a little more 
specific about it here.

It's the specific item 
of information that tracks

or indicates a program's 
success on an outcome.

It defines, oh, this is 
something I would 
underline if I were you.

I would under line that word,
"defines," right there.

Because I think if you can
get it in your head that an

indicator defines what you 
mean by the outcome,

you're a long way 
down the road.

That's a really good way 
to think about it.

The indicator defines what 
you mean by the outcome.

Whether it has a roll in 
shower defines for our friend

here whether that's a good 
room, whether it's clean

defines whether 
it's a good room.

Whether it doesn't have bed 
bugs defines whether it's 
a good room.

That's an awfully good 
way to think about it.

The indicators define what 
we mean by the outcomes.

In fact, I would go so 
far to say that outcomes 
are really warm fuzzies.

That's the way I 
describe outcomes.

They're warm fuzzies.

We could almost sing 
Kumbaya as we read 
some of them off.

They're nice.

Don't get me wrong,
I'm not making fun of them.

But there's not enough meat on 
them yet, they're just concepts,
they are just warm, fuzzy concepts.

It's the indicators that start 
to put the meat on the bones.

Because if I defined a hotel 
room as having a king-size bed,

on the first floor,
and, I don't know,

not next to an elevator, 
and you defined a nice hotel

room as on the 15th floor,
with two double beds,

and near the maid station,
hey, we've defined it 
completely differently.

And we're going to come up with 
completely different answers on

whether our room 
is nice or not.

And that's a real important 
concept to have in your head.

The indicators define what 
we mean by the outcomes.

So it also shows how much the 
outcome is being achieved,

and it's often expressed 
as a number and percent of 
participants achieving it.

[End of video.]

RICHARD PETTY:  So just 
to review, outcomes define 
what, I'm sorry,

indicators define what 
we mean by outcomes.

Outcomes themselves 
are not measurable.

Remember Pat's points?

From the video, Pat Puckett's 
points from the video,

they can be something we 
want to achieve, they can

be something that we want to 
prevent, they can be either.

Now, let's, let me just share 
with you briefly about how the 
taskforce chose the indicators.

And there are, there are 
eleven indicators for eight, 
eight outcomes.

The taskforce's own 
knowledge and expertise 
of independent living.

We reached out to 
national experts and 
international experts.

We did do 
a literature review,

but several other countries,
we reached out to them in terms

of both just evaluation issues,
outcome management,
and independent living.

We had a competition.

We asked people in the 
field if they would recommend

indicators, and we had over 150 
entries, and we awarded three 
prizes to the top folks.

We awarded gift cards,

I think around a hundred 
bucks in gift cards to each

one of the persons who had,
who had made some

recommendations 
of good indicators.

So the field was involved in 
the indicator decision process.

So remember the three, the 
three streams of the logic

model, which were services,
information and referral,
and advocacy.

And then in those streams,
we have, in services,

there were two outcomes 
and two indicators for 
information and referral.

There were two outcomes 
and four indicators, and for

advocacy there were 
four and five.

So we're going to now take a 
look at each one of these

individually, and Paula is 
going to help me with them

by reading the slides as 
we go through, and we'll talk

a little bit about them.

But before we do that, 
any questions about where 
we've gone so far?

Do you see how,
let me wait.

I'll ask that.

I'll ask that after we've gone 
through them individually.

But let's begin with --

PAULA MCELWEE:  The IL 
services stream is the first

one of these, and outcome that 
was identified was persons with

disabilities have the skills,
knowledge, and resources to 
support their choices.

And the first indicator under 
that outcome is the number and

percentage of consumers served 
by the CIL within the last nine

months of the past fiscal year 
who can list at least one

specific skill,
type of knowledge,
or resource that they have now,

that they did not have 
before approaching the CIL.

RICHARD PETTY:  Okay.

Don't get caught up on 
the time period because 
that may well change.

It had more to do with what the 
taskforce thought the centers

that were involved in 
testing could accomplish,

in terms of being able 
to put that together and 
to be able to report it.

So the field could well 
make a different decision

about the time,
the time period involved,

especially in terms of how we 
might look at 704 reporting,

and again just to recognize 
that the taskforce is

recommending to RSA that the 
704 report incorporate outcome

measures, so that's an added 
reason that you may want to

be involved in 
implementing outcome 
measures in your center.

But any questions 
about that first one?

Okay.
All right.

PAULA MCELWEE:  All right.

The second, oops,
I went the wrong way.

Let's try that.

The second outcome is under 
the IL services stream,

is persons with disabilities 
are more independent.

And the indicator is the number 
and percentage of consumers

served by the CIL, again within 
that time period, who can list

at least one specific way in 
which they are more independent

than when they 
approached the CIL.

RICHARD PETTY:  Individuals 
reporting when surveyed.

Okay.

PAULA MCELWEE:  You want 
me to go on?

RICHARD PETTY:  Yeah.

PAULA MCELWEE:  Okay.

The information and referral 
stream, the first outcome

identified is:  Persons 
with disabilities get the 
information they need.

And this slide contains 
two indicators.

The first one is the number 
and percentage of persons with

disabilities contacting the 
CIL who report they have the

information they 
requested from the CIL.

I'm just going to skip that 
time period if that's okay.

And the second is the 
number and percentage 
of persons with disabilities

contacting the CIL who used 
a new resource that they

learned about from 
the CIL I&R efforts.

So the first one they have the 
information they requested and

the second one they 
used a new resource.

And then the next outcome 
in that same area,

persons with disabilities 
advocate for increased

community supports and 
two indicators again here,

two indicators here,
the number and percentage

of consumers served who can 
list at least one specific

personal advocacy activity 
that they engaged in.

Wow, cool.

And then the number and 
percentage of consumers 
served, who can list at least

one specific systems advocacy 
activity that they engaged in.

And there is one more,
outcome and systems 
advocacy, want that one too?

Barriers and problems 
identified was the next

outcome and the indicator 
is the number of activities

conducted, such as surveys,
public meetings, focus groups

and polls to identify or 
confirm the primary barriers

and problems in the community 
that prevent persons with

disabilities from leading 
more independent lives.

Then one more outcome 
under systems advocacy,

a consumer, no, I guess there 
is more, consumer agenda for

change exists and the indicator 
for that is, presence within

the CIL's annual plan of a 
separate section containing

an explicit systems 
advocacy work plan.

And then the next outcome 
on systems advocacy,

notice a heavy emphasis on 
"systems advocacy."

Decision makers act on our 
agenda and that is the number

of positive changes achieved,
or negative changes prevented

in legislation,
policies, practices or

services at the local, state,
or federal level that address

the barriers or problems 
identified by the centers consumers.

And the next outcome 
under systems advocacy,

methods and practices 
promote independence.

Two indicators here,
the first is the number and

percentage of consumers 
served who moved out of an

institution and into a 
self-directed community-based

setting, and the number and 
percent of consumers served

who remained in a self-directed,
community-based setting on

December 31, despite 
having been at risk of 
moving into an institution.

RICHARD PETTY:  Okay.

Now the question, can you 
see how this could be a

framework for quality 
management within 
your center?

AUDIENCE MEMBER: Jason Belongy 
from Madison, Wisconsin.

I think the shift being,
instead of being one 
more thing

to do with measuring the 
impact is, that should be

more the focus than 
the stuff we are doing.

Is how we are impacting

people's lives through 
independence, how we are

impacting certain services 
we do in the community

through impact because 
policymakers in Washington 
and elsewhere.

That is what they are 
looking at that,

is the impact, the 
return on investment.

They are looking at the impact 
of their, of the money that

constituents pay through taxes 
that comes to our centers

through federal and 
state funds the impact 
on the community.

So I think, 
I try to keep in mind

not so much that this is going to 
be one more thing but this is

more of a correct process in 
doing more of a focus about

outcome versus just 
measurement 
of stuff we do, is I think vital,

not only to us being better at 
centers but also moving forward in

terms of growth and 
sustainability of standards 
nationwide.