MICHAEL HENDRICKS:  So what 
did we do this morning?

This morning we have talked about 
one leg of a three-legged stool.

Well, let's see.

If my math serves me right, 
three minus one, we have two legs to go.

Let's deal with the second of the legs.

All right.

I always promised you I'll 
be honest with you.

I will be very honest with you and say this 
may be the hardest of the three legs.

Is it doable?
Absolutely.

Is it going to take some work?
Absolutely.

Is it going to be work that 
you traditionally have done?

Maybe not.
So it may be new stuff for you.

Now if you were here in…

If you were there in Las Vegas last year, 
you got a real leg up on where we are but

don't worry even if you weren't we are going 
to cover the same stuff in some ways.

So you won't be lost.

So here is the key words on this.
By the way you have another,

you have another handout in your 
NCIL folder that is exactly what

we are doing this afternoon.

It is called evaluating progress 
on SPIL objectives.

That's the one you want to get.

And that's the key thing here.

Okay.
We have got the SPIL; right.

And we have got objectives in it.

Let's just make up a number and say 
eight because we used that earlier.

Let's say you've got eight 
objectives in that SPIL.

It is one thing to know if we have 
implemented what we need to implement.

That's one thing.
That was this morning.

But it is a completely different 
thing to say how are we doing.

So what?

No matter how well we have implemented 
any progress happening.

That's what we are into now.

Is there any progress 
on those eight objectives.

Let's remember what we are asking 
you to do because there are some

things that you may not be doing.

What we are definitely saying, 
you remember this is the same

one we looked that this morning.

You are saying it is your job, 
we think, to convene people

together to make sure this happens.

We think it is definitely your job.

I am skipping a line.

We think it is definitely your job to 
coordinate the information gathering efforts.

Those are the two, we think, 
you should do probably by yourself.

Other stuff we think you should be 
involved in for sure as a partnership.

Planning how you are going to do the 
evaluation, discussing and interpreting

the findings, decide what steps to take 
to make improvements and amending

the SPIL as appropriate.

So you are going to be involved in all of 
those either by yourself or with other people.

Here is the tricky part.

The question marks.

The question marks, I don't know, 
and right now you may not either.

I don't know if you are going be involved 
in gathering the information that's going

to tell you whether you're actually 
accomplishing your SPIL objectives or not.

And I don't know if you are going 
to be involved in compiling and

analyzing what comes back.

And I don't know if you are going 
to be involved in implementing those

improvements to the IL system 
that you come up with.

Those are question marks right now.

But you've got to accomplish all of this.

Here is that RSA requirement again.

The second leg progress on achieving SPIL 
objectives required by RSA section 364.38.

The state plan must, again not should, would 
be nice to, we would love to see, no no.

Must establish a method for the 
periodic evaluation of the

effectiveness of the plan in meeting the…

Does it say meeting the mission?  No.

Does it say meeting the goals?  No.

It says meeting the objectives, and that's 
not -- that's not random wording.

RSA used that word specifically to mean 
the objective level of the logic model.

The effectiveness of the plan to meeting the 
objectives established in blah blah blah...

So you have to evaluate your objectives 
and how well you are doing on them.

So another way to put it is are you 
accomplishing what your SPIL says

you want to accomplish?

Not like this morning, do you have stuff in 
place, you know, do you have the resources

and the staffing and do you have 
that stuff in place?

No, no, beyond that, beyond that, 
now it is in are you actually

accomplishing what your SPIL 
says you want to accomplish.

Or another way to think about it, 
are you changing your state in

the ways you want to change it?

And this is a key interest of RSA.

People who have been talking with them 
regularly say that in the discussions

it is very clear this leg of 
the three-legged stool,

this objectives leg is an 
important leg for RSA.

And you know I mentioned earlier about 
the new legislation that the President

just signed about outcomes and 
improvement and performance and all that.

This is that leg, this is the actual results, 
outcomes, this is that leg.

So this is real important.

Now the good news is, and I will say 
hopefully because I haven't seen all

of your SPILs but I will knock on wood.

The good news is if you followed anything 
close to what we suggested in Las Vegas

or what is on the wiki at ILRU, 
you've done a lot of this work already.

That's the good news.

Let's see, here is what I think.

Here is what I am hoping, how is that?

You have done or if not to be honest, 
I think you're going to have to go back

to your SPIL and kind of re--not change
it don't get me wrong I am not asking you

to rewrite your SPIL in any way.

But I am asking you to go back and maybe look 
at it through different eyes and maybe you

need to pull some stuff out in different ways.

Here is what I hope you have done.

You have identified objectives in your SPIL.

And you have realized that when we say 
objective we mean something very specific,

not the top level of the logic model,
that's the mission.

Not the middle of the logic model, 
that's the goals.

But what we mean is that level at the bottom 
of the outcomes called objectives.

These flow directly from your activities.

I think I have a -- let me 
jump ahead a little bit.
Yes.

They flow directly from the activities.

You saw this morning, 
you know this already.

It is this line of stuff right in here.

It is the things that if you do your 
activities well, they will happen.

If/then, remember yesterday-if / then.

If you do your activities well, 
then the objectives will result.

So they flow directly from your activities, 
and we happen to think that wording

is really important.

And here I will make a 
suggestion to you, my friends.

A lot of the wording in the objectives that I 
have read in the SPILs is not what I would

suggest would be the most useful 
wording for you as you go forward.

Sorry.
Hang on I can't quite hear you.

I will.
I will give you the opposite actually.

Here is how I would strongly suggest it.

You may listen to this and think, 
oh that is arbitrary or that doesn't matter,

or it is not very important.

I have been doing it a long time.

This does matter and this is not so arbitrary.

If when you are writing an objective you can 
just keep these in mind.

If you can start with your target, 
that's whomever or whatever

you are trying to change, 
and I will give you an example here.

Then follow that up 
with a present-tense verb.

I hate, hate is a little strong, 
in fact given our language,

remember what we were just saying, 
I should not say that.

I should say I don't prefer future tense,

where we say, you know persons 
with disabilities will.

No, no, let's envision the future.

Persons with disabilities do or have or are.

Let's envision present-tense verb and 
end with however makes sense.

Let me see if I can give some examples.

Okay, now look at these.

This follows that language.

Number one formerly underserved 
populations are served.

Okay.
What's the target.

Remember I said start with the target.

Formerly underserved populations.

That's who we are trying to change.
That's the target.

Present-tense verb, are not will be served.

Are served and ends with 
whatever makes sense.

It is served.

So there's target present tense 
verb and whatever makes sense.

Look at the next one.

Persons with disabilities there's our 
target have present-tense verb,

increased employment.

Policy makers here's a completely
different target, right?

Trying to change somebody 
completely different.

Here it is policy makers present tense verb 
financially support the IL network.

Persons with disabilities register to vote.

IL network key partners collaborate to 
provide services.

One more and then I will take that.

Persons with disabilities can, not will be 
able to, but can access transportation.

So here is the format that I personally find 
very useful to write your objectives.

You had a question here?
I am sorry we have two.

Let's go in the back for one.

AUDIENCE MEMBER:  Actually I am 
probably a little ahead of myself

because I agree completely 
with what you are saying.

MICHAEL HENDRICKS:  Then we 
love that kind of comment.

Feel free to repeat yourself.

AUDIENCE MEMBER:  When we 
get into the percentages et cetera,

we were told to 
take those out.

We had initially put a 
percentage of persons

will access transportation people 
with disabilities say it was 5%.

And RSA came back and said 
nope you have to take all of that out,

anything that was measurable that 
we had, we were asked to remove.

MICHAEL HENDRICKS:  RSA was 
correct because there are three

different things going on here.

I don't want to confuse you
too much but we will talk

about this to the 
extent you want.

There's the objective.

There's the indicator
of that objective,

which we will talk about in a bit.

And then there's the 
target level you are

trying to reach on that indicator.

And those are three
different things completely.

As I say, if that's confusing

forgive me but we will cover 
that as much as you want.

And I agree with RSA completely.

Sometimes I often do actually, 
and on this one for the objectives,

I would strongly recommend 
don't put any numbers of any

kind, whether they're percentages 
or numbers or anything.

Leave your numbers out 
of the objective statement.

My opinion too.

By the way, we are going to 
explicitly deal with the indicators.

I wasn't explicitly going 
to deal with targets.

But, ah, because I don't think 
RSA does not require targets,

but maybe I am wrong.

I could be wrong, 
if they do we will deal with that too.

AUDIENCE MEMBER:  Hi.  ?? from 
Maryland again.

I wondered because I wasn't 
a part of like the actual the

verbiage of the SPIL,
I was just, you know.

But I wanted to say, 
would it be okay,

or would it be recommended 
or forbidden or whatever.

MICHAEL HENDRICKS:  Yeah, 
those are different.

I am willing to say what I 
am willing to recommend

but not willing to say what's 
okay, that's too strong.

AUDIENCE MEMBER:  Something 
like persons with disabilities

register to vote and 
encourage others.

MICHAEL HENDRICKS:  Okay.
AUDIENCE MEMBER:  Or --

MICHAEL HENDRICKS:  Since 
you raised it.

AUDIENCE MEMBER:  Persons 
with disabilities can access

transportation without problems
because I don't know about here

but the subways and metros 
and stuff in DC are so -- they're

so problematic, so problematic.

MICHAEL HENDRICKS:  These are 
excellent examples, but they

are -- you just gave two completely 
different kinds of objectives,

one I don't like, one I do.

Let me take the one I do.

You said, I think, 
persons with disabilities

can access transportation 
without difficulty.

I think -- is that right?
I think that's fine.

That's a fine, why wouldn't it be?

It has got the target.

It has got the can access and 
you explain what you mean.

Transportation without difficulty.

Now, when we take the next 
step you are going to have to

know what it means to be 
without difficulty, but that

is your burden 
at that point.

But anyway, we can get to that.

The first one you gave I 
didn't like because

you said, what did you say?

You said, persons with 
disabilities, what was

it can access transportation.

Ah, thank you.

AUDIENCE MEMBER:  Persons 
with disabilities register to vote

and encourage others.

I realize that's a belief 
system, it is political and it is --

MICHAEL HENDRICKS:  It is 
something else, what else is it?

AUDIENCE MEMBER:  Someone's choice.
MICHAEL HENDRICKS:  No, keep going.

AUDIENCE MEMBER:  Giving 
assignments to.

MICHAEL HENDRICKS:  Help 
me out here, there's something

special about what 
she just said.

AUDIENCE MEMBER:  Will?
MICHAEL HENDRICKS:  It's two things.

AUDIENCE MEMBER:  Okay.
MICHAEL HENDRICKS:  It is two things, 
isn't it.

AUDIENCE MEMBER:  Okay.
Was the other stuff wrong too?

Was I correct on anything?

MICHAEL HENDRICKS:  No, 
the second one I like but

registering to vote 
is one thing; right.

Encouraging others is a 
different thing; right.

I'm just saying those are great and fine.

But separate them for 
reasons you will see soon.

By the way, may I, 
forgive me,

saw a whole 
bunch of those.

I saw a whole bunch of 
objectives with more

than one concept inside of them.

AUDIENCE MEMBER:  Okay.
So I am thoroughly confused.

On my objective that I had 
stated earlier, it had numbers.

So what do we do because 
our entire SPIL is approved

and they like the way we had it 
because we had numbers and

we were able to see how we

could achieve those 
goals and objectives.

So where do we put the 
numbers and how do we

go back and measure the 
success of our plan if we

have numbers throughout it?

MICHAEL HENDRICKS:  You 
will have numbers.

First of all I'm not RSA, 
if they approved it, hey, who

am I to say otherwise, I just give 
you my opinion of what is good.

I wonder though did your 
objectives actually have

numbers in it or do you 
have some indicators?

AUDIENCE MEMBER:  It was 
from our plan and it was to

increase the number of 
transportation services

to people with disabilities 
and there was numbers

and it increased each year.

And so we looked at the 
number of services already

provided in 2009 and then had 
incremental increases each year.

And so I am just concerned 
if we had that as a goal.

MICHAEL HENDRICKS:  Objective.

AUDIENCE MEMBER:  I mean an 
objective, how do we --

where would we have moved 
that or how would we have 
phrased that objective.

MICHAEL HENDRICKS:  May I 
ask you to hold that thought

because I think that you will see 
that as we work through the afternoon.

AUDIENCE MEMBER:  Okay 
so you will help me with that.

MICHAEL HENDRICKS:  I mean 
I -- I will show you how I would

have done it.
How is that?

These are all suggestions 
for you to consider.

I will show you how I would have done it.

AUDIENCE MEMBER:  Hi.  
I'm Peter from Vermont,

and I am brand spanking new.

So this might sound like a stupid question.

MICHAEL HENDRICKS:  There are no 
stupid questions, Peter.

AUDIENCE MEMBER:  I have 
noticed that on your list of objectives,

that it is very simple, 
the language is very simple.

It is very direct and it is very short.

As I review our SPIL, our most recently 
approved SPIL which I did not write,

I notice that our objectives are -- there's 
many objectives, almost all of the

objectives in our SPIL are more
than ten times as long as those

and they have multiple 
concepts in them.

MICHAEL HENDRICKS:  Really?
Gosh Peter, I am really surprised!

AUDIENCE MEMBER:  Well 
because I was trying to go

through the SPIL and pull 
out an objective and there's

these paragraphs, they're 
paragraphs they are not statements.

I wondered is this typical 
of what we should be doing.

MICHAEL HENDRICKS:  I am 
not going to use the should word.

I am going to say, if I were in your shoes, 
this is more typical of what I would do.

That's part of the problem that 
I saw looking through some

of the SPILs and the objectives,
there's just like a whole bunch of 
concepts in them.

Yeah.
Here we go.

AUDIENCE MEMBER:  An objective is 
something you want to achieve,

the way the sentences read it sounds 
like they have already been achieved.

MICHAEL HENDRICKS:  I think the 
question was, an objective is

something you want to achieve, 
yet the way I have worded these,

it sounds like they have 
already been achieved.

That is exactly correct.

On purpose though because 
this is optimistic.

This is the desired outcomes, 
remember, the desired outcomes.

This is the way we want the world to be.

This is the way we want the world to be.

So I am asking you to visualize 
the way the world should be.

By the way if you look on the logic model 
from yesterday, the SILC logic model,

same thing, present tense they 
have envisioned how the world

is going to be when the SILC 
is working beautifully.

So, you can disagree with that.

You can change it to 
present tense if you want.

No problem at all.

My advice, change it to 
future-tense if you want.

My advice present.

Helps you visualize what 
you are aiming for you.

But very astute of you, 
you saw exactly correct.

AUDIENCE MEMBER:  What if you had a 
time frame within the next x number of

years, people with disabilities (inaudible)

MICHAEL HENDRICKS:  So you are 
saying some of these seem very

optimistic to achieve within the 
next few years, is that the idea?

AUDIENCE MEMBER:  It would 
have more of an impact said

with the next x 
number of years.

MICHAEL HENDRICKS:  Yeah, 
one of the, you see, I think what

you are saying is it would 
have more of an impact

if we put a time line on it.

If you go back to the wiki and 
look at the kind of things we

suggested from Las Vegas 
we proposed that,

that you put something 
like that in there.

So yes, okay.

Let me take the next step then.

So, while we already know this, 
objectives flow directly from the activities.

Now as a practical matter I am 
betting that not very many of you

actually did a logic model for your 
SPIL although I know some of you did.

So, could I have a show of hands, 
anybody who actually put a logic

model in your SPIL?

Two, three, four, five, six.

Okay, that's more than I actually thought.

You folks will have a relatively 
easy job of just going to your

logic model, looking at this 
level and saying, okay,

that's where I am focusing.

The rest of you won't.

Go ahead.

AUDIENCE MEMBER:  I wanted 
to say that I wanted to do a logic

model but when I realized that
I couldn't input it into my SPIL,

then we stopped the process 
and just went with what was

required for the SPIL 
because that was another

whole level of stuff that 
you couldn't even put into it.

So I found that very 
frustrating myself.

MICHAEL HENDRICKS:  Here is 
where I will take Tim up on his

offer and the good 
recommendations

from people who have 
known him over the

years who say 
he's a good guy.

AUDIENCE MEMBER:  I am sorry.

MICHAEL HENDRICKS:  And say that 
that is one of the real problems last year.

People who did the good work 
of developing a logic model,

did the good thinking had no way 
to share that with RSA and what a shame.

But Tim is going to solve 
that in the next week or two.

[laughter]

AUDIENCE MEMBER:  Can I 
say one more thing on that one?

My other frustration with that 
was throughout your training

on outcomes, that was so 
excellent, you told us time

and again don't use the 
words goal and objective.

And I go to write my SPIL 
as a new person,

what do I have to write?

I have to write goals and 
objectives when I have

been working with my council 
to develop outcomes.

And they're different.

You told us that.

Goals are different 
than outcomes.
So.

MICHAEL HENDRICKS:  They're 
different levels.

AUDIENCE MEMBER:  I know.

But still, it is confusing to folks 
to give us an outcomes-based

training and then tell us to use 
the old terminology that doesn't

go along with that very well.

MICHAEL HENDRICKS:  As long as Tim 
is such a wonderful human being.

AUDIENCE MEMBER:  You are 
a good guy, Tim.

MICHAEL HENDRICKS:  Let's
make that point also.

It does no one any good to use
this jargon, mission, goals, objectives.

It is just confusing.

AUDIENCE MEMBER:  That's 
what I am saying.

MICHAEL HENDRICKS:  It would 
be so much better, may I suggest,

if we just talked about 
different levels of outcomes.

Simple.
That's the concept.

Let's just talk about it.

AUDIENCE MEMBER:  And then 
you can do logic model and go

through the whole 
process.

MICHAEL HENDRICKS:  Yeah, 
but we are picking on poor

Tim who has only been 
there six months.

AUDIENCE MEMBER:  No, 
I was picking on you.

[ laughter ]

MICHAEL HENDRICKS:  I have 
only been here one day.

AUDIENCE MEMBER:  Just kidding, 
Mike.

MICHAEL HENDRICKS:  Well, yeah, 
I hope what we were saying last

year was the important thing is

that these are all outcomes, 
but right now we have got to

label them different things to 
fit within the RSA terminologies.

Okay.
Others?

This is good.
This is just the kind of dialogue.

Now, by the way, some of you 
who are new are probably a

little lost still and saying I 
am not really catching this.

Go to that-wiki and watch 
last year's training or read

the materials and stuff, there's 
a lot of good stuff in there for you.

A question here.

AUDIENCE MEMBER:  Slightly off topic 
but in developing one of these models,

where do you start at the bottom or the top.

MICHAEL HENDRICKS:  It is
not at all off target.

The question was and I love it, 
and we heard it last year, when

you are starting to develop 
a logic model do you start

at the top or do 
you start at the bottom?

Those of you who were
there last year,

do you remember 
my answer to that?

What was the answer.

AUDIENCE MEMBER:  The middle.

MICHAEL HENDRICKS:  haha--No 
but that's a good idea.

I hope the answer 
was: it depends.

Because the answer 
ought to be: it depends.

Let me tell you what it depends on.
This is what it depends on.

If you are absolutely locked
in to the activities that you

have to do for tradition or 
some kind of contracts or

who knows what, and those 
in fact are the activities you

are going to do, then I would 
say start at the bottom and

just think through the process of 
what are these going to achieve.

Because lots of times it is 
not going to be the same

thing people have always 
thought they were going

to achieve if you really 
give thought to it.

Okay, well, if I have 
these eight activities

let's say what are 
they going to lead to.

If though you are not 
locked in to your activities

and you have some flexibility 
of what you do, in your

activities then I would 
strongly say start at

the top, keep the eyes 
on the prize if you will.

Why are we here?

The highest level, 
and then ask yourself

what do I have to achieve 
to get up to that.

And then below that, 
what do I have to 
achieve to get up to that.

And then eventually you 
will get down to where,

okay, what must my activities be in 
order to jump kick start this whole process.

So it really does depend.

I hope most of you are in 
that second situation where

you can start at the top 
and then let your activities

flow from what you need to 
do, not what you have always done.

You may find that there is a difference.

You may find that either you 
have been doing something

there's really no need for or 
you haven't been doing

something that you should have 
but it will be a logical flowing out.

That's a good question, 
excellent question.

Okay.
I will go on here.

So you have done that.
See, I am going to be an optimist.

Oh, I am sorry.

I was in the middle of saying 
those of you who did the logic 
model have a piece of cake.

You just go to that level.

What about those of you who 
didn't do the logic model or

maybe like Andi, she has got 
one in her back pocket even

though it is not in the SPIL, 
but if you don't have one at all,

if you just didn't think what 
we said last year made any

sense and you didn't even think 
that way, what are you going to do?

Here is my suggestion.

Go back to your SPIL 
and rethink it.

Not rewrite it, don't 
get into that, my gosh.

But rethink it and try to 
put it as closely as

you can into something like this.

You've got outcomes in there.

I actually like to write one 
outcome on a sticky and

put it up on a wall and another 
outcome on a sticky and put

it up on the wall and then 
just start moving them around,

it is a really good way to 
do it, until it makes sense.

Then you are sort of 
recreating your logic 
model after the fact if you will.

Cause that will then tell 
you where to do your evaluation.

So that is what I would do.

So you did that, and now, 
if ever in this dreaded hour

I can really get you to focus, 
these are the next 15 minutes, okay.

Because you cannot do 
this leg of the stool

without measurable indicators.

You cannot do it.

There's a big difference between a 
measurable indicator and an objective.

It is a key step.

Let me go back.

Okay.
Take number five.

IL network key partners 
collaborate to provide services.

You know what I call that?

I call that a warm fuzzy.

Wouldn't that be nice?

I mean Mr. Rogers, 
oh put on my sweater,

wouldn't that be nice if all the 
IL key partners collaborated.

Because that's what it is.

It is a great desired outcome.

We absolutely want it.

What does it mean in reality?
You tell me.

How would we know if that's 
happening or not happening?

Tricky, tricky.

That's key here.

So here is what I say.
You probably heard me 
say this last year.

No one has ever measured 
progress on an objective.

None of us in this room 
today can measure

progress on an objective, 
even though that's

what this whole session 
is about, right.

And no one ever will measure 
progress on an objective.

It cannot be done.

You measure progress 
on indicators of that objective.

If you don't have good 
indicators you are

screwed to use blunt language here.

You need good indicators, 
we will work on this.

This is a key step.

What exactly should change, 
no more warm fuzzies here,

what exactly should change.

How exactly will we know if we are 
being successful on this objective?

Or I am a real Star Trek fan.

So, if you had two parallel
universes and you worked

in one and not the other 
how exactly would those

two parallel universes 
be different.

That's what I want to really 
push you on here, what exactly,

and you have to 
push yourself,

what exactly is going 
to be different.

That's the measurable indicators 
you have got to come up with.

Let's look at some.
Okay.

Now, you may not agree 
with these, that's okay.

It is the kind of thing
we are talking about.

If your warm fuzzy or if 
your objective is PWDs

have increased employment, 
perhaps, perhaps, one

measurable indicator is the 
percentage of PWDs desiring to work.

Because maybe you are 
saying it is not everybody

just those who want to work, 
who have at least half time

paid employment.

Why not full time, 
why not three quarters, 
why not 5%?

I don't know.

It is what we decided, 
you know.

Why just desiring to work?
Does it make sense?

All right.
Good challenge here.

Can you think of another one, 
a different indicator for PWDs

have increased 
employment, 
a different one?

Something we can actually 
measure, we can go out

there and get our hands 
around and actually measure.

No.
It's exactly what will be 
different in the world.

AUDIENCE MEMBER:  Income.

MICHAEL HENDRICKS:  Okay.

AUDIENCE MEMBER:  Income.

MICHAEL HENDRICKS:  So 
someone is saying income.

That's not good enough.
I'm sorry.

This is where I get to be task master.

You have to tell me, 
say you have hired me

to go measure your objective,
to measure this outcome,

this desired objective.

What do you want
me to go measure.

AUDIENCE MEMBER:  How many 
people have signed up for MBI.

MICHAEL HENDRICKS:  Okay.
But I am going to push you a little more.

What exactly do you want me to tell you.

AUDIENCE MEMBER:  How many 
people in my state have applied

for MBI, WD, Medicaid buy in
for people with disabilities

thus increasing the number 
of people with disabilities

who want to be employed.

MICHAEL HENDRICKS:  Keep 
the mic there.

Out of everybody in your state, 
how many people have done this?

AUDIENCE MEMBER:  We have numbers.

MICHAEL HENDRICKS:  No, no, no, 
this is a question of you.

You mean you want me to 
go see out of all the total

population of your state, 
every single person in

your state how many people 
have done this?  Yes.

AUDIENCE MEMBER:  We have stats, 
yes, there are stats.

MICHAEL HENDRICKS:  Or do 
you only care about the persons

who are disabled in your state.

AUDIENCE MEMBER:  This is about 
people with disabilities and employment.

MICHAEL HENDRICKS:  Ah, 
so it is percentage of persons

with disabilities in your state.

Is that what you are saying?

AUDIENCE MEMBER:  I am sorry.

MICHAEL HENDRICKS:  The numbers
of persons with disabilities,

I am trying to pin 
you down.

AUDIENCE MEMBER:  Yes.

MICHAEL HENDRICKS: Good.
I like that.

I like a woman who says yes,
always have.

[ laughter ]

So you are saying the number of 
persons with disabilities in your state.

AUDIENCE MEMBER:  In Ohio 
who have.  Applied.

MICHAEL HENDRICKS:  Applied 
for or actually received.

AUDIENCE MEMBER:  Who have 
become a part of.

MICHAEL HENDRICKS:  Ohh, 
that's different, isn't it, that is good.

Not just applied for, right.

It has become a part of,
there's a different; right.

AUDIENCE MEMBER:  Yes, yes.

MICHAEL HENDRICKS:  You 
are doing great.

AUDIENCE MEMBER:  The 
work force because they –

MICHAEL HENDRICKS:  That's 
good right there.

AUDIENCE MEMBER:  Okay.

MICHAEL HENDRICKS:  Like it, 
like it, like it.

Can somebody come up
with a different one,

a different indicator for PWDs 
have increased employment.

AUDIENCE MEMBER:  It's 
close to that.

She half stole my idea.

MICHAEL HENDRICKS:  That's all right.

AUDIENCE MEMBER:  A percentage 
of persons with disabilities who

have completed their 90 day 
work trial and are off

social security benefits.

MICHAEL HENDRICKS:  There you go, 
the Department of Labor,

you probably know this better 
than I do considers 90 days

to be the limit to say you've now 
had the job, you are now in the job.

So you are picking up on that and 
you are saying a slightly different.

It is a quite different indicator really 
but it is an excellent idea, good.

There's three possible 
indicators for this one objective.

Keep going.

AUDIENCE MEMBER:  Well, 
my first thought is the first

thing we would have to know 
once we chose what we were

going to measure would be

what it was before we started 
implementing our plan; right.

MICHAEL HENDRICKS:  No offense you 
are jumping a little bit ahead of us.

All I want you to think about right

now is exactly and I am 
emphasizing that word exactly,

exactly what change are 
you looking to effect.

AUDIENCE MEMBER:  The 
percentage of people who are employed.

So you would be looking at 
labor statistics of the number

of people with disabilities in the state 
who had at least half time employment.

MICHAEL HENDRICKS:  Okay.
So you are --

AUDIENCE MEMBER:  Labor statistics.

So you are comfortable with 
this one, that's what this one says.

Okay.

Can we add another one, 
we have another one right here,

can we get a mic?

I want to get you on tape.

This gentlemen to your left 
to your left.
There you go.

AUDIENCE MEMBER:  I want 
to go out and say diverse

community organizations 
have begun employment

training for persons 
with disabilities.

MICHAEL HENDRICKS:  Okay.

Buzz no offense, but let's talk 
about why that wouldn't work.

I am not saying that it is 
not a good thing to measure.

That is probably an excellent 
thing to measure but is that

an indicator of this objective, 
this objective says persons

with disabilities have 
increased employment period.

That's all it says.

And you're wanting, I think you 
are proposing that we measure

whether some other people have 
training programs or something.

AUDIENCE MEMBER:  Right, 
that is what I was going for.

MICHAEL HENDRICKS:  Now, 
I hear what you are saying,

that if other people have training 
programs, that is a good activity

that is probably going 
to help achieve this:

But that is not a measure 
of whether people have

increased employment, is it?

That's a measure of something 
completely different.

So I am really focusing here 
on a measure of increased

employment, and I think you've 
got like three or four of them already.

So, okay, want to try another one.

Formerly -- what we are getting 
at here, by the way, is the

difference between 
an objective and a

measurable indicator 
of that objective.

Let me say again 
this is absolutely critical.

This is the most critical 
step in this whole second leg.

If you can't come up with 
these measurable indicators

you are dead in the water.

It is really interesting 
how your language does

include things like dead 
and stuff, doesn't it?

It is interesting now that 
you start becoming aware of it.

You are not able to progress, 
how is that, you won't be able to progress.

Okay.
So formerly underserved 
populations are served.

Here is a couple of possibilities.
I think we saw these in SPILs.

One was the percent of PWDs who 
are deaf-blind who access services.

Now you might quibble on that and say 
what does it mean to access services.

You may want that a little more 
specific so you know exactly

what to measure, 
but still, okay.

Or the percent of Native American 
tribal communities having a

relationship with the 
IL network.

You might quibble a little 
bit and say what

does it mean to have 
a relationship?

But again we are 
getting close, right?

So you come up with 
a better one if you can.

Formerly underserved 
populations are served.

What would be a precise 
measurable indicator of that?

Come up with one.
There's no right and wrong answer.

This is like there could 
be a hundred possibilities.
Anybody got one?

Use the mic, please.

AUDIENCE MEMBER:  Is it like
the actual measure or –

MICHAEL HENDRICKS:  The actual 
measure.

The thing, I like to say if there's 
a stranger that you've hired to

go get this information for you, 
what do you want that stranger to 
go learn?

AUDIENCE MEMBER:  I guess
what I am asking, is it the

thermometer or the 
temperature you are asking for.

MICHAEL HENDRICKS:  ahh, 
the thermometer.

AUDIENCE MEMBER:  Thank you.

500 Native Americans 
received information

and referral services 
through the CILs of Maryland.

MICHAEL HENDRICKS:  Okay.
I am going to ask you, 
could everybody hear that?

It was a certain number, 
actually 500 Native

Americans received services, yes.

AUDIENCE MEMBER:  Correct.

MICHAEL HENDRICKS:  Okay.
I am going to suggest that you drop the 500.

Because that's what you are trying to learn.

That's what you are 
trying to go out and measure.

So that will be -- that will be 
a blank that will get filled in

once you do your data collection.

So I would suggest you put the number,
the indicator, the thing you want to

measure is the number of Native Americans 
who have received services in some

particular way, 
leave out the 500.

For now, leave out the 500.

AUDIENCE MEMBER:  Are you saying 
the number is going to increase.

MICHAEL HENDRICKS:  No,
what we are looking for here

is what exactly are you 
going to go out and measure.

You are saying I will go out 
and measure the number

of Native Americans who have 
accessed services in this particular way.

That's fine.
That's what you are going to go measure.

That's what we want to know.

AUDIENCE MEMBER:  I thought you 
were asking for the thermometer.

That seems like the temperature.

MICHAEL HENDRICKS:  Yeah, 
that's why I suggested leave the 500 off.

AUDIENCE MEMBER:  So after that 
statement maybe you could put like,

you know, SILC makes -- SILC 
outreach to the chamber of

commerce or to the mayor of the 
town or whatever, you know.

Or to the -- whatever -- I don't even know.

MICHAEL HENDRICKS:  No, not here.
Here we want two things.

We want to know the objective
and then we also want to

know how exactly, 
how did I put it here?

Lets see.

What exactly should change, 
how exactly will we know if

we are being successful?

What exactly should we 
measure to know our progress.

That's indicator.

AUDIENCE MEMBER:  So my 
example the indicator would

be the number of Native 
American individuals

who received services 
with the CILs of Maryland.

MICHAEL HENDRICKS:  Bingo.

AUDIENCE MEMBER:  That's what 
you want, that is the information

you want; right, that's the change 
in world you want to see; right.

Go measure it, okay.

AUDIENCE MEMBER:  Repeat that.

MICHAEL HENDRICKS:  I think 
she basically said the number 
of Native Americans.

AUDIENCE MEMBER:  The number of 
Native Americans who received,

who receive information and 
referral services with the

centers for independent 
living in Maryland.

MICHAEL HENDRICKS:  So that is –

AUDIENCE MEMBER:  I don't have 
to say the increase, just the number.

MICHAEL HENDRICKS:  Yes, 
that is her definition.

She's defining what she means by formerly 
underserved populations are served.

She's defining it as Native Americans 
use I&R services at CILs.

That's her definition of what 
she means there and that

is what she's going to go measure.

None of us can tell 
her it is right or wrong.

It is her definition.
That's fine.

It fits the idea we want.

Let's go on to the next one.

Here is the one that 
was really kind of fuzzy.

IL network key partners collaborate.

You know, what are we going to measure, 
how will we know if this is happening.

That's the question here.

How will we know if this is happening?

Well, number of agencies 
providing services to veterans

that collaborate with 
the IL network.

I think that is straight 
out of somebody's SPIL.

Maybe somebody in this room,
feel free to own it if it is

somebody's in this room.

So what they were looking for, 
they were going to go out and

count the number of agencies 
providing services to veterans

and notice that.

That's a particular kind of agency, 
not all of them but just the ones

providing services to veterans 
that collaborate with the IL network.

That's what they're going 
to go measure okay.

Now, are you with me, 
or are you lost?

Because if you are lost tell me.

I need to know.
You are struggling.

We have a gentleman who 
says he is struggling.

We are going to have more 
time to work on this.

It is good to know.

We have a question.

AUDIENCE MEMBER:  Hello.
I am new to the field.

MICHAEL HENDRICKS:  Welcome.

AUDIENCE MEMBER:  In Florida, 
and my question is whether or

not these indicators 
appear in the plan?

My interpretation or 
understanding is that 
they do not.

MICHAEL HENDRICKS:  I think they do.

AUDIENCE MEMBER:  They should 
be in the plan.

MICHAEL HENDRICKS:  I think 
they're supposed to be in there.

AUDIENCE MEMBER:  Where?

I put them in mine.

MICHAEL HENDRICKS:  Andi 
has them in hers.

AUDIENCE MEMBER:  They do 
go in the plan.

Okay, cause what, 
everything I was

hearing from you is that 
there shouldn't be numbers 
or percentages.

MICHAEL HENDRICKS:  We haven't 
said numbers or a percentages yet.

We have not said a single one.

We have just said the number 
or the percentage of each.

We have not put any 
numbers up here at all yet.

Notice there are no numbers on that slide, 
not a single number on that slide.

Darrell tells, go ahead Darrell.

Darrell has the answer 
to your question better than I do.

AUDIENCE MEMBER:  I am not 
sure I have the definitive answer here,

but we have heard from a number 
of the SILCs that this was one of

the most confusing parts of filling 
out the SPIL because there is no

separate window to 
put in your indicators.

So RSA has said to put 
them into the objectives section.

So in reality, they ended up 
being next to your objectives.

We have some ah-hahs back here.

MICHAEL HENDRICKS:  Okay.
Good.

Let me go on just in the interest 
of moving us along here.

Now, so that is one thing you did.

You developed objectives, 
you developed measurable

indicators and hopefully you 
designated a lead organization

to accomplish 
each objective.

We talked about this a little 
bit this morning in a different context.

We said that hopefully for each of the 
objectives you had in your SPIL,

you said that some group was 
going to have lead, not total,

probably others will be 
working with them too but

somebody was going to have 
lead for that particular objective.

Okay.
That's good.

That's another thing 
you have accomplished.

Now, that's the good news.
You've done a lot of work.

Here is the other half of the shoe is that 
there's still five things you've got to decide.

Five things you have to decide 
and you're home free really.

It is -- in fact you are downhill, 
once you've got those

indicators you are 
sliding downhill.

Let's talk about the five things.

First of all, from what source 
are you going to get the

information you need 
for each indicators?

In other words, where is 
or are the data you need?

Where are you going to go?

Here are some examples.

Are you going to go to written records?

Are you going to go to documents?

Are you going to go to data files, 
electronic files, you gonna

go directly to program 
participants?

Are you going go to 
others who know them 
like their family perhaps?

Are you going to 
go to the general public?

Are you going to go out and 
look at the physical

environment, like curb cuts 
or something.

What do you think answer is, 
which one you are going to do.

Two word answer I used earlier.

It depends, right.

What does it depend on?

What you're measuring, 
it depends on what

you are measuring, 
absolutely.

And what do we call 
what we are measuring?

The indicators.
Right.

So it depends completely 
on your indicator.

For one indicator you may 
need to go to some data file.

For another indicator you may 
need to go to program participants.

It depends on the indicator.

So, you've got to make 
some intelligent decisions here.

Well, look at that.
You already have got 
one of them decided.

The second one you 
have to decide,

what method you'll use to 
gather each person, 
each piece of information.

So for instance, if you go 
to a program participant,

how are you going 
to learn from them?

Or if you go to the general public, 
or the physical environment or

documents, how are you going
to get the information you want?

And there's some different ways 
like for instance, a couple of these

are kind of nerdy like record 
review, document review,

data file review.

But then it gets into some that are 
kind of really practical decisions.

Say you are going to go to 
program participants, okay.

Are you going to send 
them a questionnaire?

Are you going to 
interview them?

Hey, that's -- those are two 
very different ways to do it.

Which one are you going to choose?

It depends.
What's it depend on?

What you are measuring and 
what makes most sense, right.

It depends.

So here is one where I will 
make some of you mad at me.

Have you noticed another method I have 
up here, focus groups and I put not.

I strongly am opposed, 
how is that as strong as you can get.

I am strongly opposed to the 
use of focus groups for

evaluating progress on your objectives.

Well, tell me what the purpose 
of a focus group is.

Tell me what the strength 
of a focus group is.

We all know what a focus 
group is; right, half a dozen

or eight people or something; 
they are all yeah, uh-huh.

Tell me what the 
strength of a focus group is.

AUDIENCE MEMBER:  Would it be 
because it is not numbers, it is like opinions.

Well and so so said.

MICHAEL HENDRICKS:  No, 
that is not it.

AUDIENCE MEMBER:  Is it 
too small of a sample.

MICHAEL HENDRICKS:  No, 
that is not it.

AUDIENCE MEMBER:  Okay, 
fine, I give up.

MICHAEL HENDRICKS:  No,
no, this is good.

There's a particular reason 
focus groups were developed.

They didn't just sort of pop out of thin air.

There's a reason people 
developed focus groups,

there's a reason they 
use focus groups.

What do you think it is?

AUDIENCE MEMBER:  It is a consensus.

MICHAEL HENDRICKS:  A consensus.

The whole purpose of focus 
groups is to get people to

respond to each other to influence 
each whether it is stimulating ideas

or coming to consensus or whatever,
the whole purpose of a focus

group is that you do affect the
person sitting next to you.

All right, give you an example.

Let's say you are interviewing 
program participants, okay,

and you want to know if they know 
something they didn't know before

and you have eight of them, 
what do you care about?

Do you care about whether 
the eight collectively know

something more, or do 
you care about each of the eight?

How many of them actually 
know something more?

Which do you care about?
Yeah.

You don't care about 
the collectivity, you care

about each of the eight people.

You know you are hoping 
all eight learned something

but maybe just six did 
you just don't know.

When you do a focus group, 
it mushes it all together.

You lose the individuality.

My opinion, do not use 
focus groups for

assessing outcome data.

But there's also observation, 
there's testing,

there's mechanical measurement.

There's a variety of methods 
to use to get this evaluation

information about your objectives.

Next one, which organization 
will be responsible for gathering

each piece of 
information?

You remember earlier we had said 
somebody was the lead organization.

Should it be that person 
or that group I am sorry.

Should it be a different IL partner 
in the big circle or the little circle?

Should it be another organization outside?

So this gets to the point of the question of 
who is actually going to do the work.

You know, who is actually 
going to be the one to go out

there and do those 
interviews with people,

or look in those 
files or whatever.

You've got to decide that.

AUDIENCE MEMBER:  Mike.

MICHAEL HENDRICKS:  Yes.

AUDIENCE MEMBER:  While you are 
taking polls can I ask a question.

MICHAEL HENDRICKS:  Ask anything.

AUDIENCE MEMBER:  Do you 
guys have other organizations

that compile this 
information for you?

Cause I can't imagine in my 
state another, a partner

organization being willing to 
do this part of this for me.

MICHAEL HENDRICKS:  The 
CILs don't collect data?

AUDIENCE MEMBER:  The CILs 
will provide some of the information

that's compiled but they're 
not going to gather it all.

They will send it to me if I ask for it.

They compile it from their consumers.
They compile it but they don't gather it.

Do you see what I am saying?

They gather their piece of it 
but if I want to know what all

of my centers are doing,
it is up to me to ask them

to send me that information

and then I have to put it together.

I was just curious if there 
is anybody else out here

that has got organizations that are 
willing to do that level because I sure don't.

So you contract?

She says she hires 
somebody to do that.

The SILC has to do it in Michigan.
They do it in Wisconsin.

I was just curious because 
that would be a new one on me.

MICHAEL HENDRICKS:  For you it 
sounds like which organization

will be responsible for gathering is 
going to be a very interesting question.

You are going to have to wrestle with that.

You are going to have to wrestle 
with that question.

AUDIENCE MEMBER:  We wrote 
into our SPIL that the, what the

pieces of information would be 
and that they would come directly

to the SILC from the consumers.

MICHAEL HENDRICKS:  Okay.
Interesting.

So your -- you have built into 
your SPIL not only -- let me

back up here a little bit.

You have built into your 
SPIL that for at least some

of your objectives, 
the source  -- or the program

participants, the method is 
what a questionnaire or something?

So you've also built in which 
organization is responsible

for gathering it although it 
is not really an organization,

It is the individual people.

You are kind of bypassing the
organization and you are

having it coming straight to you, 
which leads directly to the next

question, which organizations will 
receive each piece of information.

You know not only who's going to 
gather it but where it is going to go to.

For instance I think it is good ideally if all of 
the IL partners get all the information.

That's part of collaboration right?

Any reasons not to do that I 
will be curious what you think.

And then how will they 
use this information?

So you've bypassed that.

And please don't underestimate 
the importance of this question

because receiving information 
makes you a partner in the process.

Last question and then we will see 
what you have got for discussion here.

The last question you have to wrestle 
with is how often you will gather each 
piece of information.

Do you think you are going to 
need it monthly or quarterly

or twice annually or annually 
and I have given you a tip off.

What's the answer?

It depends, that's right.

It depends on the indicator 
and it depends on your needs.

There's no simple answer.

I know I have thrown a lot at you, in a 
second it will be more clear.

Let's see if I can go to the next page.
Yeah, here we go, let's pull it all together.

So this table captures 
everything we just talked about.

Now I love this table, you are going 
to have if you don't already you are

going to have a copy of this on 
your table and look what it has got.

It has got the different objectives in 
that first column down the left,

you know objective one, 
objective two, whatever they are.

So what this table says is that for 
each one of those objectives

separately, see we are back to
that objective by objective by

objective by objective, just 
like we said we would be all day.

You can't look at your SPIL 
as a whole, you have to look

at it objective by objective
by objective.

And here is what you need 
to figure out for each objective.

What is the measurable indicator?

What's the source of information 
needed for that indicator?

What's the method to gather each 
piece of information from that source?

Who's the responsible organization 
that's going to gather it?

What organizations are going to receive it?

And how often is it needed?

So it is all summarized right here, 
those decisions you've had to make.

All right, let me stop, 
see what's on your mind.
I will bet there's something.

I know, I know especially if you 
are not a research nerd like me, 
there's probably a lot on your mind.

Come on.

AUDIENCE MEMBER:  I heard
the pin drop.

MICHAEL HENDRICKS:  Nobody 
has anything on your mind

about this, they are perfectly 
happy with doing this tomorrow.

Ready to go.

AUDIENCE MEMBER:  I have got one.

To be clear, you only need indicators 
for your objective level; correct?

MICHAEL HENDRICKS:  Yes.

AUDIENCE MEMBER:  Okay.

So how do you measure whether or 
not your objectives are achieving your goals?

MICHAEL HENDRICKS:  You don't.
The question was at this point you don't.

The question was if you only need 
measurable indicators at the

objective level, let me try to go 
back if I can get that far back.

If you only have measurable 
indicators at your objective level,

then how do you know that you're 
achieving your goals or your missions?

And the answer is you don't.
At this point at least.

Let's master the ability to 
measure our objective levels

first which is going to take 
us probably a year or so.

Let's master that ability first 
and then we will wrestle

later if that's okay, because this 
is a big enough bite may I suggest.

This is the bite that RSA 
is asking us to chew on.

So we are fine with
them to do this.

So good question.
Others?
All right.

I don't know how much this has 
been crystal clear versus how

much it is that hour right after lunch 
but since I am going to put you

to work anyway, we will find out.

Let's go back so I was 
just suggesting that

you put your decisions into this table.

Let's look at an example.

Help me with this example, 
see if this makes sense for you.

Here are those seven things 
we were just talking about.

All right.

If this SPIL objective was that 
persons with disabilities have

increased employment, 
then one possible indicator

or measurable indicator we 
call them of that could be the

percent of persons with disabilities 
desiring to work, not all PWDs but

just those desiring to work who 
have at least half time paid employment.

That's what we would 
want to go out and measure.

Okay.
Where do we go?

Where do we go get 
that information?

I put two possibilities up here.

One of them, we can go 
directly to the PWDs; right.

We can do a sample of them or all of them.

The other option is computer data files, 
maybe, what was the name

of the program, I am sorry you 
were talking about in Maryland?

Somebody was saying.

AUDIENCE MEMBER:  She 
was talking about the Medicaid 
buy in for workers with disabilities.

It is different in some places, 
but that's what it is.

It is an employment enhanced 
for people on Medicaid.

MICHAEL HENDRICKS:  Okay.
You see my thin level of knowledge

about IL is exposed here, 
but thank you for that.

And so what you could say is that 
let's go to those computer data files,

we don't need to do the work of 
interviewing people or going

to people, let's go to the
computer data files.

So there's two possibilities, 
if those are the sources,

what are the methods?

Well, let's see for the persons 
with disabilities I am saying

probably the more intelligent 
method might be a phone

interview, you could send 
them a questionnaire you

could go visit them in person 
but maybe a phone interview.

And for the computer 
data files, perhaps a

good method is data file matching, okay.

You want to match your list of people with 
some other list and see how many are there.

Who might be the organization gathering it?

Maybe you hire a consultant or 
maybe the DSU or maybe

based on what you've been telling 
me maybe not the DSU, perhaps not.

I don't know.

Who should receive the information back?

The SILC obviously,  the DSU, the 
CILs obviously, and how often?

I don't know, I just made this up.

I said every three years.

But for you it might be every 
year or every six months,

you know, I just put 
that in there.

Seem reasonable to you?
At least reasonable possibility?

Let me ask you this.

If you did all of this would 
you come back with information

on how well you're 
progressing on that objective?

Yes, you would, wouldn't you?

This would give you information on how 
well you're progressing on that objective.

This would give you information for 
that second leg of the stool, okay.

Now, you have to do this same 
thing for each of your objectives.

I want to keep saying that 
for each of your objectives.

So how about an example, 
this is your turn?

Here is the objective.

Persons with disabilities 
can access transportation.

First thing we have to decide 
is the measurable indicator.

What's it going to be?

This is the thing we are 
going measure, Andi

I am going to ask you not 
to if you don't mind.

This is the thing we will 
measure to tell us

whether we are having progress on 
PWDs can access transportation.

What exactly do we want to measure?

AUDIENCE MEMBER:  The 
number of public buses or

public transportation that has 
accessible technology,

like kneeling buses.

MICHAEL HENDRICKS:  Okay.
Let me ask you this.

Is that an indicator 
that they can -- well, okay.

AUDIENCE MEMBER:  Yeah because you 
can't access it if it is not accessible.

MICHAEL HENDRICKS:  Yeah, 
no I started to call you on that 
but I think I am wrong.

AUDIENCE MEMBER:  Got you.

MICHAEL HENDRICKS:  You got me 
because it doesn't say

access transportation, does it.

It doesn't say PWDs access transportation.

It says PWDs can access transportation.

My friend, I think you got me.

AUDIENCE MEMBER:  I have a 
problem with that.

MICHAEL HENDRICKS:  Okay, 
he has a problem with that.

AUDIENCE MEMBER:  I have 
a problem with that one

because it doesn't say 
anything to me.

It just says people are able to, 
if one person with a disability

can access transportation, 
then you've done your job,

it is a bunch of fluff that doesn't 
say anything as far as I'm concerned.

MICHAEL HENDRICKS:  I will push you 
because this is a great time to take this apart.

Are you objecting to his indicator of the 
objective or are you objecting to the 
objective itself?

The objective it says PWDs 
can access transportation.

Is that your concern?
Yes.  I think it is too.

I think that is what you are 
saying you don't like this

objective because it doesn't 
really mean that anybody is

actually out there 
doing it.

AUDIENCE MEMBER:  It doesn't 
say anything really.

If one person in the state 
can get on a bus, then you've

done your job according to this wording.

MICHAEL HENDRICKS:  Yes, 
according to this.

AUDIENCE MEMBER:  That's 
not what we are supposed to be doing.

MICHAEL HENDRICKS:  Okay.

Well, now see that's where, 
that's where I would never

second guess your objectives 
because you've got to decide

in your own state what your 
objectives ought to be.

If you have a state where access 
to transportation is awful and no

matter how many people want to
get on a bus, none of them can,

then this might be a good objective.

If you have -- I am -- this is my opinion.

If you have a state though where
actually accessing them is fine,

the physical accessing is fine, 
but there's some other reasons

why they're not, then you 
probably have a very good

point of how many are 
actually doing it.

I don't know, I am always 
very hesitant to say it is

a bad objective because I don't 
know your situation, but this

gentlemen making an excellent 
point that if you take the word

can out of that it changes that 
objective very much, doesn't it?

AUDIENCE MEMBER:  This is 
Jesus Garcia, I think that we

need to go in a little deeper 
because this may be true in

other places too, but I will stay 
with transportation since this

where it all began right now.

When people say to me and 
especially in areas of Florida

in the north in the panhandle, 
we have a horrible transportation

system, I always say no you don't.

You have a wonderful transportation 
system you have i-10, i-75.

What you are saying to me is if I don't drive, 
I have a terrible transportation system.

That is true, but if you drive, 
if you have access to a vehicle

you have a wonderful 
transportation system.

MICHAEL HENDRICKS:  So Jesus 
might be -- if this were his objective,

he might tweak it to say PWDs who 
do not drive can access transportation

and that would be 
quite legitimate too.

He is -- remember we talked 
about starting off with the

target group, he's narrowing 
down the target group.

He is saying my target group
is the people who don't drive.

That's good thinking in my mind.

He is being very clear on 
who he is trying to affect.

AUDIENCE MEMBER:  This is 
Roberta from Minnesota.

Is that what we are 
going to have for our indicator?

I was going to do an indicator 
but I am not exactly clear on

the objective but maybe
it will work for any of the

objectives we come up with.

MICHAEL HENDRICKS:  Just for the 
exercise, let's keep this objective.

AUDIENCE MEMBER:  The indicator,

could an indicator be the number of people 
with disabilities using public transportation?

MICHAEL HENDRICKS:  Well, it 
is a good question.
You tell me.

Would that been an indicator 
that they can access it or

would that be an indicator 
that they access it?

AUDIENCE MEMBER:  Well, 
if they access it then they can.

MICHAEL HENDRICKS:  That's true, 
that's true.
That's a tough one.

You know that's on the line.
I am not sure.

AUDIENCE MEMBER:  Because
I know that for, for us the,

as accessibility of the buses of the 
paratransit increases then

obviously there's an 
increase in the number of users.

So that's one way to judge 
whether it is effective.

MICHAEL HENDRICKS:  Well, yes, 
but let me give you an example

that you may 
have run into too.

You realize, of course, 
that opening women's

shelters increases the incidence 
of domestic violence.

So therefore we should close 
all the women's shelters

because they're 
counterproductive

and the more womens 
shelters we open the

more domestic violence there is.

Because the indicator is clear, 
when you have more women's

shelters you will have more 
reported domestic violence.

It will go up.

You've got to be a 
little careful some times.

AUDIENCE MEMBER:  Because 
you have a set number of people

with disabilities and you have a 
set number of women who are 
being victims of violence.

While the people with disabilities 
either can or cannot access

transportation, whether or not it is 
available, so by measuring the

number of people who 
are accessing it you

are measuring whether it is available.

Just like by measuring the number 
of people accessing shelters.

I mean if they're not available, 
women can't access them.

MICHAEL HENDRICKS:  Absolutely.

AUDIENCE MEMBER:  So I don't know.

MICHAEL HENDRICKS:  But, if --

We are probably getting too picky here for 
the level we are working at, but yeah, okay.

We are getting -- I think we 
are getting too picky.

So thank you, that's good.
Yeah.

AUDIENCE MEMBER:  How about 
counting seeing if there is a

discounted rate for persons with 
disabilities for a bus and then

counting up or counting the number 
of disability passes the discounts

that have been purchased or 
sold that, that year or the next

year or the next year?

MICHAEL HENDRICKS:  Okay.
So this is a really interesting point.

So you are saying your definition, 
of being able to access

transportation, is kind 
of a financial one?

AUDIENCE MEMBER:  Well, 
that and special discounts.

You can't access it if you don't 
have buses that provide that

disability services, but yeah 
we are on the same page.

MICHAEL HENDRICKS:  But you 
are saying one dimension of it,

at least for you is 
financial.

AUDIENCE MEMBER:  Yes, it is.

MICHAEL HENDRICKS:  So by your 
indicator you are defining and saying

I want to measure the financial accessibility.

AUDIENCE MEMBER:  Right.

MICHAEL HENDRICKS:  That's good.
Yep.

MICHAEL HENDRICKS:  In fact 
may I say to really make it even

a little more complicated you might 
end up with two indicators of this.

You might say, one indicator is the 
percentage of buses that have

kneeling what are they called kneeling 
buses and a second indicator might

be the number of people 
who have bought passes.

You might be interested in both of them.

AUDIENCE MEMBER:  There's 
a third one.

If the relative salaries are really low, 
in that community or in that area,

that you could say, you know, 
the financial aspect comes in

because you can't access 
if you can't pay for it.

So you can measure the 
number of disability discount

passes that are sent out or that are 
purchased as a third indicator.

MICHAEL HENDRICKS:  I will say 
just in the interest of time unless

somebody has a burning question
they must ask at this moment,

let's take one of those 
and work through it.

Let's take the kneeling 
buses and quickly work through it.

So if the indicator is the number 
of kneeling buses what might 
be a source of that information.

AUDIENCE MEMBER:  Transportation 
department.

MICHAEL HENDRICKS:  The 
transportation department.
What else?

I want to push you to think 
of options because sometimes 
another option is easier for you.

What else?
Come on.

This is an obvious one.
It is so obvious you are overlooking it.

The buses.
The buses themselves.

So you could -- one source could 
be the transportation department,

another source could be 
the buses themselves.

If it is the transportation department, 
what method might you use to

get the information from them?

Somebody said interview.
Possibly.

Might there be there an easier way.

Statistics or phone call.
Might there be an easier way.

Web site, perhaps the 
web site has that information.

Do you think they might put 
out an annual report and you

could just look in the annual 
report that might have it.

A lot easier than interviewing right, 
looking on the web site or the

annual report; right.

Okay, if instead of the transportation 
department, the source was the

buses themselves, what would be

the method you would use 
to get the information.

AUDIENCE MEMBER:  I would get my 
information from the local transit authority.

MICHAEL HENDRICKS:  All right.

That's the first thing we were 
working with the transit authority.

What if your source was the bus, 
the buses themselves,

what method would you use.

Counting, yeah, but that's -- we
got have to have is a fancier term than that.

Get back to this page.

I love you.
Yes, what method would you use.

Observation of the buses or 
you could use a document

review of the transit authority, 
both might give you that piece 
of information.

Who is going -- here is a tough 
one, who's going to get this information?

Who's going to get this information?

The SILC you think?
There's a hand in the back.

So the SILC is a possibility.
Who else might get it.

You could ask the centers to 
do something in your local area.

Microphone, please.

AUDIENCE MEMBER:  A lot of people 
like myself, as a consumer of the

center for independent living,

like where I am, we have 
consumers on the local transit board.

We use her as for accessibility,
the -- for anything with the ADA.

They have to be in 
compliance with the ADA.

So consumers like myself or board 
members from SILCs, we integrate

with the disability organizations or 
centers to -- we even --

myself I go to the board meetings, 
open to the public.

If we need anything whatsoever 
I have a good relationship.

I used local buses.

I want to tell you I went to ADA 
update in Baltimore, the

Department of Transportation 
is one of the best resources,

but if you used the local transit, 
they're more than happy to

help you get any information out there.

That's your best source.

MICHAEL HENDRICKS:  Okay. 
The Department of Transportation.

So, who is going to gather it?

AUDIENCE MEMBER:  Yes, sorry.
I have a question about the source

when you 
said the buses --

MICHAEL HENDRICKS:  Hang on, let me 
go back.

Let me go back.
Yes.

AUDIENCE MEMBER:  Okay.
So when you said the buses,

so you are observing which buses 
have what sort of accessible

technology on them; correct?

MICHAEL HENDRICKS:  Well, if I 
understand the indicator correctly,

it is going to be the 
number and percentage

of buses in the area that are kneeling 
buses which I would put under,

I would call that part of 
the physical environment.

You are going to go out there and 
look at the physical environment.

That's what I would say.

AUDIENCE MEMBER:  How could you 
begin to look at all of the buses or know

I mean it would be really hard to 
keep track of which buses you

have looked at and whether 
or not this bus is actually the

same bus you saw three weeks 
ago and whether or not it is 
a statistically relevant sample.

I guess I am really confused 
why that's a good idea and

you have been saying this one 
is not such a good idea and this one is.

So we are continuing 
to work with this one.

It leads me to believe that you

either think it is a good idea or you 
think it is a bad idea and we

should continue doing 
it so we can see why.

MICHAEL HENDRICKS:  No, I think 
the answer is it depends.

I live in an area, a rural 
area with three buses.

I can easily go down and count 
how many of them are kneeling.

It sounds like you don't.

It sounds like you live in an area 
where it would be pretty darn hard to do that.

I think it depends on 
the situation, you know.

AUDIENCE MEMBER:  I think 
your question was who's going

to get this information; is that correct.

MICHAEL HENDRICKS:  Yeah, let's go 
back.
Yes.

Who's going receive the information.

AUDIENCE MEMBER:  In collaboration.

MICHAEL HENDRICKS:  I am sorry, 
who is going to gather it.

AUDIENCE MEMBER:  Who's going to 
gather it.

MICHAEL HENDRICKS:  Who is going to 
gather this information.

AUDIENCE MEMBER:  In Nebraska 
I would say the IL partners are,

and that would be the SILC as well 
as other, the partners, the SILC

may be actually compiling the report 
but it would be the IL partners who

are getting that information, 
gathering it.

MICHAEL HENDRICKS:  Okay, so 
the IL partners would actually 
be gathering it,

Okay.
All right.
We had one more here.

AUDIENCE MEMBER:  You had 
mentioned about kneeling bus,

how would we find out,
the word kneeling bus

usually is on that bus when the door opens.

So I can't say it is a federal law that they 
have to put it on there because I don't know,

but I know most buses I ride and I ride 
Ohio, Kentucky, West Virginia, Baltimore,

and every bus I have gotten on 
said kneeling bus.

MICHAEL HENDRICKS:  That's good.

AUDIENCE MEMBER:  A lot of 
new buses now are in

compliance with the ADA.
Of course.

MICHAEL HENDRICKS:  Okay, good.

Last question, how often do 
we need this information?

Think about it.
You want it so you can use it.

How often do you need this information?

Somebody says annually.

What do you mean by continually.

AUDIENCE MEMBER:  You are always 
gathering that information.

MICHAEL HENDRICKS:  So how often 
would you –

AUDIENCE MEMBER:  as often as you 
possibly can.

MICHAEL HENDRICKS:  That is a very 
different attitude isn't it.

One person says annually, 
one person says probably

like weekly check of the 
web site or something.

Okay.
Both are quite legitimate.

AUDIENCE MEMBER:  First it depends.

MICHAEL HENDRICKS:  It depends.  
What's it depend on.

AUDIENCE MEMBER:  It depends 
on what you are going to be doing

with the information.

MICHAEL HENDRICKS:  Bingo

AUDIENCE MEMBER:  What we 
are doing in Nebraska is have

quarterly reports so we would need 
to collect that information quarterly.

MICHAEL HENDRICKS:  There 
you go, excellent, excellent.
All right.

We will take -- we have one more 
and then I think it is break time.

And when we come back, 
it is your turn. [ laughter ]

Was there one more or not?
Maybe not.

Yeah, over here is one.

AUDIENCE MEMBER:  My question is 
about --

MICHAEL HENDRICKS:  Let's listen
 to our friends question please.

AUDIENCE MEMBER:  Evaluating 
independent intervening causes.

MICHAEL HENDRICKS:  Ohh, 
you are a nerd like me.

AUDIENCE MEMBER:  And 
how do you factor that in.

MICHAEL HENDRICKS:  At this point, 
the question was what do you do

about evaluating intervening
independent causes.

AUDIENCE MEMBER:  Independent 
intervening causes.

MICHAEL HENDRICKS:  Let me see 
if I can describe what I think you are saying.

Actually I have to ask you 
a question about that.

Are you talking about their 
intervening here between the

activities and the objectives, 
or their intervening above?

AUDIENCE MEMBER:  In between 
the activities and the objectives.

For example increasing 
employment in people with disabilities.

The independent intervening causes 
can be the economy, the job market,

so that if you were in a deteriorating 
job market, if your goal, if your

objective was to increase the 
number of people with disabilities

become employees you can be 
doing all of the right

things and getting wrong results.

If you're -- if your objective was
to increase ridership of public

transportation and you did 
that in a declining market,

where people were all losing 
their car and had to use public

transportation, you could be 
doing all of the wrong things

and getting the 
right results.

MICHAEL HENDRICKS:  So everybody 
heard what he said I hope because it

was a very thoughtful, very insightful 
point of does the fact that we are

doing good or our indicators are 
looking good mean that actually

good things are happening or other 
stuff going on or whatever.

And the answer is that it ties back 
to what I mentioned earlier today,

the data do not 
speak for themselves.

They must be discussed 
and interpreted within the

context of whatever it is you are 
talking about at the time.

You are exactly right.

The problem with too many 
outcome performance systems

is it allows no place for you to 
add that kind of interpretation 
and discussion.

But I think we must have that.
So very thoughtful.

Yes?
The lady here.

AUDIENCE MEMBER:  Yeah, Denise 
Figeroa.

My thought was that wouldn't that 
really be after you have done your

evaluation that then when you really
sit down and sit and analyze what

you collected, that is where that
discussion comes in and you

determine whether or not the
reason that you haven't been

successful is because 
of these outside factors.

And then, as you're developing 
your SPIL for the next year you

use that information to 
decide whether it makes

sense to continue to 
pursue that or whether

you need to change 
what you are doing, or if,

you know, maybe the 
economy has changed

and now it is 
okay, you know.

So it just seems to me that 
is after all of this is done.

MICHAEL HENDRICKS:  There are 
at least two different ways to

handle that situation.

One is up front with a, a more
sophisticated data collection design.

Let's not go there, okay.

Let's bite off this part and 
get it before we go there.

The second part is exactly what you 
are saying, collect the data just

as we are talking here and then

as you sit with people and discuss 
it and interpret it then let's bring

that information in.

That would be my strong suggestion, 
let's not overwhelm ourselves.

I am already asking you to do stuff 
I think you haven't been doing before.

Let's not do too much here.

But yes, I think at this point 
you have got it exactly correct.

Thank you.
Here is one.

AUDIENCE MEMBER:  What
would be your recommendation

on the number, the average 
number of measurable indicators?

Back home we are 
all very what's the word?

There's a lot of looking back on our plan, 
there's a lot of measurable indicators

and I am thinking whoa we could 
have just done a couple and people

would have been happy.

But as an aside to that, 
it was the whole arena of

home and community based 
services that the objective was

centered around that and we 
developed four measurable

indicators to go along with that.

But there's a lot of levels of home 
and community based services.

So we were trying to encompass some of 
those but yes we do have four for that one.

Is that too many?

MICHAEL HENDRICKS:  Whenever 
I am asked that question I am reminded

of what Abraham Lincoln said once.

Abraham Lincoln was a little bit taller than 
me which was quite tall for his day.

He had long legs I guess and 
somebody said to him once

how long do you think a 
man's legs ought to be.

His answer was I reckon they ought 
to be able to reach the ground.

I think, I reckon you ought have as 
many indicators as you need to

do the improvement you want.

So it is really hard to say.
I can say this.

It is easy to overwhelm a system with too 
many indicators, really, really easy.

So if you have got 40, hmmmmm, 
and you are committed to it then

I can't help you, but if you are not 
fully committed to it you might

want to rethink all 40 of them perhaps.

But even as I say that, you saw here, 
some of them might be as simple

as going to a web site, 
just go to a web site and

look in their annual report.

That is easy.
That depends.

A lot of this depends, doesn't it.