ROUGH DRAFT 9-14-11, Outcome Measures for Centers for Independent 
Living – An IL NET Resource Presented by ILRU

>> Mike Hendricks: , GOOD
MORNING, EVERYBODY, IT'S NOT
9:00, DON'T WORRY, ONLY QUARTER
TILL.
BECAUSE OF OUR ONLINE FRIENDS WE
WILL BE STARTING RIGHT AT 9:00,
JUST FACTOR THAT INTO YOUR
EATING AND MOVING AND THINGS
LIKE THAT.
>> GOOD MORNING, EVERYONE.
>> GOOD MORNING!
>> LET'S GET STARTED.
FOR THOSE OF YOU IN THE ROOM, I
HOPE EVERYONE'S BEEN ENJOYING
THIS MAGICAL CITY OF PORTLAND
WHERE THEY HAVE WATER FOUNTAINS
AT EVERY CORNER AND ALL SORTS OF
EXCITING THINGS TO DO.
I HOPE THIS IS GOOD FOR YOU.
AND FOR THOSE WHO ARE WATCHING
BY OUR LIVE WEBCAST, WE HOPE
THAT IS GOING WELL FOR YOU.
YESTERDAY WE HAD A FAILURE OF
OUR REALTIME CAPTIONING, AND IN
THE FLURRY OF ACTIVITY OF TRYING
TO GET EVERYTHING COORDINATED
FOR THIS NEW TECHNOLOGY, WE HEAR                   THE 
MANAGEMENT TEAM DID NOT
REALIZE THAT REALTIME CAPTIONING
WAS NOT BEING FED OUT TO THOSE
WHO WERE VIEWING THE VIDEO, AND
THAT IS A PROFOUND FAILURE.
WE DEEPLY REGRET THAT, AND WE
HAVE FIXED THAT.
THERE SHOULD BE REALTIME
CAPTIONING GOING NOW.
IF WE HAD REALIZED THAT, WE
WOULD HAVE PULLED THE VIDEO
YESTERDAY UNTIL EVERYONE COULD
HAVE PARTICIPATED EQUALLY, AND
WE WILL DO THAT IF THAT OCCURS
AGAIN.
BUT WE BELIEVE NOW THAT WE DO
HAVE CAPTIONING.
IN ADDITION, WE HAVE AN ADDED
FEATURE THAT IS WORKING FOR
THOSE WHO WOULD LIKE TO FOLLOW
THE POWERPOINTS IN REALTIME.
THERE IS A LINK FOR THOSE IN THE
ROOM TO CONNECT TO NETWORKS,
THERE'S A NETWORK CALLED OUTCOME
MEASURES.
THAT'S FOR THOSE OF YOU ONLY IN
THE ROOM, CONNECT TO THE NETWORK
FOR OUTCOME MEASURES AND YOU CAN
FOLLOW THE POWERPOINTS IN
REALTIME AND IN ACCESSIBLE
ALTERNATE FORMAT.
FOR THOSE OF YOU WHO ARE VIEWING
THE VIDEO, THERE IS A LINK ON
THE WEB PAGE FOR EACH
PRESENTATION, AND YOU SHOULD
FIND IT FOR REALTIME STREAMING
OF THE POWERPOINT.
AND OF COURSE THERE IS ALSO A
COPY OF THE POWERPOINT ON THE
SCREEN THAT YOU CAN USE IF YOU'D
LIKE TO DO IT THAT WAY.
SO THAT'S EVERYTHING.
AGAIN, WELCOME, AND WELCOME TO
OUR TRAINERS.
WE ARE READY FOR ANOTHER PACKED
DAY OF INFORMATION ON OUTCOME
MEASURES, AND I WILL NOW LEAVE
BOB AND MIKE TO THE
PRESENTATION.
SO THANK YOU ALL.                                                     3
>> Mike Hendricks: THANK YOU,
RICHARD.
GOOD MORNING, EVERYBODY, AGAIN.
>> GOOD MORNING!
>> EVERYBODY HERE HAD A
WONDERFUL, WONDERFUL DINNER, AT
LEAST I HAD DINNER AT THE
RECEPTION LAST NIGHT, I DON'T
KNOW ABOUT YOU BUT IT WAS REALLY
NICE OF THEM TO PUT ON A
WONDERFUL RECEPTION.
HOPE THOSE OF YOU ONLINE HAD AN
EQUALLY NICE DINNER.
COULDN'T HAVE BEEN QUITE AS
NICE, COULD IT?
>> I WANT TO THANK ALL OF YOU
STARTING OFF WITH FOR THE NICE
SUGGESTIONS YESTERDAY.
WE ASKED TO YOU WRITE DOWN SOME
THINGS WE COULD DO A LITTLE BIT
BETTER, AND WE WERE VERY HAPPY
TO READ THESE.
YOU MADE SOME REALLY GOOD
SUGGESTIONS AND SOME NIGHT
COMPLIMENTS, TOO WHICH WE DIDN'T
ASK FOR BUT WE'LL CERTAINLY
TAKE, SO THANK YOU.
THE SUGGESTIONS, I WAS
WONDERING, ARE THEY ALL GOING TO
FALL INTO ONE CATEGORY?
LIKE 10 ARE GOING TO SAY YOU'RE
NOT DOING THIS VERY WELL AND YOU
NEED TO DO IT BETTER TODAY.
THAT DIDN'T HAPPEN.
YOU MADE DIFFERENT SUGGESTIONS
SO THEY DIDN'T REALLY FALL INTO
ONE CATEGORY.
WE WILL TRY TO ADDRESS EVERY
SINGLE ONE OF THEM AT DIFFERENT
TIMES AS WE GO, SO WE THANK YOU
FOR THEM.
I WANT TO LET YOU KNOW, TOMORROW
THE LAST SESSION IS CALLING
PULLING IT ALL TOGETHER.
IT'S AN HOUR AND 15 MINUTES, AND
WE'VE DELIBERATELY, FROM THE
VERY BEGINNING, NOT SCHEDULED
ANYTHING AT ALL FOR THAT
SESSION.
IT WAS ALWAYS INTENDED TO BE                                          4
WHATEVER CAME UP OVER THE FIRST
COUPLE DAYS THAT SEEMED TO BE ON
YOUR MIND, SEEMED TO BE A
STICKING POINT, SEEMED TO BE
SOMETHING THAT WOULD BE USEFUL
TO FOLLOW UP.
IN OTHER WORDS, BUILT-IN FREE
TIME TO COVER WHATEVER MADE MORE
SENSE FOR ANYBODY.
SO WE'RE GOING DO THAT TOMORROW.
WE WILL STILL DO IT, COVER SOME
OF THESE THEN, COVER SOME OF
THEM BETWEEN NOW AND THEN.
I SAY THAT SO THAT IF YOU HAVE
OTHER THINGS THAT YOU DIDN'T
WRITE DOWN, OR YOU WANT TO
E-MAIL IN TO US, THAT YOU WILL
REALLY LIKE TO MAKE SURE GETS
COVERED DURING THAT PULLING IT
ALL TOGETHER SESSION, JUST DO
THAT.
IT'S REALLY YOUR SESSION.
AND THIS WOULD BE GOOD FEEDBACK
AND SORT OF THE GRIST FOR THAT
MILL, IF YOU WILL.
AGAIN, THANKS FOR GIVING US
THOSE.
SO WHERE ARE WE TODAY?
FIRST OF ALL, WE HAVE AN
EXCITING LUNCHTIME, WE'RE HAVING
THE COMMISSIONER OF RSA, I THINK
SHE'S STILL COMING AND SHE WILL
BE SPEAKING TO US AT LUNCH.
IT'LL BE VERY EXCITING TO HEAR
HER TAKE ON OUTCOME MEASURES AND
THINGS LIKE THAT.
I'M REALLY LOOKING FORWARD TO
THAT.
BEFORE THAT, A LITTLE BIT MORE
ON THE YELLOW BRICK ROAD TO GO
DOWN.
ONE OF THE GOOD SUGGESTIONS YOU
MADE, LET'S MAKE SURE WE'RE
ALWAYS WHERE WE ARE, WE ALL KNOW
WHERE WE ARE IN THE PROCESS.
SO LET'S JUST REVIEW, IF WE CAN,
WHERE WE ARE IN THE PROCESS.
WE STARTED WITH OUTCOMES AND
OUTCOMES MANAGEMENT, MAKING SURE
WE KNEW WHAT AN OUTCOME WAS AND                                       5
WHY WE'RE MEASURING THEM AND
MANAGING THEM.
THEN WE TALKED ABOUT LOGIC
MODELS AND THE VALUE OF MODELS
TO UNDERSTAND YOUR PROGRAM.
AND TO ASK YOURSELF, DOES MY
PROGRAM REALLY HAVE GOOD LOGIC
BEHIND IT?
IS IT MAKING SENSE?
DO I NEED TO RETHINK IT?
THEN WE REALIZED WE CAN'T
PROBABLY MEASURE ALL OF THOSE
OUTCOMES ON THE LOGIC MODELS SO
WE HAVE TO MAKE SOME TOUGH
CHOICES.
WE SAW HOW TOUGH SOME OF THOSE
ARE.
ONCE WE'VE MADE THOSE TOUGH
CHOICES, WE REALIZE WE DON'T
MEASURE OUTCOMES, WE MEASURE
INDICATORS OF OUTCOME.
THAT'S WHERE WE LEFT IT
YESTERDAY.
NOW WE SHOULD HAVE -- LET ME
REACH FOR MINE -- YOU SHOULD
HAVE YOUR OUTCOMES MANAGEMENT
WORKSHEET THAT HAS FOUR
INDICATORS ON IT, SOMEWHERE
THERE SHOULD BE FOUR INDICATORS.
THE NEXT LOGICAL QUESTION IS
OKAY, NOW, HOW DO I GET THAT
INFORMATION?
AND THERE ARE SEVERAL DIFFERENT
PARTS OF THAT.
THE FIRST ONE HAS TO DO WITH
SOURCES AND METHODS.
AND WHAT ARE THE DIFFERENCES
BETWEEN SOURCES AND METHODS.
PRETTY STRAIGHTFORWARD.
I'M GOING TALK ABOUT TWO MINUTES
ON WHAT THEY ARE AND THEN BOB
WILL TELL YOU AND WALK THEW THE
HARD DECISIONS THAT THE TASK
FORCE HAD TO MAKE, IN TERMS OF
WHAT SOURCES AND MEDICINES IT
WAS GOING USE IN THE FIELD TEST.
FIRST OF ALL, WHAT ARE SOURCES?
WE ALL KNOW THIS, SOURCES ARE
PRETTY STRAIGHTFORWARD, IT'S
WHERE THE INFORMATION RESIDES                                         6
THAT YOU'RE GOING TO NEED.
AND THERE'S DIFFERENT KIND OF
SOURCES.
FOR INSTANCE, THERE'S EXISTING
INFORMATION.
SOMETIMES THIS IS GREAT, BECAUSE
IT'S CHEAPER AND EASIER TO GET
TO, FIT WORKS FOR US.
SOMETIMES THERE ARE WRITTEN CASE
CLIENT RECORDS, MAYBE THEY ARE
IN FOLDERS OR DOCUMENTS OF
VARIOUS TIMES AND REPORTS.
OR DATA FILES, AND THEY ARE
INCREASINGLY ELECTRONIC.
THOSE ARE THE EXISTING TYPES OF
INFORMATION.
THERE'S ALSO INDIVIDUALS,
PROGRAM PARTICIPANTS, OFTEN THE
CLIENTS.
OTHERS WHO KNOW THE
PARTICIPANTS, AND BOB WILL TALK
A BIT ABOUT HOW WE USED THAT,
SOMETIMES.
OR OTHER INDIVIDUALS, GENERAL
PUBLIC.
THERE'S ALSO THE PHYSICAL
ENVIRONMENT.
YOU THINK, THAT'S KIND OF FUNNY,
WHAT DO YOU MEAN THAT THE
PHYSICAL ENVIRONMENT IS A
SOURCE?
IF YOU THINK ABOUT IT, IF THE
INFORMATION YOU WANT TO KNOW IS
CURB CUTS ON STREETS.
YOU COULD GO LOOK ON A REPORT
AND SEE WHAT CURB CUTS HAVE BEEN
PUT IN A CITY.
OR YOU COULD GO INTERVIEW
SOMEONE WHOSE JOB IT WAS TO PUT
THE CURB CUTS IN.
OR, JUST GO OUT ON THE STREETS,
RIGHT?
SO THE STREET ITSELF, THE
PHYSICAL ENVIRONMENT IS THE
SOURCE YOU WOULD USE.
YOU JUST GO OUT AND LOOK AND
MAKE LITTLE TALLIES.
THESE ARE THE DIFFERENT KINDS OF
SOURCES.
AND AS YOU CAN IMAGINE, EACH ONE                                      7
HAS CERTAIN PROS AND CONS.
THAT'S YOUR CHALLENGE, WHEN YOU
GET INTO THIS, TO BALANCE THE
PROS AND CONS OF EACH ONE AND
MAKE THE DECISION THAT'S BEST
FOR YOU.
EVERY SOURCE HAS PROS AND CONS.
FOR INSTANCE, FILES ARE VERY
EASY TO ACCESS BUT DO THEY HAVE
WHAT YOU NEED?
AND WORSE YET, THEY MIGHT HAVE
CLOSE TO WHAT YOU NEED BUT NOT
EXACTLY WHAT YOU NEED.
SO YOU KIND OF MISS THE MARK
BECAUSE IT'S NOT EXACT, SO
YOU'VE GOT TO BE CAREFUL.
PARTICIPANTS ONCE KNEW WHAT
HAPPENED?
DO THEY REMEMBER IT?
DID YOU ALL READ THAT VERY SCARY
STUFF THIS LAST WEEK ABOUT THE
EVIDENCE FOR EYEWITNESSES IN
TRIALS AND STUFF?
SOME RESEARCH HAS COME TOGETHER
SHOWING THAT BASICALLY
EYEWITNESSES IN TRIALS ARE VERY
UNRELIABLE.
SO I THINK THERE WAS A SUPREME
COURT DECISION OR SOMETHING,
SOMEONE KNOWS MORE THAN I DO
ABOUT THIS, SOMETHING THAT CAME
OUT AND SAID, WE SHOULDN'T EVER
USE THEM AGAIN.
PARTICIPANTS, DO THEY REMEMBER
WHAT HAPPENED?
AND DIFFERENT SOURCES HAVE PROS
AND CONS.
SO THE KEY FOR THIS OBVIOUSLY,
ALTHOUGH IT'S EASIER TO SAY THAN
DO, THE KEY IS TO BRAINSTORM
MORE THAN ONE SOURCE, AND THEN
CHOOSE THE BEST SOURCE FOR YOUR
UNIQUE SITUATION.
OKAY.
WHAT METHODS:  NOW YOU KNOW
WHERE THE INFORMATION IS, AND
YOU'VE GOT TO FIGURE OUT HOW TO
GET FROM IT THAT SOURCE.
MAY I SUGGEST, MY FRIENDS, THAT
IT'S REALLY A VERY COMMON                                             8
MISTAKE TO BLEND THESE TWO
TOGETHER.
PEOPLE OFTEN BLEND THE TWO
TOGETHER.
KEEP IN MIND, THEY ARE TWO
COMPLETE SEPARATE DECISIONS.
THAT'S A REALLY IMPORTANT THING
TO KEEP IN YOUR HEAD.
SOURCES, THAT'S ONE DECISION.
AND METHODS IS A SEPARATE
DECISION.
IF YOU KEEP THAT IN MIND IT'LL
HELP YOU MAKE GOOD CHOICES FOR
YOURSELF.
SO THERE ARE OBVIOUSLY AGAIN
SOME OPTIONS FOR METHODS.
LIKE FOR INSTANCE A CASE RECORD
REVIEW, OR DOCUMENT REVIEW OR
DATA FILE REVIEW OR
QUESTIONNAIRES OR INTERVIEWS.
I'M GOING SKIP FOCUS GROUPS FOR
A SECOND.
OBSERVATIONS, YOU CAN WATCH
THINGS HAPPENING.
TESTING, IF YOU ARE INTERESTED
IN WHETHER SOMEBODY LEARNED HOW
TO DO SOMETHING, JUST GIVE THEM
A TEST AND SEE.
MECHANICAL MEASUREMENT, THAT
COULD BE WITH THE DOORWAYS,
RIGHT?
ALL SORTS OF THINGS THERE.
WHY DID YOU PUT NOT NEXT TO
FOCUS GROUPS?
ANYBODY HAVE ANY IDEA?
IN MY PERSONAL OPINION, YOU CAN
FEEL FREE TO DISAGREE, IN MY
PERSONAL OPINION WE SHOULD NOT
USE FOCUS GROUPS FOR OUTCOMES
INFORMATION.
WHY DO YOU THINK I FEEL THAT
WAY?
ANYBODY WANT TO TAKE A GUESS?
YES.
OH, AND BY THE WAY, AS YOU'RE
REACHING FOR THE MIC, SOMEONE 
GAVE A VERY NICE SUGGESTION TO 
ME THIS MORNING.
THEY SAID, I KNOW SOME OF THE
PEOPLE IN THE ROOM BUT NOT ALL                                        9
OF THE PEOPLE.
WHEN A PERSON FIRST STARTS TO
SPEAK, IF THEY COULD SAY THEIR
NAME AND WHAT STATED THEY ARE
FROM, THAT WOULD REALLY BE NICE
FOR ME.
IF YOU'RE WILLING TO DO THAT.
>> I'M WILLING TO DO THAT.
I'M JOAN CLAYPOOL FROM OREGON, I
WORK HALF TIME FOR THE DSU AND
HALF TIME FOR THE SILC.
>> A LITTLE LOUDER PLEASE.
>> I THINK FOCUS GROUPS TEND TO
BE BIASED BY THE PARTICIPANTS
AND THE INFORMATION THEY
PRESENT.
>> Mike Hendricks: I AGREE WITH
YOU 100%.
IF YOU THINK ABOUT THE HISTORY
OF FOCUS GROUPS AND YOU KNOW THE
HISTORY, THE WHOLE PURPOSE OF A
FOCUS GROUP IS THAT THE PEOPLE
SORT OF STIMULATE EACH OTHER'S
THINKING.
THAT'S WHY THEY ARE IN A GROUP.
THEY ARE SUPPOSED TO BRAINSTORM
AND SPARK OFF EACH OTHER AND THE
GROUP IS SUPPOSED TO COME UP
WITH THINGS THE INDIVIDUALS
WOULDN'T HAVE COME UP WITH ON
THEIR OWN.
THAT'S THE EXACT OPPOSITE OF
WHAT WE WANT.
IF YOU GET FIVE CONSUMERS, SAY,
AROUND A TABLE, YOU WANT TO KNOW
INDIVIDUALLY THEIR OUTCOMES.
YOU DON'T WANT THEM TO BE
INFLUENCING EACH OTHER.
IN MY OPINION, WE WANT TO BE
CAREFUL AND NOT USE FOCUS GROUPS
FOR THE KINDS OF OUTCOME
MEASURES WE HAVE.
YOU CAN FEEL FREE TO DISAGREE
AND MAYBE HAVE YOU AN EXAMPLE
THAT WORKS FOR YOU.
AGAIN, THOUGH, PROS AND CONS OF
EACH METHOD.
EVERYONE HAS PROS AND CONS.
QUESTIONNAIRES FOR INSTANCE ARE
CHEAP TO DISTRIBUTE.                                                 10
BUT WILL YOU GET THEM BACK?
WE'LL TALK MORE ABOUT THIS IN A
MINUTE, TOO.
INTERVIEWS MAKE IT REALLY EASIER
TO MAKE CONTACT BUT WILL PEOPLE
TALK WITH YOU?
DIFFERENT METHODS HAVE DIFFERENT
PROS AND CON AND IT'S THE SAME
KEY, THE EXCITING THING FOR YOU
IS TO BRAINSTORM MORE THAN ONE
METHOD, AND THEN CHOOSE THE BEST
ONE FOR YOUR UNIQUE SITUATION.
SO HERE'S AN EXAMPLE OF A SOURCE
AND METHOD.
IF THE OUTCOME IS TEENS ARE
KNOWLEDGEABLE OF PRENATAL
NUTRITION GUIDELINES, AND IF THE
INDICATOR IS THE NUMBER AND
PERCENT OF TEENS ABLE TO
IDENTIFY FIVE DIFFERENT FOOD
ITEMS THAT ARE GOOD FOR
THEMSELVES AND THEIR BABY --
THIS BRINGS UP THE DIFFERENCE
BETWEEN OUTCOMES AND INDICATORS
FROM YESTERDAY -- WELL, THEN THE
SOURCE AND THE MED MIGHT BE THAT
AFTER THE SECOND WEEK OF THE
PROGRAM WE ASK EACH TEEN -- SO
THE TEEN IS THE SOURCE -- TO
WRITE DOWN -- AND THAT WOULD BE
OUR METHOD, THEY ARE GOING WRITE
IT DOWN, AS MANY GOOD FOODS AS
SHE CAN REMEMBER FROM THE
TRAINING.
SO YOU SEE WHAT HAPPENS IS THE
SOURCE AND THE METHOD HAS TO
FLOW DIRECTLY FROM THE
INDICATOR.
THE INDICATOR NEEDS A SOURCE,
AND THE INDICATOR NEEDS A
METHOD.
PRETTY STRAIGHTFORWARD STUFF.
LET'S HEAR WHAT BOB SAYS ABOUT
HOW THE NICKEL GROUP MADE ITS
DECISION.
>> Bob Michaels: G THANK YOU.
SO WHERE WERE WE HERE?
WE'VE ALREADY CREATED THE
IDENTIFIER FOR OUR OUTCOMES AND
IDENTIFIED OUR LOGIC MODEL AND                                       11
PUT IT TOGETHER.
WE'VE GOT YOUR INPUT ON THAT.
THEN WE WANT TO ADD, AND START
PARING IT DOWN.
WE'LL HAVE YOU PICK THE EIGHT
MOST IMPORTANT FOR YOU, THE AIL
THE DESIRED OUTCOMES.
HE FOR THEM, THE INDICATORS, THE
12 THE FIRST YEAR, AND THEN 11
THE SECOND YEAR, INDICATORS THAT
YOU FELT BEST REFLECTED WHAT YOU
NEED.
SO NOW WE'RE AT A PLACE WHERE WE
HAVE TO SAY, OKAY, HOW WILL WE
GO ABOUT GATHERING THAT
INFORMATION?
WHAT WE DID, WE WENT TO THE --
OUR SUBGROUP, OUR TASK FORCE, AND
PICKED OUT A SUBGROUP FROM THAT
POPULATION, FIVE DIFFERENT
PEOPLE THAT WERE IN THERE.
MIKE, MYSELF, AND THEN THREE
OTHERS.
WE SAT DOWN AND TOOK A LOOK AT
EACH ONE OF THESE INDICATORS TO
TRY TO IDENTIFY WHAT THE PLACE
WAYS WERE THEN TO GATHER THAT
INFORMATION.
SO WHAT I'VE DONE HERE IS I'VE
HIGHLIGHTED THE ONES THAT WE
PICKED.
SO WE CHOSE AFTER THINKING AND
TALKING ABOUT THIS AND WORKING
WITH THIS AND GETTING FEEDBACK
FROM THE REST OF THE TASK FORCE
-- AND REMEMBER, THEY WERE A
BUNCH OF RESEARCHERS AND PEOPLE
WHO HAD EXPERTISE IN GATHERING
INFORMATION AND EVALUATION.
WE DECIDED WHAT TO DO THEN FOR
GATHERING THIS INFORMATION, THE
SOURCES THAT WE WOULD USE WERE
TO GATHER WRITTEN CASE AND
CLIENT RECORDS; AND USING OTHER
EXISTING INFORMATION, DOCUMENTS
OF VARIOUS TYPES, AND THEN DATA
FILES THAT CENTERS HAVE.
ON AN INDIVIDUAL BASIS THEN,
GATHERING INFORMATION FROM
PROGRAM PARTICIPANTS, FROM                                           12
CONSUMERS AND I&R CALLERS.
AND OTHERS WHO KNOW THE
PARTICIPANTS, FAMILY MEMBERS.
ONE OF THE THINGS WE DID THINK
ABOUT EARLY ON AND DIDN'T
REALIZE, IS HOW MANY CHILDREN
THE CENTERS ARE PROVIDING 
SERVICES TO.
OF SO IT BECAME REALLY IMPORTANT
FOR US TO BE TALKING TO PARENTS
OF MINORS.
WE DID NOT PICK OTHER
INDIVIDUALS AND PHYSICAL
ENVIRONMENT.
WE'RE NOT SAYING THAT THEY ARE
BAD BUT FOR THE PURPOSE OF
REHAB, WITH INDICATORS THAT WE
WERE LOOKING AT, THEN THEY
WOULDN'T BE ANY HELP AT ALL.
OTHER METHODS, WE USED A CASE
RECORD REVIEW, DOCUMENT REVIEW,
DATA FILE REVIEW, AND
INTERVIEWS, TELEPHONE
INTERVIEWS.
WE DID NOT SEND OUT
QUESTIONNAIRES.
I REMEMBER ALL OF YOU HAVE DONE
THIS AT SOME POINT OR ANOTHER.
MY EXPERIENCE ON CENTERS YOU YOU
SEND QUESTIONNAIRES, IF YOU GET
11, 15% BACK THAT'S GOOD.
YOU KNOW, USUALLY WE DON'T GET
ANY MORNING THAT.
FOR OUR PURPOSES HERE, AND
EVERYBODY WILL TELL YOU, MIKE
WILL TELL YOU, PEOPLE THAT KNOW
EVALUATION WILL SAY YOU NEVER
WILL BE SATISFIED WITH 10% OR
15%.
EVEN IF YOU CALL IT, YOU'RE
LOSING RANDOMIZATION AND THAT
WAS REALLY IMPORTANT FOR US.
WE OF COURSE DID NOT DO FOCUS
GROUPS.
AND THOUGHT THAT OBSERVATION,
TESTING AND MECHANICAL
MEASUREMENT WAS -- THEY WERE
GOOD METHODS BUT NOT GOOD
METHODS FOR WHAT WE'RE DOING
HERE.                                                                13
OUR BEST METHODS WERE THE CASE
RECORDS, DOCUMENT REVIEW, DATA
FILE REVIEW AND THEN INTERVIEW.
OKAY?
SO THAT'S BASICALLY ALL IT WAS.
WE PICKED THOSE OUT AND DECIDED
THAT WAS WHERE WE WERE GOING
START TO GATHER THE INFORMATION.
SO WITH THAT, I'LL TURN IT BACK
TO MIKE HERE.
>> Mike Hendricks: THANKS, BOB.
WELL, AS YOU CAN IMAGINE, IT'S
YOUR TURN.
NOW, THOSE OF YOU IN THE ROOM
NOW KNOW YOU'VE GOT THIS
OUTCOMES MANAGEMENT WORKSHEET.
LET ME TAKE A SECOND FOR OUR 
ONLINE FRIENDS.
THIS DOCUMENT IS CALLED OUTCOMES
MANAGEMENT WORKSHEET, IT'S IN
YOUR MATERIALS, ACCESSIBLE
SOMEHOW ON YOUR LINK THERE,
YOU'LL SEE IT.
YOU NEED TO PULL IT OUT AND
PAINT COPY OF IT, IF YOU WOULD.
YOU ALL KNOW WHAT THE TASK IS,
I'M SURE.
YOU'VE PICKED YOUR TWO OUTCOMES.
YOU'VE FOUND FOUR INDICATORS FOR
THOSE TWO OUTCOMES.
WHAT YOU DON'T HAVE YET IS
EITHER A SOURCE OR A METHOD FOR
GETTING THE INFORMATION ON THOSE
INDICATORS.
THAT'S WHAT I NEED YOU TO DO AT
YOUR TABLE, TALK ABOUT IT AGAIN
AS A GROUP AND ONLINE AND AROUND
YOUR TABLES.
TALK BIT, AND PLEASE, REMEMBER
WHAT WE SAID A SECOND AGO.
THE KEY TO THIS IS TO NOT JUMP
TO THE OBVIOUS CONCLUSION, NOT
JUMP TO THE FIRST OBVIOUS
SOURCE.
NOT JUMP TO THE FIRST OBVIOUS
METHOD.
YOU MAY END UP THERE, DON'T GET
ME WRONG, YOU MAY WELL END UP
THERE.
BUT DEPARTMENT OF JUMP TO THAT.                                      14
KIND OF FORCE YOURSELF, IS THERE
NOT SOME POSSIBILITY, OTHER
OPTION, OTHER OPTION, OTHER
OPTION?
SOMETIMES THE OTHER OPTIONS ARE
THE ONES THAT REALLY, REALLY ARE
BETTER.
YOU WON'T THINK OF THEM IF YOU
DON'T MAKE YOURSELF THINK OF
THEM.
DO YOU HAVE A QUESTION?
AND PLEASE, THE INTRODUCTION, IF
YOU WOULD.
>> MY NAME IS JOSE CALAZANO, I
REPRESENT CALIFORNIA
REHABILITATION SERVICES.
I HAVE A QUESTION.
>> PLEASE.
>> TO FIND THE INDICATORS, DON'T
YOU NEED THE SOURCE FIRST TO
TELL YOU WHERE THE INDICATORS
ARE COMING FROM?
>> Mike Hendricks: WELL, TELL US
WHY YOU THINK SO.
>> Audience Member: FIRST OF
ALL, THE SOURCE WOULD GIVE YOU
AN IDEA OF WHAT TO LOOK FOR OR
HOW TO ANALYZE YOUR INDICATOR OR
WHAT TYPE OF METHOD YOU MIGHT
WANT TO USE TO GET THE INDICATE.
>> Mike Hendricks: WITH ALL DUE
RESPECT, I DISAGREE 100%, NO
OFFENSE.
TO ME, IT ALL FLOWS FROM THE
OUTCOME.
THE DESIRED GOALS, WHAT YOU'RE
TRYING TO ACHIEVE, THEN YOU ASK
YOURSELF, YES, BUT HOW EXACTLY
AM I DEFINING THAT OUTCOME?
WHAT EXACTLY DO I MEAN BY THAT
OUTCOME.
THAT'S WHAT THE INDICATOR Z. AND
ONLY AT THAT POINT DO YOU THEN
ASK YOURSELF, OKAY, NOW THAT'S
EXACTLY WHAT I NEED, THAT'S THE
RIGHT, GETTING THE INFORMATION
YOU NEED.
ONLY ONCE I KNOW EXACTLY WHAT I
NEED, NOW I'VE GOT FIGURE OUT
HOW TO GET IT.                                                       15
IN MY OPINION I THINK YOU SHOULD
NOT THINK ABOUT METHODS OR
SOURCES UNTIL YOU'VE DEFINED
EXACTLY THE PIECE OF INFORMATION
YOU NEED.
THAT'S MY OPINION.
MAYBE I'M WRONG AND WE'LL TALK
AT BREAK.
OKAY.
SO IF YOU'LL DO THAT, PLEASE, AT
YOUR TABLES.
COME UP WITH A METHOD, HOPEFULLY
MORE THAN ONE, MAY BE -- I'M
SORRY, A SOURCE AND A METHOD FOR
EACH OF YOUR FOUR INDICATORS.
>> Mike Hendricks: JUST IN THE
INTERESTS OF TIME, MAYBE A FEW
MORE MINUTES ON THIS EXERCISE
AND THEN WE'LL SEE WHAT WE HAVE.