ROUGH DRAFT 9-14-11, Outcome Measures for Centers for Independent Living – An IL NET Resource Presented by ILRU >> Mike Hendricks: OKAY. WHY DON'T WE SEE SOME IDEAS WE'VE GOT. I'M GUESSING YOU HAVEN'T GOTTEN ALL FOUR OF THEM DONE, THAT'S JUST REALITY. BUT LET'S HEAR MAYBE SOME SOURCE AND METHOD COMBINATION FOR AN INDICATOR OR TWO. WHO HAS SOMETHING THEY WANT TO SHARE WITH US? I CAN CALL ON A TABLE. COME ON, COME ON, IT'S EARLY IN THE MORNING. AND PLEASE, INTRODUCE YOURSELF. >> Audience Member: I'M FRED JOHNSON FROM NORTH CAROLINA. [CHEERING] >> Mike Hendricks: THANK YOU, BY THE WAY, FRED, FOR STANDING UP. I'M SURE THAT'S HELPFUL FOR PEOPLE. >> Audience Member: ONE OF OUR INDICATORS WAS TO IMPROVE -- TO ASSIST THE STUDENT IMPROVING THEIR ACADEMIC STANDING BY A LETTER GRADE. SO HOW DO WE IDENTIFY THAT, WHAT'S OUR SOURCE? OUR SOURCE IS TO HAVE THE STUDENT PROVIDE US WITH THE 16 REPORT CARD AT THE END OF THE QUARTER OR WHATEVER. >> Mike Hendricks: SO THE SOURCE WOULD BE THE REPORT CARD ITSELF, NOT THE KID, NOT THE SCHOOL, THE REPORT CARD. INTERESTING. >> Audience Member: AND THEN THE METHOD WOULD BE GET A REPORT CARD AND REPORT ON THE FINDING. >> Mike Hendricks: YOU'D HAVE A FORM OR SOMETHING TO USE TO KEEP TRACK OF THE DIFFERENT REPORT CARDS. >> Mike Hendricks: I THINK THAT'S REALLY CLEVER. WHAT ELSE COULD THEY HAVE DONE? THEY COULD HAVE ASKED THE KIDS THE GRADES. YOU COULD HAVE ASKED THE PARENTS FOR THE GRADES. YOU COULD HAVE TRIED TO GET THEM FROM THE SCHOOL. I KNOW THERE'S PROBLEMS THERE BUT IT MIGHT HAVE BEEN POSSIBLE. YOU HAD SEVERAL POSSIBILITIES AND DECIDED NO, YOU WANTED THE REPORT CARD ITSELF TO BE THE SOURCE. THAT'S THE EXACT KIND OF DECISION-MAKING I'M TALKING ABOUT. THINK ABOUT YOUR OPTIONS AND DECIDE THE ONE THAT MAKES THE MOST SENSE FOR YOU. THAT'S GREAT, THANK YOU, FRED. HOW ABOUT SOMEBODY ELSE? ONE MORE, COME ON. ESPECIALLY IF YOU THINK YOU HAD ONE THAT WAS EITHER TOUGH OR CREATIVE OR INTERESTING OR YOU WERE ABLE TO COME UP WITH SOMETHING DIFFERENT. ALL RIGHT. I WILL CALL ON MY FRIENDS AT THIS TABLE RIGHT HERE. OR MY FORMER FRIENDS AT THIS TABLE RIGHT HERE. HOW'S THAT? AND ANYWAY, IF YOU'RE WILLING TO 17 STAND UP, ALSO, THAT WOULD BE WONDERFUL. FRED'S GIVEN US A GOOD MODEL HERE. >> Audience Member: THANKS, FRED. >> Mike Hendricks: NOW WE HAVE SOMETHING IN COMMON WITH THIS TABLE, FRED. >> Audience Member: I'M JOHN MADISON WITH A CIL IN OREGON. AND WE HAD AN INDICATOR, THE NUMBER OF PARTICIPANTS WHO REPORT FEELING MORE CABLE OF WORKING ON A TEAM WITH THEIR FEARS OF COMPLETION OF THE AFTER-SCHOOL PROGRAM. SO WE CHOSE -- WE THOUGHT WE HAD TWO POTENTIAL SOURCES TO WORK WITH, THAT IT COULD BE BOTH THE CHILDREN AND THE PARENTS. AND THE BEST METHOD TO GATHER WHETHER OR NOT THEY WERE EXPRESSING A HIGHER CAPABILITY THAN THEIR PEERS WOULD BE THROUGH AN INTERVIEW WITH BOTH THE CHILDREN AND THE PARENTS. >> Mike Hendricks: SO YOU WERE GOING ACTUALLY USE TWO DIFFERENT SOURCES? >> Audience Member: YEAH, BECAUSE WE'RE SMART. [LAUGHTER] >> Mike Hendricks: ALL RIGHT. I CAN THINK OF ANOTHER POSSIBLE METHOD THEY MIGHT HAVE USED. CAN ANYBODY ELSE THINK OF ANOTHER POSSIBLE METHOD THEY MIGHT HAVE USED? WHAT WAS THE INDICATOR AGAIN? >> Audience Member: IT WAS NUMBER OF PARTICIPANTS WHO REPORT FEELING MORE CAPABLE OF WORKING IN A TEAM WITH THEIR PEERS. I GUESS IT WOULD BE THEIR PEERS, AS WELL. >> Mike Hendricks: COULD BE PEERS. UNDER ANOTHER POSSIBLE 18 SITUATION, IF THE INDICATOR WERE WORDED DIFFERENT, FIT WASN'T THE NUMBER WHO REPORT BUT THE ONES THAT ARE MORE COMFORTABLE, YOU COULD HAVE USED OBSERVATION, PUT THEM IN A SITUATION AND OBSERVED. JUST BRAINSTORMING. JUST THE KIND YOU OUGHT TO DO. WHAT ELSE CRAZY THING MIGHT WORK? I WORKED WITH A GROUP ONCE THAT CAME UP WITH THE IDEA OF WATCHING SURVEILLANCE CAMERAS AT GROCERY STORES. AND IT WORKED REALLY, REALLY WELL FOR THEM, TURNS OUT. YOU MIGHT WONDER WHAT THE HECK WAS THAT B. THAT REALLY WORKED FOR THEM IN REAL LIFE. ALL RIGHT. I DON'T WANT TO BELABOR THIS. I THINK YOU DEFINITELY GET THE IDEA. IN MY OPINION YOU START WITH THE INDICATORS, AND THEN YOU ASK YOURSELF, WHAT SOURCE, WHAT METHOD. OKAY. LET'S, IF WE CAN, MOVE ON DOWN THE YELLOW BRICK ROAD. NOW WE'VE GOT THAT FIGURED OUT. NOW YOU KNOW THE SOURCE, YOU KNOW THE METHOD. NOW YOU'VE GOT TO GO DO IT. AND THAT ISN'T ALWAYS THE EASIEST THING IN THE WORLD, IS IT? SO WE'VE GOT THIS WHOLE SESSION ON GATHERING OUTCOME INFORMATION, A LARGE CHUNK IS GOING TO BE BOB TALKING WITH YOU ABOUT THE PRACTICAL KINDS OF DECISIONS THAT THESE 28 CILs THAT WE JUST ENDED UP WORKING WITH HAD TO MAKE WHEN THEY DID THEIR FIELD TEST, BECAUSE THAT'S GOING TO BE VERY RELEVANT TO YOU AND YOUR WORLD. BEFORE WE GET TO BOB AND DO 19 THAT, I WANT TO COVER SOME OF THE KINDS OF QUESTIONS YOU'RE EVENTUALLY GOING TO HAVE TO ASK YOURSELVES. WE CAN'T BEGIN TO GIVE YOU THE ANSWERS, IT'S SITUATION SPECIFIC AND UNIQUE TO YOU. WE THINK IT'S HELPFUL TO AT LEAST PUT THE QUESTIONS DOWN ON PAPER. MY GUESS IS THESE NEXT SHEETS OF PAPER WILL BE USEFUL FOR YOU WHEN YOU GO BACK AND ACTUALLY START THINKING ABOUT GATHERING YOUR INFORMATION. SO MAYBE YOU HANG ON TO THESE AND MAYBE THEY TRIGGER GOOD THOUGHTS BACK AT THE TIME. BY THE WAY, ARE ANY CILs IN HERE ACTUALLY RIGHT NOW COLLECTING THE KINDS OF OUTCOMES INFORMATION THAT WE'RE TALKING ABOUT? ARE YOU ACTUALLY ALREADY DOING IT OR ARE MOST OF US SORT OF REALIZING WE HAVE TO START DOING IT? >> Audience Member: THE LATTER. >> Mike Hendricks: THAT'S WHAT I WAS GUESSING. THERE WERE SOME HANDS GOING UP. FOR THOSE ONLINE, LOOKS LIKE MAYBE 15% OF THE HANDS WENT UP, AND 85% OF THE PEOPLE KNOW THIS IS COMING DOWN THE TRACKS BUT WE HAVEN'T STARTED TO DO IT YET. FOR THOSE OF YOU, THESE PAGES MIGHT BE ESPECIALLY GOOD THINGS TO HANG ON TO. LET'S CUFF SOME QUESTIONS. WE COULD BE WRONG BUT BOB AND I THINK THAT PROBABLY YOU'LL END UP COLLECTING MOST OF YOUR OUTCOME INFORMATION EITHER DIRECTLY FROM CLIENTS, BEING EITHER CONSUMERS OR I&R CALLERS, OR SOMEHOW OUT OF YOUR MIS SYSTEM. THAT'S OUR GUESS, MOST OF IT. 20 LET'S ASK OURSELVES SOME OF THE KINDS OF QUESTIONS THAT YOU'RE EVENTUALLY GOING HAVE TO WRESTLE WITH, STARTING WITH THE CLIENTS. ONE OF THEM IS, WHAT INFORMATION SHOULD YOU ASK THEM? WE THINK THERE'S DIFFERENT KINDS. ONE IS SORT OF OBVIOUSLY THE OUTCOME INFORMATION. WHATEVER IT IS, YOU'RE TRYING TO GATHER, THAT'S THE MEASURABLE INDICATORS. THOSE CAN BE ATTITUDES, BEHAVIORS, CONDITION OR STATUS, THOSE CAN BE THOSE KINDS OF SEVEN MAGIC WORD THINGS WE'VE TALKED ABOUT. BUT ALSO WE MIGHT SUGGEST -- AND YOU'LL SEE THAT WE DID THIS -- ASK THEM MORE, ASK THEM FOR SOME DETAILS, SOME EXPLANATIONS. WE THINK IT'LL HELP GAVE RICHNESS TO THE NUMBERS, IF YOU WILL. SOME PEOPLE TALK ABOUT THE STORY BEHIND THE NUMBERS. I THINK THAT'S REALLY IMPORTANT TO TRY TO GET AT. BUT BE SELECTIVE. ASK ONLY WHAT YOU NEED. IT'S EASY TO ASK TOO MUCH, TO GET BURNED DOWN, VERY DOWN. SO BE CAREFUL ABOUT WHAT INFORMATION YOU ASK. AND THEN WHO SHOULD YOU ASK? WELL, YOU KNOW, OKAY, CLIENTS. WE'LL ASK CLIENTS. TURNED OUT NOT TO BE QUITE THAT SIMPLE. WHAT IS A CLIENT? WHAT ABOUT SHORT-TIMERS? WHAT ABOUT SOMEONE WHO'S WORKED WITH YOU FOR THREE WEEKS. THAT IS A LEGITIMATE PERSON TO INTERVIEW? WELL, MAYBE. WHAT ABOUT A DROPOUT WHO WORKED WITH YOU FOR A MONTH AND LEFT? MMM, MAYBE. 21 WE DO THINK IT'S IMPORTANT TO ASK INDIVIDUALS, NOT GROUPS, WE TALKED ABOUT THAT. DO YOU EVER NEED TO ASK SURROGATES? WE THINK SO, PARENTS, CAREGIVERS. THE KEY IS YOU'VE GOT TO GET THE INFORMATION YOU NEED. SO WHO HAS THE INFORMATION YOU NEED? OKAY. NOW, SHOULD YOU ASK ALL CLIENTS OR ONLY A SAMPLE? THE TABLE I WAS AT WAS DISCUSSING THIS. SOMETIMES YOU MIGHT THINK, LET'S ASK ALL OF THEM, HECK YES, WHY NOT? THAT'S THE MOST RIGOROUS, THE BEST METHODOLOGY, RIGHT? NO, OFTEN NOT, IN FACT. FIRST OF ALL, YOU MAY HAVE LOTS AND LOTS OF CLIENTS AND YOU JUST CAN'T. IT COSTS A WHOLE BUNCH OF MONEY TO ASK THEM ALL. IT TAKES MORE TIME TO GET THE RESULTS IF YOU HAVE TO ASK THEM ALL, AND IT PUTS A BURDEN, EVEN THOUGH NOT A BIG BURDEN, BUT IT DOES PUT A BURDEN ON EVERY CLIENT. SOMETIMES IT'S BETTER TO ASK ONLY A SAMPLE OF THE CLIENTS. IF YOU ASK ONLY A SAMPLE, HOW TO PICK, HOW TO PICK WHICH ONES. YOU DON'T WANT TO BIAS IT, YOU DON'T WANT TO PICK THE 12 PEOPLE YOU WORKED MOST RECENTLY WITH, OR THE 15 YOU KNOW ARE YOUR BIGGEST SUCCESSES, BECAUSE AGAIN, LIKE WE TALKED ABOUT YESTERDAY, THAT MAKES YOU LOOK GOOD BUT IT DOESN'T HELP YOU REALLY KNOW HOW YOU'RE DOING. THAT'S WHAT THIS IS ALL ABOUT. SO THERE ARE LOTS OF WAYS TO BIAS, BY THE WAY, I'M SURE YOU'RE CLEVER AT IT, TOO. 22 WE THINK PICKING THEM RANDOMLY IS THE ONLY LEGITIMATE WAY TO DO IT. I KNOW SOMETIMES THAT WORD RANDOM ASSIGNMENT, RANDOM SELECTION, THAT CAN BE SCARY. I'M NOT A RESEARCHER, HOW THE HECK DO I DO THAT? THERE'S LOTS AND LOTS OF WAYS TO DO IT. HERE'S JUST ONE DOWN HERE. THERE'S A WEBSITE TO HELP YOU DO EXACTLY THAT, SELECT A RANDOM GROUP, NOT A PROBLEM AT ALL, IT'S LISTED HERE. HOW MANY CLIENTS SHOULD WE ASK? LET'S SAY -- SOMEBODY TELL ME, HOW MANY CLIENTS DO YOU HAVE IN YOUR CIL. SHOUT OUT SOME NUMBERS. >> Audience Member: 212. >> Mike Hendricks: SOMEONE HERE HAS 212, OKAY. SHOULD YOU RANDOMLY SAMPLE HALF OF THOSE? SHOULD YOU RANDOMLY SAMPLE A FOURTH OF THOSE, A TENTH OF THOSE? HMMM. THIS GETS TRICKY, THIS GETS TRICKY. IT CAN GET COMPLICATED DEPENDING ON WHAT YOU WANT TO KNOW AT THE END. DO YOU WANT TO KNOW HOW ALL 212 OF THOSE DID, OR BE ABLE TO SAY HOW 212 DID? OR ARE YOU INTERESTED IN LEFT-HANDED FEMALES AND HOW THEY DID? IF YOU'RE INTERESTED IN LEFT-HANDED FEMALES, THAT HAS DEFINITE SAMPLING IMPLICATIONS. IT REALLY DEPENDS ON WHAT YOU WANT TO KNOW. YOU WILL SOMETIMES NEED EXPERT STATISTICAL HELP. I'M NOT GOING SUGARCOAT IT, SOMETIMES YOU WILL NEED THAT KIND OF HELP. 23 WHEN SHOULD YOU ASK? WHEN IT'S CONVENIENT? MAYBE LIKE A SPECIFIC TIME OF THE YEAR OR AS THE WORKLOAD ALLOWS? ONE OF THE THINGS BOB'S GOING TALK ABOUT IS WE REALLY, REALLY WRESTLED WITH WHEN TO CALL BACK I&R CALLERS. SOMEBODY CALLED INTO THE CIL AND WE WERE GOING CALL THEM BACK AND FIND OUT SOME OUTCOME INFORMATION. WHEN DO YOU DO THAT? A WEEK AFTER THEY HAVE CALLED THE CIL? TWO MONTHS AFTER THEY HAVE CALLED THE CIL? A YEAR AFTER THEY HAVE CALLED THE CIL? >> Audience Member: NO. >> Mike Hendricks: IT GETS TRICKY. OR MAYBE AT A CERTAIN TIME DURING WORKING WITH PEOPLE, WHILE THEY WERE RECEIVING IT, AT THE END OF THE RECEIVING IT, AFTER SERVICES HAVE STOPPED. THESE ARE PRACTICAL QUESTIONS YOU HAVE TO WRESTLE WITH. THEN HERE'S SOME IMPORTANT QUESTIONS ABOUT CLIENTS. WHAT TYPE OF INTERVIEW? ARE YOU GOING TO DO IN-PERSON GO, TO THEIR HOUSE OR HAVE THEM COME TO YOU? BY PHONE? BY E-MAIL? WHERE WOULD YOU DO IT, IF YOU DO GO DO IT IN PERSON? WHO'S GOING CONDUCT IT? THE SAME PERSON WHO GAVE THEM SERVICES? IS THAT GOOD OR BAD? >> Audience Member: BAD. >> VOLUNTEERS? HOW DO YOU TRAIN THEM? HOW WOULD YOU ENSURE THE SAFETY OF PEOPLE WHO GO THERE? OTHER QUESTIONS. 24 AND OTHER IMPORTANT ISSUES ABOUT INTERVIEWS. ARE YOU GOING TO GET THEIR CONSENT? ARE THEY ABLE BY MEMORY OR LANGUAGE TO BE INTERVIEWED? ARE THEY WILLING TO TALK WITH YOU? THIS IS A LONG LIST OF THINGS THAT ARE PROBABLY MAKING YOU SAY, OH, MY GOSH, THIS IS HARD WORK. BUT THE THING IS, EACH OF THESE CAN BE SOLVED, YOU JUST HAVE TO THINK THESE THROUGH. THAT'S THE KEY, YOU HAVE TO THINK THESE THROUGH. HOW MUCH EFFORT IS REQUIRED, WHAT TYPE OF ANSWERS DO YOU WANT. HOW LONG ARE THEY WILLING TO SPEND. HOW ABOUT SOME QUESTIONS ABOUT THE MIS SYSTEMS? EVERYBODY HAS SOME KIND OF ELECTRONIC MIS SYSTEM, RIGHT? ANYBODY WHO DOESN'T? EVERYBODY HAS SOME KIND OF ELECTRONIC MIS SYSTEM. THERE'S SOME INFORMATION WE GOT THAT BOB WILL TELL YOU ABOUT. YOU CAN GET ADDITIONAL OUTCOME INFORMATION. AND YOU CAN ALSO GET WHAT MIGHT INFLUENCE THOSE OUTCOMES. ARE YOU DOING A BETTER JOB WITH WOMEN OR MEN? WITH OLDER PEOPLE OR YOUNGER PEOPLE? WITH CERTAIN RACES OR ETHNICITIES, CERTAIN DISABILITIES? THAT INFORMATION IS IN YOUR MIS. THE SERVICES THEY HAVE RECEIVED, HOW MUCH WHAT, TIME, THAT'S ALL IN THERE. OKAY. FINAL ONE I THINK FOR ME, HOW ARE YOU GOING TO RECORD WHAT YOU'VE GOT? 25 YOU'VE TALKED TO CONSUMERS, GOTTEN STUFF OUT OF THE MIS. YOU'VE GOT PUT IT IN SOME KIND OF INFORMATION-GATHERING FORM. YOU CAN USE THESE DOCUMENT REVIEW ORS QUESTIONNAIRES OR WHATEVER. WE STRONGLY SUGGEST YOU GET BOTH NUMBERS AND THE WORDS, THE STORY BEHIND THE NUMBERS. KEEP THE ANSWERS CONFIDENTIAL. BETTER YET, ANONYMOUS. THE DIFFERENCE BETWEEN CONFIDENTIAL AND ANONYMOUS, CONFIDENTIAL WE KNOW WHO YOU ARE BUT WE'RE NOT GOING TO TELL. ANONYMOUS IS WE DON'T KNOW WHO YOU ARE. THAT'S A BIG DIFFERENCE. FINALLY, FIELD-TEST EVERYTHING. ONCE YOU'VE FIGURED IT OUT AND YOU THINK, MY GOSH, I'VE GOT IT NAILED, I GUARANTEE YOU YOU HAVEN'T. WE CERTAINLY DIDN'T. SO TRY IT OUT. TEST ALL YOUR PLANS, HOW TO IDENTIFY THE SOURCE, INFORMATION, GATHERING IT, RECORDING IT, PUTTING IT IN A COMPUTER, ANALYZING IT. TRY THE WHOLE A TO Z BALLGAME, TRY THAT. LET'S HAVE BOB TELL WHAT YOU WE HAD TO WRESTLE WITH. YOU'RE PROBABLY GOING TO HAVE TO MAKE MANY OF THESE VERY SAME KINDS OF DECISIONS. >> Bob Michaels: THANKS, MIKE. THIS IS REALLY COMPLEX. THAT IS REALLY DIFFICULT STEP IN THE PROCESS. BUT THE FIELD TEST IS ABSOLUTELY NECESSARY. SO ANY TIME, IF YOU'RE DEVELOPING THESE, IF YOU DEVELOP ANYTHING NEW, AND YOU SAY, OH, WE KNOW THE ANSWERS, DON'T DO THAT, DO A FIELD TEST FIRST. TRY IT OUT AND MAKE SURE YOU 26 KNOW AND UNDERSTAND THE QUESTIONS THAT YOU'RE GOING ASK PEOPLE, AND THAT THEY KNOW AND UNDERSTAND HOW YOU HAVE IT WORDED. SO YOU'LL CHANGE A LOT, I GUARANTEE YOU. WE CERTAINLY DID. WE HAD THE BEST MINDS IN THE COUNTRY WORKING ON THIS, AND WE CHANGED ALL KINDS OF THINGS BETWEEN THE FIRST AND SECOND YEAR, AND IF WE HAD TO DO IT A THIRD YEAR, WE WOULD DO IT AGAIN. SO THERE ARE LOTS OF CHANGES AND YOU REALLY NEED TO LOOK AT THIS. SO HERE WE ARE. WE'VE GONE AHEAD AND WE'VE IDENTIFIED THE OUTCOMES AND WE'VE PICKED OUT HOW WE'RE GOING TO GATHER SOME OF THE INFORMATION AND WE'VE GOT DO IT. SO A CUSTOM THINGS WE WENT THROUGH IN THIS PROCESS. THE FIRST WAS ACTUALLY SETTING UP THE FIELD TEST SO. WHAT WE DID IS WE WENT OUT IN THE SECOND YEAR, WE WENT AND RECRUITED 28 VOLUNTEER CENTERS, VARYING ON GEOGRAPHIC LOCATION, SIZE OF BUDGET, NUMBER OF STAFF, NUMBER OF CONSUMERS, AND SOURCES OF FUNDING. IN ADDITION, WE DEVELOPED A COMPREHENSIVE TRAINING. AND YOU HAVE THAT. REMEMBER, WE HAD JUST SHOWED YOU THE MANUAL YESTERDAY AND JUST POINTED OUT IN YOUR INFORMATION IN THERE, MAKE SURE THAT IT'S IN YOUR PACKET. WE CREATED A 15-PAGE TRAINING MANUAL. WE REQUIRED PARTICIPATION IN TWO NINETY-MINUTE TELECONFERENCES. IF YOU DIDN'T PARTICIPATE, WE KNOCKED YOU OUT OF THE TEST. YOU HAD TO PARTICIPATE IN THIS. AND THEN WE PROVIDED INDIVIDUAL 27 ASSISTANCE AS NEEDED. SO IF ANYBODY HAS ANY QUESTIONS ABOUT ANYTHING, THEY HAD A PLACE TO GO TO GET THEM ANSWERED. WE KEPT AN ACTIVE, PRIVATE INTERNET-BASED LISTSERV. IF PEOPLE HAD QUESTIONS TO ASK, THEY USED A IT A LOT IN THE FIRST YEAR BUT NOT THAT MUCH IN THE SECOND YEAR. THE PEOPLE THAT -- SOME PEOPLE -- MOST OF THE PEOPLE STAYED ON THE SECOND YEAR AND JUST DIDN'T NEED IT AS MUCH THE SECOND TIME AROUND. IN TERMS OF URBAN AND RURAL B HALF THE CENTERS SERVED BOTH URBAN AND RURAL AREAS. 25% JUST RURAL, AND 25% JUST URBAN. HOW DID WE GATHER THE INFORMATION? WHAT WE DID THEN IS CREATED THREE PAPER FORMS TO GATHER OUTCOME INFORMATION. ONE RELATED TO CONSUMERS, PERSONS WITH A CSR, I&R CALLERS, AND THEN THE CIL ITSELF. THESE ARE IN YOUR SUPPLEMENTAL MATERIALS, AS WELL, THESE THREE FORMS. THEY ARE CALLED INFORMATION GATHERING FORMS. THERE IS WITH FOUNTAIN CIL DATA, ONE FOR CONSUMERS AND WITH FOUNTAIN I&R. WE LOOK TED DIFFERENT METHODS FOR GATHERING INFORMATION AND WE FELT WE COULD GATHER EVERYTHING WE NEEDED BY THESE THREE METHODS, AND DEVELOPED THESE THREE INFORMATION FORMS. NOW, THEN WHAT WE D WE ESTABLISHED A SURVEY MONKEY THE 5:00 NEWS EACH CENTER. WE WILL TALK MORE ABOUT THIS NEXT SESSION, BUT SURVEY MONKEY IS A WAY OF GATHERING INFORMATION, MOST OF YOU I THINK ARE FAMILIAR WITH IT, A WAY OF 28 RECORDING INFORMATION YOU DO IN A SURVEY. WE ESTABLISHED A SURVEY MONKEY LINK FOR EACH CENTER SO IT COULD ENTER THE OUTCOME DATA IT GOT INTO A CENTRALIZED DATABASE. WE DID THE FIELD TEST. AFTER WE DID THE FIELD TEST WE DEBRIEFED THE CENTERS. WE ARE GOING TO TALK A LOT MORE ABOUT THIS IN HERE. WE DID THREE DIFFERENT WAYS. WE HAD THEM LOOK AT THE THREE FORMS AND DID TRACK CHANGES TO OUR MATERIALS. THEY ALSO DID A SURVEY MONKEY SURVEY, WHICH REALLY TALKED ABOUT THEIR EXPERIENCE AS ONE OF THE PRESENTERS. WE DID A DEBRIEFING CONFERENCE CALL. WE WILL GO OVER THIS IN MORE DEPTH IN THE SESSION LATER THIS AFTERNOON, MAYBE TOMORROW MORNING. THEY ANALYZED THE OUTCOME DATA IN SEVERAL DIFFERENT WAYS AND WE WILL SHOW YOU WHAT THAT MEANS LATER THIS MORNING. WE PROVIDED FOUR SEPARATE REPORTS TO EACH CENTER. WE WROTE AN OVERALL REPORT OF THE ENTIRE FIELD TEST PROCESS AND RESULTS WILL BE AVAILABLE ONLINE. EVERYTHING RELATED TO THIS IS GOING TO BE AVAILABLE TO YOU, IT IS NOW AVAILABLE TO YOU ONLINE. FOR THOSE OF YOU IN THIS CLASS OR THOSE PEOPLE WHO ARE WATCHING THE VIDEO, I WANTED TO LET YOU KNOW IF YOU -- YOU CAN GET ACCESS TO ALL OF THIS INFORMATION BY JUST PUTTING IN ILRU WIKI, AND OUTCOME MEASURES. IN ANY SEARCH ENGINE. ILRUTIKI IN ANY SEARCH ENGINE. IT'LL BE ONE OF THE TWO THINGS THAT POPS UP, THE LINK OF THIS. EVERY AMOUNT OF DATA, EVERYTHING 29 WE'VE GRADED, INCLUDING THE OVERHEADS, IS IN THERE. SO HAVE YOU ACCESS TO IT. NOW, HOW DID WE GATHER THE INFORMATION? WE USED THIS APPROACH FOR ALL THREE OF OUR SOURCES. WE TRANSLATED EACH INDICATOR INTO ONE TO TWO SPECIFIC INTERVIEW QUESTION QUESTIONS, A REAL KEY STEP FOR US. FOR EXAMPLE, IF THE INDICATOR WAS TELL THE NUMBER OF PERCENT OF PERSONS WITH DISABILITIES CONTACTING THE CIL, WHO REPORT THEY HAVE THE INFORMATION THEY REQUESTED FROM THE CIL, THE QUESTION WE DEVELOPED IS: SOMETIMES WE'RE ABLE TO HELP PEOPLE GET THE INFORMATION THEY NEED FROM US AND SOMETIMES WE'RE NOT. IF YOU PERSONALLY, DID YOU RECEIVE THE INFORMATION YOU NEEDED FROM US? SO YOU SEE, WE JUST PUT IT -- THIS IS THE INTERVIEW QUESTION. SO WHEN THE PERSON WAS TALKING TO THE CONSUMER OR WHOEVER, THIS IS THE QUESTION THEY ASKED. -OF-ANOTHER EXAMPLE, THE INDICATOR WAS THE NUMBER AND PERCENT OF CONSUMERS SERVED BY THE CIL WHO CAN LIST AT ONE ONE SPECIFIC WAY IN WHICH THEY ARE NOW MORE INDEPENDENT THAN WHEN THEY APPROACHED THE CIL. TWO QUESTIONS. FIRST ONE WAS, SOMETIMES WE'RE ABLE TO HELP PEOPLE BECOME MORE INDEPENDENT, AND SOMETIMES WE'RE NOT. FOR YOU PERSONALLY, DO YOU THINK YOU'VE BECOME MORE INDEPENDENT, LESS INDEPENDENT, OR THE SAME US A WERE WHEN WE FIRST STARTED WORKING TOGETHER? THE SECOND QUESTION WAS, IN WHAT WAYS ARE YOU MORE INDEPENDENT THAN YOU WERE BEFORE WE STARTED 30 WORKING TOGETHER? SO WE ACTUALLY MADE PEOPLE IDENTIFY HOW THEY WERE MORE INDEPENDENT. OKAY? THEN AFTER THEY PUT THE INFORMATION INTO THE DIFFERENT QUESTIONNAIRES, AND THEN RETRIEVED THE CONSUMER DATA RELATED TO THE AGE, RACE, ETHNICITY, DISABILITY FROM THE FILES, RATHER THAN ASKING THE CONSUMER ABOUT IT. I THINK WE'LL JUST TAKE IT FROM THE FILES. SO IF YOU WERE SUCCESSFUL IN MAKING YOUR CONTACT WITH THE CONSUMER, YOU WOULD KNOW THAT AND PULL OUT INFORMATION FROM THE FILE. YOU MAY ONLY HAVE -- WE DID IT DIFFERENTLY THE FIRST YEAR. THE FIRST YEAR WHAT WE DID IS WE ASKED PEOPLE THAT INFORMATION, AND THEY COMPARED IT WITH THE INFORMATION FILE. AND BY THE SECOND YEAR WE FOUND IT WAS JUST UNNECESSARY TO DO THAT. WE JUST WENT AHEAD AND GATHERED IT THIS WAY. AGAIN, THIS WAS THE IDEA, WE DECIDED TO DO IT THIS WAY RATHER THAN ASKING THE CONSUMER, BECAUSE WE WANT TO REDUCE THE AMOUNT OF TIME WE HAD TO KEEP THEM ONLINE, SO WE'D MAKE THE PROCESS OF BEING INTERFERES LESS CUMBERSOME FOR THEM. THEN SPECIFIC INFORMATION ABOUT THE CONSUMERS: NOW, -- SO ON THE CONSUMERS, THE PERSON WITH A CSR, WE HAD TO DEVELOP A LIST OF ALL CONSUMERS IN JANUARY, SEPTEMBER 2010 WHO LEFT A PHONE NUMBER. WHERE HE RANDOMLY SAMPLED 25 OF THEM, CALLED EACH ONE AND ASKED TO TALK TO THEM, NOT SO EASY, BY THE WAY. 31 WE ASKED FOUR CLOSED-ENDED QUESTIONS, PLUS 0-4 OPEN-ENDED QUESTIONS. WHERE WE WROTE THE RESPONSES ON A PAPER INTERVIEW GUIDE. WE CALLED UNTIL WE 25 COMPLETED GUIDES. ON OUR PEOPLE, WE CREATED A LIST OF NONCSR IN OUR CLIENTS,-OF-OUR COLLARS WHO WERE TOYED BETWEEN JANUARY AND DECEMBER OF 2010. SOMETIMES THIS WAS NOT BE A TO DO THIS. WE RANDOMLY SAMPLED 25. THEY CALLED EACH ONE AND ASKED TO TALK TO THEM. THAT, AGAIN, WASN'T VERY EASY. AGAIN, THEY ASKED TWO CLOSED-ENDED QUESTIONS. BOTH IN RESPONSE TO THE GUY, WHO CALLED AND COMPLETED THE GUIDE. OKAY? NOW, THE THIRD ONE WAS GATHERING INFORMATION ABOUT THE CENTER FOR INDEPENDENT LIVING. THIS IS MOSTLY INFORMATION YOU GOT DIRECT OUT OF THE DOCUMENTS OR RECORDS. WE FILLED OUT THE INFORMATION GATHERING FORM, THE DOCUMENTS AND REPORTS, YOU ASSESSED THE CIL'S MIS, INTERVIEWING THE EXECUTIVE DIRECTOR, OTHERS. SOME OF THE CIL QUESTIONS ARE COMPLEX OF CONSUMERS AT RISK OF BEING INSTITUTIONALIZED. THE DEFINITION, WE'LL GO OVER THAT AGAIN. BUT THE DEFINITION FOR AT RISK IS IN THE TRAINING MANUAL. AND THEN THE TRAINING MANUAL, THE CALLS HELPED SOME BUT IT WAS NOT EASY. SO THIS IS WHAT WE DID. AND THIS WAS THE DIFFICULT PART OF IT. WE DID DO THIS ON NEXT TIME WE WOULD TALK BIT, RECORDING THE INFORMATION. WE TALK THAT INFORMATION FROM 32 THE FORMED AND THEN INTO A SURVEY MONKEY, SOMETHING WE CAN JUST AN LIES THE DATA ON. ANY QUESTIONS NOW? PLEASE IDENTIFY YOURSELF. >> Audience Member: DAN KESSLER FROM ALABAMA. BOB, DURING THE DEVELOPMENT PROCESS THIS PAST SEVERAL YEARS, DID THE QUESTION OR ISSUE COME UP OF THOSE PROGRAMS AND SERVICES, FOR EXAMPLE, LIKE THESE PERSONAL ASSISTANT SERVICES THAT MAY BE OFFERED, PERMITTING SERVICES, HOME MODIFICATION PROGRAMS. WHERE SOMEBODY COMES TO THE CENTER, THEY WANT TO RING UP, THEY DON'T WANT TO SEE YOU AGAIN. THAT'S WHAT THEY WANT FROM THE CENTER, CONSUMER CONTROL. WERE THE CENTERS ASKED TO SORT OF SELECT WHICH SORT OF POPULATION TO DRAW FROM OR SORT OF DIVIDE THEIR PROGRAMS UP INTO DIFFERENT -- >> Bob Michaels: WE GAVE THEM GUIDELINES AND SAID, HERE'S THE PERIOD YOU'RE LOOKING AT. IF YOU GAVE THAT PERSON THE CSR AND THEY FOLLOW WITHIN THAT PERIOD OF TIME, THEY WERE CONSIDERED REGARDLESS OF WHICH SERVICE THEY HAD RECEIVED. >> AS LONG AS THEY KNOW THE DEFINITION OF CSR. OKAY. >> Audience Member: PAT PUCKETT FROM GEORGIA. HOW DID YOU DECIDE ON 25? >> Bob Michaels: THAT WAS REALLY INTERESTING. HOW IT WAS DECIDED ON 25. THE FIRST YEAR WE DID 30 AND WENT DO TOWN 25 BECAUSE WE THOUGHT IT WOULD BE EASIER. BUT WE WILL PICK 30. WE WERE WRESTLING WITH HOW MANY WE NEEDED TO DO ON THIS. 33 I TALKED TO A PERSON WHO DOES A LOT OF RESEARCH FOR A MATHEMATIC GUY, WHICH IS A RESEARCH FIRM THAT RUNS OUT OF THE BELTWAY AND OUT OF WASHINGTON, D.C. THEY SEEM TO UNDERSTAND THEY HAD DONE A LOT OF RESEARCH LIKE THIS, ESPECIALLY PERSONS WITH DISABILITIES. THEY USED THE NO. 30. BASICALLY WHAT SHE SAID WAS, WE'VE FOUND IF YOU RANDOMLY SAMPLE 30 FOR EACH ONE OF THESE QUESTIONNAIRES, THEN YOU'LL BE FINE. SHE SAID, IF IT GETS MORE SPECIFIC AND YOU WANT TO KNOW THE NUMBER OF LEFT-HANDED FEMALES, FOR INSTANCE, THEN EACH TIME YOU START TO ADD IN A CHARACTERISTIC THAT YOU'RE GOING OH TO ANALYZE, YOU HAD TO DO ANOTHER 30. SO YOU WOULD DO ANOTHER 30 IF YOU WANTED TO JUST KNOW FEMALES, ANOTHER 30 IF YOU JUST WANTED TO KNOW LEFT-HANDED ONES. YOU WOULD END UP DOING MORE LIKE THAT. WE WENT WITH 30. AFTER THE FIRST YEAR WE FOUND THAT MOST OF THE PEOPLE HAD TAKEN A LOT OF TIME THERE, AND IT REALLY WASN'T ANY DIFFERENCE BETWEEN 25 AND 30. WE HAD MORE PEOPLE INVOLVED, MORE CENTERS INVOLVED THE SECOND YEAR. WE WILL HAVE GOOD DATA REGARDLESS OF THE NUMBER SO, THAT WAS KIND OF HOW WE CHOSE THAT. OKAY? MAUREEN? >> Audience Member: MAUREEN RYAN, WISCONSIN. I'M CURIOUS AS TO, WHEN YOU WERE ASKING, YOU ASKED ONE QUESTION ABOUT THE BEAM CONTACTED THE CIL AND DID THEY 34 GET THE INFORMATION. AND THAT WAS JUST A YES OR NO. BUT THEN WHEN YOU ASKED, ARE YOU MORE INDEPENDENT, YOU DID A FOLLOW UP QUESTION AS TO HOW. AND WHILE YOU DIDN'T ASK THAT FOR THE FIRST QUESTION, WHAT WAS THE INFORMATION? >> THE FIRST QUESTION WAS A QUESTION OF THE I&R PEOPLE, PEOPLE WHO CALLED IN FOR I&R. THE SECOND QUESTION WAS ONE FOR CONSUMERS. WE FOUND IT WAS REALLY IMPORTANT THAT CONSUMERS UNDERSTAND AND BE ABLE TO SPECIFICALLY SAY HOW THEY HAD BENEFITED FROM SERVICES. SO WE WERE MORE SPECIFIC THERE. BUT THERE WAS A LOT OF QUESTIONS ON WHAT WE EVEN DO BEHIND OUR PEOPLE. I THINK WHAT YOU'RE SAYING IS WE MIGHT WANT TO GO THERE AT SOME POINT. WE JUST WERE KIND OF FILLING OUT, YOU HAD HAD A QUESTIONNAIRE WE'RE CONCERNED ABOUT. . WHAT WE'RE CONCERNED B AND RIGHTFULLY SO, PEOPLE CALLING IN I&R. PEOPLE WOULDN'T REMEMBER HOW THAT IMPACTED THE CENTER. THAT WAS PROBLEM FROM THE CALL ITSELF. SO WE WERE LESS SPECIFIC THAN WE NEEDED TO BE. DO YOU REMEMBER ANYTHING ELSE, MIKE? >> Mike Hendricks: LET ME JUST MENTION ONE MORE THING. THAT'S A REALLY GOOD QUESTION. KIND OF WHAT YOU'RE SAYING IS, HOW COME ON THE ONE, WHEN THEY HAVE A FOLLOW-UP QUESTION WHERE WE WERE TALKING YESTERDAY AND WE WERE KIND OF TOUGH ON OURSELVES, WHERE WE ACTUALLY HAD TO LIST SOMETHING, AND ON THE OTHER ONE 35 WE WEREN'T SO TOUGH ON OURSELVES. WE WERE A LITTLE TOUGHER THAN WE REALIZE. THE SHEET ON THE HANDOUT CALLED THE INDICATORS MEASURES, LISTING THE EIGHT OUTCOMES AND THE 11 INDICATORS. UNDER THE OUTCOME, PERSONS WITH DISABILITIES GET THE INFORMATION THEY NEED. WE DID HAVE THAT ONE INDICATOR YOU'RE JUST NOW MENTIONING AND BOB MENTIONED EARLIER. WHO REPORT THEY HAD THE INFORMATION THEY REQUESTED. BUT NOTICE THERE'S A SECOND INDICATOR THERE, TOO. FOR US THEY HAD TO BE ABLE TO SAY THAT THEY HAD USED THAT NEW RESOURCE. YOU MIGHT SAY THAT'S ACTUALLY TOO TOUGH ON OURSELVES. BUT WE THOUGHT SAYING THEY GOT IT WASN'T GOING TO BE RIGOROUS ENOUGH FOR US. WE WERE GOING TO BE HARDER ON OURSELVES THAN THAT. WE WERE GOING TO HOPE THEY WOULD ALSO SAY THAT THEY HAD USED IT. IF THEY DIDN'T SAY THEY USED IT IT WAS GOING CAST SOME DOUBT. SO WE KIND OF FOLLOWED UP WITH THAT ONE IN A DIFFERENT WAY, NOT AN IMMEDIATE FOLLOW-UP QUESTION, BUT WITH ACTUALLY A SECOND INDICATOR. I JUST WANTED TO POINT THAT OUT. >> Bob Michaels: WE'RE BETTER THAN I THOUGHT. SO ANYTHING ELSE? DOES THAT ANSWER YOUR QUESTION, MAUREEN? >> Audience Member: HOW DID YOU FIND OUT IF THEY USED IT? I MISSED THAT. >> Mike Hendricks: AGAIN, THAT'S THAT QUESTION ABOUT ARE YOU GOING TO RELY ON SELF-REPORT OR TRY TO FIND AN OBJECTIVE WAY TO 36 CHECK. FOR THIS FIELD TEST, WE RELIED ON SELF-REPORT. IS THAT IDEAL FOR THE FUTURE? PROBABLY NOT. WE SHOULD PROBABLY WORK ON THAT BUT THAT'S WHERE WE ARE RIGHT NOW. LET ME MENTION ONE OTHER THING, IF I CAN. I SAW SO MANY PEOPLE NOT RAISE THEIR HANDS WHEN WE ASKED IF YOU'RE COLLECTING OUTCOME INFORMATION. I WONDER IF THE PORTION OF THIS, HAVE WE CON VAYED THE IMPORTANCE OF THIS. IF WE WERE GATHERING HERE 15 MONTHS AGO, AND YOU WERE TO ASK US, HAVE ANY FILLS AROUND THE COUNTRY SYSTEMATICALLY GATHER THE OUTCOME INFORMATION, WE'D SO NO. BUT TODAY WE'D STAND HERE AND SAY YES. 28 CILs FOR THE VERY, VERY FIRST TIME, HAVE FOUND A WAY, AS HE'S BEEN EXPLAINING TO YOU, GONE OUT AND DONE IT. IN A LITTLE BIT HE'S GOING TO GIVE YOU THE RESULTS. YOU ARE THE FIRST PEOPLE ANYWHERE WHO ARE GOING TO SEE THE RESULTS. NO ONE ELSE HAS SEEN THEM YET. IT'S REALLY A BIG THING THAT'S HAPPENED. KIND OF LIKE PIONEERING WHAT YOU WILL HAVE TO BE DOING HOPEFULLY BETTER BUT AT LEAST THE SAME IDEAS. >> Bob Michaels: JULIA. >> JULIA, CHARLOTTE, NORTH CAROLINA. I WOULD JUST LIKE TO SAY THAT OUR CENTER HAS BEEN DOING OUTCOME MEASURES BASED ON THE BOOK YOU WROTE SINCE 1995, BECAUSE WE HAVE -- WE HAD TO BECAUSE WE ARE A UNITED WAY 37 MEMBER AGENCY. AND I THINK THAT MIGHT NOT HAVE BEEN CONVEYED WELL TO THE PEOPLE ON THE TASK FORCE, BUT SOME OF US HAVE BEEN DOING IT FOR A VERY LONG TIME. IF YOU THINK THAT 15 MONTHS AGO YOU WOULD HAVE SAID NO, THAT WOULD HAVE BEEN OUR LACK OF SHARING WITH YOU THE FACT THAT A LOT OF US WHO ARE UNITED WAY, MEMBER AGENCIES HAVE BEEN BEEN DOING IT. >> I EITHER SPOKE BADLY OR TOO QUICKLY. LET ME CORRECT IT. IN THIS PAPER THAT BOB AND I HAVE WRITTEN, WE EXPLICITLY SAY UP FRONT THAT NUMBER OF CILs ON THEIR OWN WERE OUT THERE DOING THIS. WHAT I TRIED TO SAY A SECOND AGO WAS, ON A NATIONAL LEVEL SYSTEMATICALLY, ALL GATHERING THE SAME STUFF. THAT'S WHAT HASN'T HAPPENED. IN OTHER WORDS THIS, WONDERFUL WOMAN FROM RSA COMING HERE, WE WOULD HAVE SAID TO HER, AS A NATIONAL PROGRAM WE DON'T HAVE ANYTHING FOR YOU. NOW WE CAN SAY, AS A NATIONAL PROGRAM, WE'VE STARTED AT LEAST. SO THAT'S WHAT I MEANT TO SAY. YES, IN FACT, WE LEARNED A LOT FROM INDIVIDUAL SILs AND WHAT THEY HAD BEEN DOING AND BUILT THAT INTO OUR WORK. >> Bob Michaels: OKAY. ANYTHING ELSE? OH, HERE, PAT'S GOT ONE. >> Audience Member: I'M NOT SURE IF WHAT WE -- THE NEWEST ADDITIONS TO THE 704 REPORT, THE ACCESS TO TRANSPORTATION, HEALTH CARE AND AT ARE IN FACT OUTCOMES. BUT IF THOSE ARE OUTCOMES, THEN ALL CILs HAVE BEEN COLLECTING THAT. 38 OR HAVE BEEN STICKING SOME NUMBERS IN THERE, I DON'T KNOW WHICH. BUT I HAVE A FEELING IT'S PROBABLY A LITTLE BIT OF BOGUS OUTCOME REPORTING BUT I DON'T KNOW THAT FOR SURE. I'M NOT A CIL, SO -- >> Bob Michaels: YOU KNOW, I THINK THAT PROBABLY REPORTING IN A SELF-REPORT, HOW DO YOU IN TERMS OF EFFICACY, OR IN TERMS OF TRANSPORTATION, AT AND HEALTH CARE, YOU KNOW, SOME OF YOU PROBABLY ARE GIVING ACCURATE INFORMATION, SOME OF YOU ARE GIVING THE BEST YOU CAN GIVE. BUT CLEARLY WHAT THEY HAVE DONE IS THEY HAVE JUMPED PAST THE PROCESS THAT WE WENT THROUGH, AND DECIDED THESE WERE THE THREE MOST IMPORTANT THINGS IN YOUR COMMUNITY THAT YOU NEED TO EMPHASIZE THROUGH MOST IMPORTANT SERVICES. SO YOU KNOW, WHEN I FIRST HEARD TALK ABOUT OUTCOMES, EVERYBODY CAME AND SAID, YOU'VE GOT A LOGIC PROBLEM. OH, YEAH, THEY HAVE GOT A LOGIC MODEL. >> THEY WOULD SAY, OH, YEAH, THEY HAD NO IDEA WHAT A LOGIC MODEL WAS, WHY WE NEED TO DO IT OR ANYTHING ELSE. BUT NOW THAT WE'VE DONE IT, YOU CAN REALLY SEE THAT THE PROCESS WE'RE SUGGESTING PEOPLE GO THROUGH IS MORE LEGITIMATE IN TERMS OF REALLY IDENTIFYING WHAT IT IS THAT YOUR COMMUNITY WANTS YOU TO DO, AND HOW YOU GO ABOUT DECIDING THAT IT IS GOING TO BE A PRIORITY AND WHAT YOU NEED TO DO TO MAKE IT A PRIORITY AND TO GET DECISION MAKERS TO AGREE WITH YOU. SO IT MAY IN FACT BE TRANSPORTATION, AT AND HEALTH CARE BUT IT MAY NOT BE. 39 IT MAY BE HOUSING, EDUCATION AND SOMETHING ELSE. SO WE ALL NEED -- I THINK THAT'S THE MESSAGE WE'RE GETTING ACROSS IS THIS IS REALLY KIND OF REFLECTED, EVERY CENTER IS DIFFERENT, EVERY COMMUNITY IS DIFFERENT AND YOU'LL BE DEVELOPING SERVICES THAT THE COMMUNITY WANTS, NOT WHAT THE BUYER ROCRACY WANTS, CLEANING IT UP A LITTLE BIT. RICHARD? >> Audience Member: I WANTED TO ASK CAROL IF WE HAVE WEB QUESTIONS. >> Audience Member: NO, WE DON'T HAVE ANY WEB QUESTIONS AT THIS TIME. >> Bob Michaels: THANK YOU. OKAY? >> Mike Hendricks: WE HAVE ONE HERE, I'M SORRY. >> Bob Michaels: ONE MORE QUESTION. >> Audience Member: KARLA LAW SON, TULSA, OKLAHOMA. WE'VE BEEN A UNITED WAY AGENCY SINCE 1989. WE'VE BEEN PUSHED TO DO A -- TREMENDOUS MORE FOLLOW-UP QUESTIONS AND SURVEYS THAN THE 25 TO 30. CAN YOU GIVE US SOME MORE INFORMATION FROM THAT COMPANY IN DEFENSE OF THIS? WE HAVE TRIED EVERY COUNSELOR DOING THE FOLLOW UP AND IT HAS REALLY MADE A DIFFERENCE. THERE WAS A LOT OF RESISTANCE AT FIRST BUT IT HAS HELPED US IMPROVE THE PROGRAMS. AND OUR REFERRAL SOURCE, WE WERE REFERRING TO A PARTICULAR AGENCY THAT DIDN'T HAVE A PERSON DEDICATED TO ANSWER THE PHONE. YOU GOT SERVICE IF YOU WENT THERE. WE WOULDN'T HAVE GOTTEN THAT INFORMATION IF WE DIDN'T DO 40 THE FOLLOW UP. BUT I'M REALLY STRUGGLING WITH THE NUMBER 25 OR 30. >> Mike Hendricks: YEAH, WELL, AS BOB SO CORRECTLY MENTIONED, IT DEPENDS ON WHAT YOU WANT TO DO WITH THE INFORMATION AFTER YOU GET IT. IF WHAT YOU CARE ABOUT IS HOW YOUR ENTIRE CIL IS DOING ON SOMETHING, THEN THE BEST ADVICE WE COULD GET FROM A STATISTICS WAS THAT SOMEWHERE 25, 30 WAS FINE. IF YOU FOR INSTANCE -- OR MAYBE HAVE YOU SOME DIFFERENT CENTERS OR LOCATIONS OR DIFFERENT TYPES OF SERVICES OR YOU WANT TO BREAK THAT DOWN OR DISAGGREGATE IT IN SOME WAYS, SAY YOU WANT TO DISAGGREGATE IT IN THREE-WAYS, YOU'VE GOT 10, 10, 10, THAT'S A PRETTY SMALL NUMBER. DEPENDING ON HOW YOU WANT TO USE IT YOU MAY HAVE TO UP THOSE NUMBERS. WITHOUT KNOWING YOUR SITUATION IT'S HARD TO KNOW WHERE THEY WERE SUGGESTING MORE THAN 30. I HAVE BEEN INVOLVED IN THE UNITED WAY A LOT. IN FACT, I'LL MAKE THIS OFFER AND I'LL PROBABLY REGRET IT BUT I'LL MAKE THIS OFFER. A COUPLE OF THE KEY PEOPLE AT UNITED WAY OF AMERICA AND I WROTE AN ARTICLE A COUPLE YEARS BACK, BASICALLY LOOKING BACK OVER THE WHOLE UNITED WAY OUTCOMES EFFORT. I HAD AN ELECTRONIC COPY OF THAT. I THINK IT'S NOT BAD ACTUALLY, AND I'LL SHARE IT WITH ANYBODY WHO WRITES ME AND WANTS I. HOW'S THAT? ONE OF THE THINGS WE SAY IN THERE, THE UNITED WAY SYSTEM IS NOT LIKE McDONALD'S WHERE A HAMBURGER HERE IS A HAMBURGER 41 THERE. IT'S MORE LIKE RE/MAX. EACH FRANCHISE INDEPENDENTLY OWNED AND OPERATED, THAT'S RE/MAX'S SLOGAN. YOUR UNITED WAY AND SOMEBODY ELSE'S UNITED WAY COULD HAVE A COMPLETELY DIFFERENT TAKE ON THIS. I'VE SEEN SOME WONDERFUL UNITED WAYS AND SOME I WISHED DIDN'T HAVE THE TAKE THEY HAVE. LET'S TALK PRIVATELY. BUT ANYWAY, THE GENERAL LESSON FOR EVERYBODY IS THAT THOSE NUMBERS DEPEND ON WHAT YOU WANT TO DO WITH THE INFORMATION. >> Bob Michaels: FOR THOSE OF YOU WHO PROBABLY DON'T KNOW THIS, I WAS RECENTLY ELECTED AS CHAIR OF THE SILC IN ARIZONA. ONE OF THE THINGS WE'VE BEEN DOING IS TRYING TO COORDINATE THE CONSUMER SATISFACTION SURVEYS ACTUALLY AFTER YOUR PRESENTATION LAST YEAR, MIKE. AND THEN WE'VE BEEN TRYING TO DO THAT. ONE OF THE THINGS WE'RE TRYING TO DO IS DECIDE JUST HOW MANY PEOPLE WE NEED TO INTERVIEW. AND SO WE ARE LUCKY BECAUSE WE HAVE A PROBABILITY PSYCHCOLOGIST ON OUR SILC. SHE HAS ACTUALLY GONE IN AND SAID, WELL, IF YOU'RE CENTER HAS 500 AND SOME CONSUMERS, -- AND SHE'LL GIVE YOU A NUMBER, AND IT'LL SAY, YOU KNOW, 5%, 10%, 0% ACCURACY. THEN IF YOU WANT TO KNOW SOMETHING WYNN WITHIN 10%, THERE WAS A 90% CHANCE THAT THIS WAS TRUE OR SOMETHING LIKE THAT, OR A 95% CHANCE THIS WAS TRUE, EVEN IF YOU JUST BASED IT ON THE NUMBER OF CONSUMERS. SHE GAVE ME ACTUALLY LIKE A MATRIX. WE WILL HAVE TO AT SOME POINT 42 DECIDE, HOW ACCURATE DO WE WANT TO BE IN IDENTIFYING YOUNG PEOPLE. SITE VARIES DEPENDING ON HOW ACCURATE YOU WANT TO BE, AND HOW LONG IS -- HOW NET IS IT FOR YOU TO GATHER THE CURRENT INFORMATION YOU NEED. BUT THAT'S FROM -- WHAT WE KNOW IS WE'VE DONE A FIELD TEST AND THIS IS WHAT WE'VE GOT, AND I'M GOING SHARE THE RESULTS WITH YOU SHORTLY. OKAY? ANYTHING ELSE? YOUR TURN. >> Mike Hendricks: IF NOT WHAT, WE'D LIKE YOU TO DO FOR THE NEXT 10 MINUTES IS JUST TALK ABOUT THIS A LITTLE BIT AT YOUR TABLE. THERE'S A LOT OF STUFF WE'VE THROWN OUT AT YOU, THE WHOLE PROCESS OF GATHERING IT. YOU'RE NOT GOING ANSWER ALL OF THEM BUT START WRESTLING WITH SOME OF IT. SEE WHAT'S ON YOUR MIND. HOW CAN YOU BEST GATHER OUTCOME INFORMATION BACK AT YOUR OWN CIL. THOSE ONLINE, TALK BIT TOO. THERE ARE SOME IMPORTANT QUESTIONS YOU NEED TO START GETTING THROUGH YOUR HEAD. DO THAT AND WE'LL LET YOU KNOW WHEN BREAK HAS COME. >> Mike Hendricks: FOLKS, WE ARE OFFICIALLY ON BREAK. WE START BACK AGAIN AT 10:45. THANK YOU SO MUCH.