ROUGH DRAFT 9-14-11, Outcome Measures for Centers for Independent Living – An IL NET Resource Presented by ILRU GOOD AFTERNOON. THIS IS A SECTION WE'RE CALLING VOICES FROM THE FRONT LINE. THE THREE PEOPLE SITTING HERE WITH ME ARE PEOPLE WHO ACTUALLY ARE PARTICIPATING IN THE FIELD TEST, AND THEY ARE GOING TO SHARE WITH YOU THEIR 117 EXPERIENCES. LET ME INTRODUCE THEM. TO MY LEFT IS DAISY FEIDT, FOR AXIS LIVING, A CENTER IN CHICAGO. TO MY IMMEDIATE RIGHT IS MICHELLE WAKELEY FROM NORTHERN WEST VIRGINIA. TO MY FAR RIGHT IS APRIL REED FROM THE ARIZONA CENTER FOR INDEPENDENT LIVING. ONE PERSON WHO WAS GOING TO SPEAK WITH US HAS RUN INTO SOME PROBLEMS. AT THE LAST MINUTE THEY HAD TO PULL OUT FINANCIALLY. EACH OF THESE PEOPLE ARE GOING TAKE 10 MINUTES AND THEY ARE GOING TALK A LITTLE BIT ABOUT THEIR EXPERIENCE. AND THEN WE WILL OPEN IT UP FOR QUESTIONS FROM YOU. SO WHY DON'T WE START WITH DAISY. >> Daisy Feidt: I'M DAISY FROM AXIS LIVING IN CHICAGO. WE PARTICIPATED IN THE SECOND YEAR OF FIELD TEST, SO NOT THE FIRST. WE ONLY HAVE ONE YEAR UNDER OR BELT. I'M GOING TO TRY TO TALK A LITTLE BIT ABOUT WHAT WE GOT OUT OF IT AND WHAT WE LEARNED. IN TERMS OF THE AMOUNT OF TIME AND EFFORT, IT TOOK -- I FELT LIKE FOR THE MOST PART IT WAS REALLY DOABLE. BUT KEEP IN MIND, THAT WAS IN THE CONTEXT OF HAVING THIS FRAMEWORK ALREADY GIVEN TO US AND THE SURVEYS ALREADY DEVELOPED. ALSO WE HAD BOB AND MIKE TO ANALYZE THE DATA FOR US. SO I THINK THOSE ASPECTS OF IT MADE EIGHT LOT MORE DOABLE. THE THING FOR US THAT WAS DIFFICULT, WHEN WE NEEDED 25 SURVEYS COMPLETED, WE NEED TO DO 118 ACTUALLY ASK A LOT MORE -- OR CALL LOT MORE CONSUMERS THAN THAT. OFTEN CALLED CONSUMERS MULTIPLE TIMES TO GET INFORMATION BACK. THAT PART TOOK A LOT MORE TIME THAN I ANTICIPATED. BUT FOR THE MOST PART I FELT LIKE IT WAS PRETTY DOABLE. WEIGHED FEW HUNDRED WORKING AT THE CENTER SO THAT HELPED US TO DIVIDE UP THE WORK A LITTLE BIT, MADE IT MORE DOABLE. OUR REASONS FOR DOING IT WERE 1 HUH% SELF-SERVING A I DO THINK WE GOT A GOOD AMOUNT OUT OF IT. FOR US IT WAS KIND OF A TEST RUN TO SEE HOW IT WORKS BEFORE THE RESULTS THEMSELVES ACTUALLY MATTERED THAT MUCH. I FEEL LIKE DO THIS KIND OF THING OR SOMETHING LIKE IT IS I AM INDEPENDENT, AND IT WAS WAY FOR US TO DO IT BEFORE WE ACTUALLY HAD TO SHOW THE RESULTS TO FUNDERS. I ALSO WANTED THE CHANCE TO SEE HOW WE WERE DOING COMPARED TO THE REST OF THE FIELD. IF WE WEREN'T DOING AS WELL, TO HAVE A CHANCE TO FIGURE OUT WHY AND CORRECT IT. AGAIN, THE FACT THAT WE WERE IN THIS PROCESS GIVEN A FRAMEWORK ALREADY, IT JUST FELT LIKE MUCH MORE MANAGEABLE WAY TO KIND OF GET INTO OUTCOMES MEASUREMENT THAN IF WE WERE DOING EVERYTHING ALL ON OUR OWN. IN TERMS OF WHETHER WE USE THE DATA: WE JUST GOT THE RESULTS BACK AT THE END OF MAY. WHAT WE'VE DONE SO FAR IS SHARED THE RESULTS WITH MANAGERS. WE HAVE A QUALITY IMPROVEMENT TEAM THAT HAS ALSO LOOKED AT THE DATA AND THE AREAS WHERE WE DID WELL AND AREAS WHERE WE NEED TO PROBABLY DO A LITTLE BIT BETTER. WE'RE SORT OF WRESTLING WITH 119 WHAT TO DO NEXT, ESPECIALLY AMONG THIS QUALITY IMPROVEMENT TEAM. FOR EXAMPLE, ONE OF THE AREAS FOR US, WHERE WE WERE MUCH LOWER THAN OTHER CILs, WAS WHETHER I&R CALLERS GOT WHAT THEY WANTED. AXIS LIVING WAS AT 48% AND OTHERS WERE AT 72%. THAT DATA WAS REALLY ALARMING TO ME. I THINK WE REALLY NEED HOW TO IMPROVE THAT OUTCOME. BUT IT'S A LITTLE BIT HARD TO FIGURE OUT -- I MEAN, NOW WE HAVE TO FIGURE OUT, OKAY, WHY IS THAT? WHY DID WE GET THAT RESULT? AND WHAT ARE WE GOING TO CHANGE IN ORDER TO GET A BETTER RESULT. I THINK THAT'S A DIFFICULT QUESTION AND THIS QUALITY IMPROVEMENT TEAM HAS A LOT OF DIFFERENT IDEAS ON WHY WE MIGHT HAVE GOTTEN THAT RESULT. THE NEXT STEP FOR US IS TO ACTUALLY DO SOME FOLLOW-UP WITH PEOPLE WHO HAVE GOTTEN OUR SERVICES AND ASK MORE IN-DEPTH QUESTIONS TO TRY TO GET AT, WHY AREN'T THEY GETTING WHAT THEY NEED. WE HAVE NOT YET TAKEN THAT STEP BUT WE'RE PLANNING TO. THE NEXT QUESTION, OKAY, WHAT ARE YOUR OVERALL FEELINGS ABOUT MEASURING CIL OUTCOMES? I THINK FOR US WE FEEL LIKE IT'S A REALLY VALUABLE PROCESS A REALLY GREAT TOOL TO BE ABLE TO USE FOR OUR CENTER. AT THE SAME TIME, IT REALLY FEELS LIKE IT TAKES A WHOLE CULTURE SHIFT WITHIN THE ORGANIZATION. YOU KNOW, IT SEEMS LIKE EVALUATION MEASUREMENT IS KIND OF EVERYBODY'S JOB AT THE CIL BUT ALSO NOBODY'S JOB AT THE 120 CIL. SO TRYING TO FIGURE OUT THAT PIECE OF IT IS REALLY DIFFICULT, ESPECIALLY AT A LARGE CENTER LIKE ACCESS LIVING. ON WHAT FYING I WOULD HAVE FOR OTHER CILs, WE HAVEN'T DONE ENOUGH TO MAKE ME FEEL LIKE I'M REALLY QUALIFIED TO GIVE ADVICE TO THE REST OF YOU. I THINK WE HAVE THIS QUALITY IMPROVEMENT TEAM THAT'S SET UP WITHIN THE CENTER. FOR US THAT'S ABOUT AN REALLY USEFUL TOOL. SO I GUESS I WOULD SAY THAT, I THINK, FOR US TOO, ONE OF THE THINGS WE DEFINITELY REALIZED AROUND THIS PROCESS, THE DATA GOING INTO YOUR MIS SYSTEM HAS TO BE GOOD DATA OR IT'S GOING TO BE REALLY DIFFICULT TO GET ACCURATE OUTCOMES. SO THAT'S A PIECE WE'RE WORKING ON TO DO SOME SPOT-CHECKING OF THE THAT THE GOES INTO OUR MIS SYSTEM SO WE CAN HAVE BETTER DATA INFORMATION THERE. I THINK THAT'S ALL I WANTED TO SAY. >> Bob Michaels: GOOD. THANK YOU. MICHELLE? >> Michelle Wakeley: THANK YOU. AGAIN, MY NAME'S MICHELLE, I'M FROM THE NORTHERN WEST VIRGINIA CENTER FOR INDEPENDENT LIVING. WITH THE FIRST QUESTION, HOW MUCH TIME AND EFFORT DID THE FIELD TEST TAKE: WE PROBABLY HAD MYSELF, ONE OTHER PERSON AND THEN TWO PEOPLE WHO SORT OF HELPED OUT WORKING ON THIS PROJECT. I'D SAY BETWEEN THE FOUR OF US IT PROBABLY TOOK US A COMBINED MAYBE ABOUT 200 HOURS TO DO. BUT THAT WAS SOME OF US, TOO. WE'VE NEVER DONE THIS BEFORE AND WE DIDN'T DO THE FIRST YEAR. 121 THIS WAS OUR FIRST GO WITH IT. WE HAD A LITTLE BIT OF A LEARNING CURVE. I DON'T THINK IT WOULD BE THAT WAY IF WE DO IT AGAIN. IT DID TAKE A LOCALITY OF EFFORT ON OUR PART JUST SORT OF GETTING OUR MINDS WRAPPED AROUND WHAT WE WERE DOING. I THINK THAT WE ARE VERY, VERY CONCERNED ABOUT DOING GOOD, BUT UNDERSTANDING WHAT "GOOD" MEANS IS SOMETIMES DIFFICULT AND PROBABLY WHY WE'RE HERE. ONE THING THAT SAVED US A LOT OF TIME IS THAT WE DO HAVE A DATABASE SYSTEM THAT WE WERE ABLE TO EXTRACT INFORMATION FROM, PULL DOWN NAMES, AND IT WAS VERY, VERY QUICK. YOU KNOW, MAYBE A FEW YEARS AGO WE DIDN'T HAVE THAT SYSTEM, IT WOULD HAVE TAKEN US A LOT MORE TIME TO DO THAT MANUALLY. HAVING A GOOD SYSTEM IN PLACE TO BEGIN WITH MADE ALL THE DIFFERENCE IN OUR TIME. WE ARE A SMALL CENTER. WE HAVE A VERY SMALL STAFF, AND I THINK THE WAY THAT WE FELT AT THE TIME WAS THAT, LIKE DAISY HAD SAID, THIS ISN'T ANYBODY'S JOB. SO WE'RE DOING IT IN ADDITION TO WHAT WE'RE ALREADY DOING. WHEN YOU SAY IT LIKE THAT, IT SOUNDS NEGATIVE BUT IT'S NOT. IT REALLY IS A PART OF WHAT WE DO, IT IS WHAT WE DO, AND WE HAVE TO UNDERSTAND AND EVALUATE AND REEVALUATE WHAT WE'RE DOING, IF WE'RE GOING TO DO WHAT WE DO. OF I THINK THAT'S A PART OF WHAT WE GOT OUT OF IT WAS A BETTER UNDERSTANDING OF WHAT WE'RE DOING, HOW IT'S IMPACTING THE PEOPLE THAT WE CARE ABOUT MOST. IT REALLY IMPRESSED THE IMPORTANCE OF GOOD DATA COLLECTION, LIKE DAISY SAID. 122 IT MADE YOU MORE AWARE OF THE QUALITY OF INFORMATION WE WERE PUTTING IN BECAUSE WE HAD TO GET IT BACK OUT AND ACTUALLY LOOK AT IT AGAIN. WE'RE NOT GIVING IT TO SOMEBODY TO THROW ONTO THE 704. IT REALLY ALSO LET US SEE WHAT WE'RE DOING, BESIDE WHAT WE INTEND TO DO, WHAT WE SAY WE'RE GOING TO DO. YOU SORT OF HAVE GOT THAT SIDE BY SIDE. WHAT HAVE WE DONE WITH OUR RESULTS? I THINK THAT WHEN WE GOT THE RESULTS BACK IT WAS EXTREMELY HELPFUL TO SEE HOW WE STOOD BESIDE THE OTHER CENTERS, HOW EVERYONE ELSE WAS DOING. AND TO SEE IF WE MEASURED UP, HOW ARE WE DOING. OUTSIDE OF THAT, I THINK THAT IT WILL IMPROVE OUR DATA COLLECTION. IT WILL IMPROVE HOW WE PUT DATA IN. AND THEN I HOPE THAT THIS CAN ALSO BE INFORMATION THAT WE CAN USE FOR FUNDING REQUESTS, AND THINGS LIKE THAT, THAT AGAIN, WE DO GOOD. IF I SAY THAT I'M GOOD AT BOWLING, IT DOESN'T TELL YOU VERY MUCH. I MAY BOWL AN 80 EVERY TIME. BUT IF I CAN TELL YOU THE NUMBER, THEN IT TELLS YOU A LITTLE MORE WHEN. WE TELL OUR FUNDERS WE'RE DOING A GOOD JOB, EVERYTHING'S WONDERFUL, IT'S NOT ENOUGH. IT REALLY HELPS US BE ABLE TO TELL THEM IN A DEFINITIVE WAY HOW GOOD WE'RE DOING. WE'RE NOT. I THINK OUR ENTRIES WERE VERY HONEST. THAT'S SORT OF DIFFICULT WHEN YOU'RE CALLING PEOPLE AND YOU 123 HAVE A VESTED INTEREST TUBE PRESENTED IN A GOOD LIGHT. THAT WASN'T OUR INTEREST. WE REALLY WANTED TO SEE HOW ARE WE DOING. WE DID GO THROUGH SEVERAL CONSUMERS AND I&Rs TO GET OUR 25. WE'VE DEVELOPED A SPREADSHEET, KEPT TRACK OF EVERYONE. THE WORKSHEETS WERE SO VERY HELPFUL. SO IT TOOK A LITTLE BIT OF TIME AND IT TOOK GOING THROUGH A LOT OF PEOPLE. BUT ULTIMATELY WE WERE ABLE TO GET OUR 25 WITHOUT MUCH OF A PROBLEM. I THINK THAT AT LEAST WITH OUR CENTER DIFFERENT STAFF MEMBERS HAVE DIFFERENT STRENGTHS. I THINK WHO YOU HAVE ON YOUR TEAM IS EXTREMELY IMPORTANT. I TEND TO BE POSSIBLY A LITTLE MORE TECHNICALLY SAVVY THAN SOME OF THE OTHER STAFF MEMBERS, WHERE SOME OF THE OTHER STAFF MEMBERS ARE MAYBE A LITTLE MORE PATIENT AND DON'T MIND GOING THROUGH 30 CALLS TO GET 10 PEOPLE. BECAUSE WE WOULD CALL PEOPLE AND ONE OF OUR MAJOR DIFFICULTIES WAS, WE WANTED TO TALK DIRECTLY TO THE CONSUMER, RIGHT? WE WANTED IT DIRECTLY FROM THE CONSUMER, CONSUMER-STATED. WE WOULD CALL AND MAYBE HEALTH CARE WORKER OR SOMEONE WOULD ANSWER AND WOULD SAY, I'M SORRY, BOB'S DEAF. HE'S NOT GOING TO BE ABLE TO TALK TO YOU. AND IN OUR WORLD, THAT'S NOT SUFFICIENT. BOB CAN STILL COMMUNICATE WITH US. A LOT OF PEOPLE THOUGHT THAT WE WERE TELEMARKETERS. SO WITH THE SCRIPT I TENDED TO 124 DEVIATE A LITTLE BIT. AND BE A LITTLE MORE PERSONABLE SO I DIDN'T AUTOMATICALLY GET SOMEONE HANGING UP ON ME. IT WAS AT TIMES VERY DIFFICULT FOR A FAMILY MEMBER OR A HOME HEALTH WORKER OR SOMETHING LIKE THAT TO LET US TALK TO THE CONSUMER DIRECTLY. WE WERE ABLE TO GET AROUND THAT BUT THAT WAS ONE OF THE DIFFICULTIES WE HAD. OVERALL, I THINK THAT DATA COLLECTION IS INHERENTLY DIFFICULT. IT'S NOT A PASSION FOR MOST OF US. SO IT'S INHERENTLY DIFFICULT, BUT ALSO ESSENTIAL, ESPECIALLY IN THIS TIME, THAT WE HAVE TO BE ABLE TO PRODUCE MEASUREMENTS FOR WHAT WE'RE DOING. WE CAN SEE THE FRUIT IN THE TREES. MY ADVICE, AGAIN, I DON'T HAVE A WHOLE LOT BECAUSE I HAVEN'T BEEN DOING THIS MUCH. BUT MY ADVICE WOULD BE THAT IT'S WORTH THE TIME, IT'S WORTH OF INTEREST. IT MAY NOT SEEM LIKE IT BUT I THINK -- WHILE YOU'RE DOING IT, WHAT AM I DOING THIS FOR? WE HAD SOME PROBLEMS WITH OUR INTERNAL DATA TRYING TO FIND OUT SOME OF THE OTHER QUESTIONS FROM THE CIL DATA COLLECTION, NOT THE CONSUMER OR THE IN-HOUSE. THIS IS TAKING LOT OF TIME AND ENERGY, WHY ARE WE DOING THIS AGAIN? WHEN WE GOT THE DOCUMENTS BACK IT WAS VERY EASY FOR US TO ANSWER THAT AND SAY, OKAY, THIS IS WHAT WE WERE LOOKING FOR. WE NEED TO FIGURE OUT AND TO BE ABLE TO EVALUATE HOW WE'RE DOING. SO THAT WOULD BE MY ADVICE, THAT IT'S WORTH THE REVIEWING YOUR 125 EFFECTIVENESS AND EVALUATING YOUR PROGRAMS. THANKS. >> Bob Michaels: APRIL? >> April Reed: I'M APRIL REED AND I'M FROM PHOENIX. THIS WAS ALSO OUR FIRST YEAR OF PARTICIPATING. WE WERE VERY LUCKY IN THAT WE HAD SOME VOLUNTEERS THAT WERE ABLE TO ASSIST US. WE HAD AN AMERICORPS MEMBER WITH A MASTER'S IN ONE OF OF OUR PEER MENTORS WAS A MASTER'S IN SOCIAL WORK. WE WERE ABLE TO PULL IN SOME OF THOSE PEOPLE TO ASSIST US. THAT WAS VERY HELPFUL. OVERALL WE HAD THREE STAFF THAT WORKED ON THIS WITH US. AND THEN VOLUNTEERS. AND WE ESTIMATED THAT IT PROBABLY TOOK US ABOUT 55 HOURS TO COLLECT ALL THE INFORMATION TO, DO THE CALLS, TO ENTER THE INFORMATION INTO THE SURVEY MONKEY AND COLLECT THE CIL DATA. ONE OF THE THINGS THAT MADE US INTERESTED IN DOING THIS WAS JUST ALL THE SUPPORT THAT WE WERE OFFERED. THE QUESTIONNAIRES WERE ALREADY PUT TOGETHER, WE HAD THE OPPORTUNITY TO PARTICIPATE IN TWO TRAINING CALLS, WHICH WAS GREAT, VERY THOROUGH. AND SO WHAT WAS HELPFUL ABOUT THE TRAINING CALLS IS THAT MAYBE THERE WAS A QUESTION I DIDN'T KNOW TO ASK BUT ONE OF THE OTHER CENTERS WOULD ASK, SO THAT WAS GREAT. ALSO TO HEAR ABOUT FEEDBACK FROM CENTERS WHO HAD DONE IT FROM THE FIRST YEAR. SO THAT WAS VERY ATTRACTIVE TO US AND MADE US REALLY WANT TO PARTICIPATE IN THIS SECOND YEAR. ONE OF THE MAJOR TAKE-AWAYS WE GOT WAS JUST THAT SIMPLY ALL OF 126 OUR PROGRAMS ARE DOING EVALUATION. WE ALL HAVE DIFFERENT EVALUATION SURVEYS AND THINGS THAT WE'VE DEVELOPED. SO WE'RE COLLECTING THAT INFORMATION. BUT WE'RE NOT NECESSARILY PUTTING THAT TOGETHER INTO AN AGENCYWIDE REPORT. SO THAT WAS A BIG TAKE-AWAY FOR US, SOMETHING THAT WE LEARNED. ONE OF THE OTHER THINGS, AND THE OTHERS HAVE MENTIONED THIS, JUST THE IMPORTANCE OF DATA COLLECTION, THE IMPORTANCE OF SIMPLE THINGS LIKE, YOU KNOW, ONE OF THE PROGRAMS WE FOUND THE STAFF PERSON MAYBE WAS JUST MISSING UPDATING PHONE NUMBERS AND THAT KIND OF THING. SO THAT WAS IMPORTANT INFORMATION FOR US TO KNOW, WE MIGHT NOT HAVE CAUGHT THAT AND HAD THE OPPORTUNITY TO CORRECT THAT. SO THAT WAS VERY HELPFUL. ONE THING THAT SURPRISED US AS THE OTHERS MENTIONED, JUST THE TIME THAT IT TOOK. 25 CALLS DOESN'T SOUND LIKE LOT BUT THAT CAN TAKE A LOT OF TIME. WE ALSO RAN INTO THE CHALLENGE OF FINDING THAT MAYBE SOMEWHAT CAUGHT BECAUSE SOMEONE WITH A COGNITIVE DISABILITY WASN'T ABLE TO COMMUNICATE WITH US. WE TALKED TO MAYBE CAREGIVERS. THAT WAS A CHALLENGE, WE WANTED TO AS MUCH AS POSSIBLE BE ABLE TO TALK DIRECTLY TO THE CONSUMER. I THINK FOR OUR CENTER, YOU KNOW, WE DEFINITELY REALIZE THAT THIS IS DOING THESE OUTCOMES IS THE WAY THAT WE PROVE THE VALVE OUR PROGRAMS. WE JUSTIFY OUR EXISTENCE. AND SO WE WERE REALLY OPEN AND JUST CURIOUS, EXCITED TO GET 127 WHATEVER FEEDBACK WE COULD. I THINK ONE OF THE THINGS WE WERE EXCITED ABOUT IS, YOU KNOW, WE MAYBE NECESSARILILY DIDN'T REALIZE ALL THE ADVOCACY THINGS THAT WE ARE DOING. WE DON'T -- ONE OF THE THINGS IN OUR TRAINING MANUAL, YOU'LL SEE THEY GAVE US A DESCRIPTION OF WHAT A WORK PLAN WAS. SO WE DON'T HAVE A WRITTEN WORK PLAN ACCORDING TO THE CRITERIA THAT BOB GAVE US. BUT WE ARE DOING A LOT OF ADVOCACY ACTIVITIES. IT WAS RELATIVELY EASY FOR US TO PULL ALL OF THAT INFORMATION TOGETHER BECAUSE OF THE WAY WE WERE ENTERING IT INTO OUR DATABASE SYSTEM. SO THAT WAS EXCITING. ONE KIND OF PIECE OF ADVICE, ONE THING WE FOUND HELPFUL WAS, BECAUSE WE KIND OF HAD A TEAM WORKING AROUND US, IT WAS GREAT FOR US TO OCCASIONALLY MEET AND GIVE EACH OTHER FEEDBACK AND UPDATE BUSY HOW THINGS WERE GOING, YOU KNOW, GIVE SUPPORT TO OUR VOLUNTEERS IF THEY HAD A DIFFICULT CALL WHERE SOMEBODY HUNG UP ON THEM OR, YOU KNOW, THEY DIDN'T HAVE A GOOD EXPERIENCE. SO THAT WAS REALLY HELPFUL. WE ALL FELT LIKE, YOU KNOW, JUST PRACTICALLY WE WERE ABLE TO KIND OF SHARE THE WORKLOAD, BUT ALSO JUST TO SUPPORT EACH OTHER AND WORK TOWARDS THIS GOAL. WE WERE ABLE TO DO THE DATA COLLECTION PRETTY QUICKLY. >> Bob Michaels: OKAY. THANKS VERY MUCH, EVERYBODY. ANY QUESTIONS? COME ON, NOW. ANN? >> Audience Member: ANN FROM WEST VIRGINIA. MICHELLE WHEN YOU WERE TALK 128 BEING YOUR INTERNAL THING, BECAUSE I KNOW I HAD TALKED JAN ABOUT SOME OF THE PROGRAMS AND DIFFERENT THINGS THAT YOU'RE JUICING IN CELL SUITE. WAS THAT WHAT YOU WERE FINDING, HOW YOU WERE TRACKING SOME OF THE PROBLEMS WITH GATHERING YOUR DATA? >> Michelle Wakeley: I THINK LIKE APRIL HAD SAID, GARBAGE IN, GARBAGE OUT. WE WOULD MAYBE RANDOMLY PICK A CONSUMER OR AN I&R, AND THEN GO INTO CELL SUITE AND LOOK FOR THE INFORMATION AND FIND OUT THERE'S NO PHONE NUMBER, MAYBE THERE WAS INSUFFICIENT INFORMATION THAT WE HAD ENTERED. THAT WAS ONE OF THE THINGS I THINK THAT REALLY DID IMPRESS THE IMPORTANCE OF BETTER DATA COLLECTION ON OUR END. >> Audience Member: DID ALL OF YOU DO STRICTLY 100% PHONE INTERVIEWS HOW DID YOU ADDRESS FOLKS THAT DIDN'T HAVE PHONES? >> Michelle Wakeley: WE HAD ONE SPECIFIC SITUATION WHERE A GENTLEMAN WAS DEAF AND HE DID NOT HAVE A PHONE. WE WRESTLED WITH THAT AT FIRST. I CONTACTED BOB AND SAID, ONE OF THE CRITERIA IS THEY HAVE TO HAVE A TELEPHONE NUMBER. BUT WE WERE VERY CAREFUL, WE DID ANOTHER -- ANOTHER STAFF MEMBER MET WITH THE INDIVIDUAL. IN ORDER TO NOT SKEW THE DATA, BECAUSE HE DOESN'T USE AMERICAN SIGN LANGUAGE, HE IS VERY HOME-TAUGHT. WE HAVE UBI-DUOs. HE WAS ABLE TO TYPE HIS ANSWERS AND SHE COULD TAPE THEM VERBATIM RATHER THAN TRYING TO INTERPRET THEM HERSELF. >> Bob Michaels: SHEILA? >> Audience Member: THIS IS SHEILA FROM EUGENE, OREGON. 129 ALL THINGS BEING EQUAL AFTER YOU'VE DONE THIS, YOUR SURVEY -- [NO AUDIO -- WOULD YOU ALL SAY THAT YOU WOULD CERTAINLY CONTINUE DOING THIS IN YOUR CENTERS OR FOR THE FORESEEABLE FUTURE, IF NOT FOREVER? >> Bob Michaels: WOULD YOU DO IT AGAIN? >> Audience Member: WOULD YOU KEEP DOING IT? >> April Reed: DEFINITELY WE WOULD DO IT AGAIN. WE ARE PUTTING TOGETHER A PLAN, OUR SEPARATE PROGRAMS ARE DOING EVALUATIONS AND WE ARE NOW PUTTING TOGETHER A PLAN AND WE WILL BE IN THE NEXT FEW MONTHS COORDINATING THOSE PROGRAMS AND WRITING A FULL REPORT, AN AGENCY REPORT ON EACH PROGRAM, THE EVALUATIONS THAT WE'VE BEEN COLLECTING BUT HAVE THEM PUT INTO A WHOLE AGENCY REPORT FOR OUR DEPARTMENT. >> Michelle Wakeley: I THINK WE WOULD DO IT AGAIN, LIKE DAISY HAD SAID, YOU SEE A NUMBER THAT SEEMS SORT OF JARRING TO YOU. IN ADDITION TO BEING ABLE TO COMPARE IT AGAINST OTHER CENTERS, IT WOULD BE GOOD FOR US TO COMPARE THIS SAMPLE VERSUS THAT SAMPLE TO SEE WHAT IS CONSISTENT. 40%, ANOTHER 40%, THEN WE FEEL THAT'S MORE TRUE, AS OPPOSED TO 40 NOW AND THEN 80%. WAS IT THE SAMPLE, WAS IT THE ANALYST, WHAT WAS GOING ON THERE. I THINK SO. I THINK IT'S TELLING. >> Daisy Feidt: FOR US WE WOULD DEFINITELY DO IT AGAIN. I THINK THERE ARE SOME THINGS ABOUT THE QUESTIONS IF WE WERE JUST DOING IT ON OUR OWN WE WOULD CHANGE. SOMEBODY BROUGHT THIS UP THIS 130 MORNING ABOUT, WHY WASN'T THERE A FOLLOW-UP QUESTION ON THE I&R QUESTIONS. SINCE THAT WAS A LOWER PERCENTAGE FOR US, THE QUESTION ABOUT DID THE PERSON GET WHAT THEY NEEDED FROM YOUR I&R, I THINK YOU WOULD WANT TO ASK A FOLLOW-UP QUESTION. I'M INTERESTED TO KNOW, WHAT REALIZE WERE YOU LOOKING FOR? MY SUSPICION IS A LOT OF THOSE PEOPLE ARE LOOKING FOR HOUSE WHICH GO DOES NOT EXIST IN CHICAGO. THOSE KINDS OF THINGS, YOU REALLY FINED YOUR RESULTS SO IT WOULD BE INTERESTING TO SEE A LITTLE MORE DETAIL ON SOME OF THOSE QUESTIONS. >> Bob Michaels: OTHER QUESTIONS? >> Audience Member: >> Audience Member: PAT LAYERED FROM MARYLAND. ONE OF YOU MENTIONED YOU FORMED A QUALITY IMPROVEMENT COMPLETED. I WAS WONDERING WHO WAS ON THAT COMMITTEE. I BELIEVE THE OTHER TWO OF YOU DIDN'T HAVE SUCH A COMMITTEE AND I WAS WONDERING WHY YOU DIDN'T FORM ONE. AND WOULD YOU FORM ONE NOW LOOKING BACK ON IT? SO THAT'S MY MAIN QUESTION BUT I HAVE LOTS OF OTHERS, IN TERMS OF STARTING THIS AND WORKING WITH STAFF AND SO MANY OTHER THINGS. MAYBE I CAN TALK TO SOME YOU HAVE LATER ABOUT THAT. THANK YOU. >> Daisy Feidt: I MENTIONED HAVING THE QUALITY IMPROVEMENT COMMITTEE. FOR US, ACTUALLY, NO. THIS WASN'T SOMETHING WE FORMED AS A RESULT OF THE FIELD TEST, IT CAME OUT OF OUR STRATEGIC PLAN. 131 SO WE'VE HAD IT FOR A YEAR AND A HALF OR SO NOW. IN TERMS OF WHO'S ON THE COMMITTEE, IT'S ALL LEVELS OF STAFF AND ACROSS DEPARTMENTS. I THINK THAT WAS THE QUESTION. >> Michelle Wakeley: AGAIN, WE'RE A VERY SMALL CENTER. WE HAVE UNDER 10 EMPLOYEES WHICH MEANS THREE LOCATIONS THAT SERVES OVER 13 COUNTIES. SO WE'RE SPREAD A LITTLE THIN. WE DON'T HAVE AN ORGANIZED QUALITY ASSURANCE TEAM OR ANYTHING LIKE THAT. BUT I THINK THAT WE HAVE TORT OF THE PEOPLE ININVOLVED HAVE BEEN INVESTED IN WHAT WE WERE DOING. WE'VE TAKEN IT UPON OURSELVES TO MAKE SURE THAT'S GETTING BACK TO THE OTHER STAFF MEMBERS AND TO KEEP WORKING TOWARD THE SAME GOAL. I KNOW I'VE TALKED WITH MY BOSS ABOUT WHEN WE COME BACK, WE REALLY NEED TO MAKE SURE THAT EVERYBODY'S ON THE SAME PAGE, EVERYBODY'S UNDERSTANDING WHAT WE'RE DOING, WHY WE'RE DOING IT. IT'S NOT JUST ANOTHER THING TO DO, IT'S EVERYTHING THAT WE'RE DOING. >> April Reed: THIS WAS A -- KIND OF A BEGINNING POINT FOR US. SO I THINK THIS HAS GENERATED A LOT OF CONVERSATION. OUR DIRECTORS ARE BEGINNING TO THINK ABOUT WHAT COULD WE DO PRESCRIPTIONS FIRST FORMALIZING IT AND HAVING A QUALITY ASSURANCE TEAM AT OUR AGENCY. >> Bob Michaels: ONE OF YOUR BOARD IS SITTING IN THE ROOM. >> April Reed: I KNOW. >> Bob Michaels: OTHER QUESTIONS? >> Bob Michaels: I WAS WONDERING ABOUT SURVEY MONKEY. HOW DID THAT WORK FOR YOU? 132 YOU GOT THE INFORMATION, ALL OF YOU HAVE GOTTEN INFORMATION ON THE QUESTIONNAIRE AND THEN TRANSFERRING? >> April Reed: SURVEY MONKEY WAS GRAY, WE DIDN'T REALLY HAVE ANY ISSUES WITH IT. >> Daisy Feidt: US, TOO. >> Michelle Wakeley: AGAIN, WE HAD A TEAM THAT WAS A LITTLE DIVERSIFIED. AND ONE OF OUR EMPLOYEES IS THE -- JUST NOT VERY COMFORTABLE WITH TECHNOLOGY. SO THEY HAD A LOT OF DIFFICULTY. THEY HAVE HEARD OF SURVEY MONKEY BEFORE, SO SHE WOULD TALK TO ME AND SAY, WHAT IS THIS I'M SUPPOSED TO DO WITH THE MONKEY. HOW DO I WORK THE MONKEY? SHE WAS VERY GOOD WITH THE INDIVIDUALS. >> HOW OLD IS SHE? >> Michelle Wakeley: OLDER THAN ME. >> Bob Michaels: ARE YOU SURE? >> Michelle Wakeley: I'LL BE POSSESS LATE. SHE CALLS HERSELF OLD, SHE'S NOT VERY OLD. SHE WAS I THINK BETTER WITH THE ACTUAL PHONE CALLS AND THE SURVEYS. I WAS A LITTLE BETTER ORGANIZING THE DATA AND KEEPING IT TOGETHER. >> Daisy Feidt: AND FOR US, THE SURVEY MONKEY PART WAS PROBABLY THE EASIEST PART. IT WORKED REALLY WELL. >> Bob Michaels: HOW WAS THE TRAINING THAT WE GAVE YOU? YOU SAID APRIL IT WAS GOOD. BUT IT WAS IT SATISFACTORY? HOW IMPORTANT WAS IT? WHAT GROUP WOULD HAVE BEEN BETTER? >> Daisy Feidt: I THOUGHT THE TRAINING WAS REALLY GOOD. I THINK IT WAS -- SEEMS LIKE 133 THERE WERE A LOT OF LESSONS LEARNED FROM THE FIRST YEAR TO THE SECOND YEAR SO A LOT OF THINGS ALREADY GOING IN WERE MORE KIND OF CLARIFIED. THE TRAINING MANUAL WAS REALLY WELL LAID OUT, ALL THE MATERIALS WERE REALLY HELPFUL. AND IT WAS ALSO AS SOMEBODY ELSE SAID, REALLY HELPFUL THAT THERE WERE ALL THE OTHER CILs ON THE CALL BECAUSE PEOPLE ASKED QUESTIONS THAT WERE REALLY GOOD. IT WAS VERY GOOD. >> Michelle Wakeley: I THINK THE TRAININGS WERE VERY GOOD. THEY ALLOWED US AND OPPORTUNITY TO SPEAK WITH ONE ANOTHER A TO SPEAK WITH BOB AND HIS TEAM. AND I THINK THAT IT MADE US MORE OPEN TO CRITICIZE THE INSTRUMENTS, POSITIVELY OR NEGATIVELY. THEY MATE MADE US MORE OPEN TO BEING ABLE TO TALK TO ONE ANOTHER AND GAIN SUPPORT FOR ONE ANOTHER, LIKE WHEN WE HAD INDIVIDUALS WHO HAD THE TELEPHONE. IT GAVE THAWS PERSONAL FEEL AND THE SUPPORT THAT WE NEEDED. >> April Reed: WELL, WITH US USING A COUPLE OF VOLUNTEERS TO HELP MAKE THE CALLS, THE TRAINING MANUAL WAS REALLY IMPORTANT IN SETTING THE EXPECTATIONS. WHAT THEIR ROLE WAS, YOU KNOW, WHAT WAS OKAY FOR THEM TO ASK, HOW SHOULD THEY COMMUNICATE WITH THE CONSUMERS. IT KIND OF GOT US ALL STARTED OUT ON THE SAME PAGE. WHEN THEY CAME BACK WITH QUESTIONS, LET'S CHECK THE MANUAL, LET'S ASK THAT QUESTION ON THE TRAINING CALL SO. THAT WAS VERY HELPFUL. >> Bob Michaels: COMMENTS? >> Mike Hendricks: SO HERE'S A 134 QUESTION ABOUT THE ACCURACY OF THE DATA. WE LOOK AT IT, AND I MENTIONED COUPLE THINGS EARLIER THAT KIND OF GIVE US SOME CONFIDENCE THAT THE DATA ARE CREDIBLE AND TRUSTWORTHY. AND BY THE WAY, I FORGOT TO MENTION THIS. WE ONLY SHOWED YOU THE MOST RECENT YEAR'S WORTH OF DATA. WE DIDN'T SHOW YOU LAST YEAR'S DATA. SOMEONE HERE MENTIONED CONSISTENCY. WE DID, WE ACTUALLY COMPARED WHAT WE LEARNED THE FIRST YEAR WITH WHAT WE LEARNED THE SECOND YEAR. BOB AND I HAD NO IDEA HOW IT WAS GOING TO COMPARE BUT IT COMPARED VERY, VERY CONSISTENTLY, WE WERE SURPRISED. NONE OF THE NUMBERING DIFFERED BY MORE THAN 7% FROM ONE YEAR TO THE OTHER. WHICH FRANKLY STUNNED US. THAT GIVE US A LITTLE MORE CONFIDENCE IN THE CREDIBILITY. HERE'S MY QUESTION TO THE PANELISTS IF I CAN. THERE'S NOT REALLY ANY WAY TO KNOW IF THE INFORMATION YOU GOT FROM THOSE PEOPLE WAS ACCURATE OR NOT, RIGHT? WE COULDN'T LOOK INSIDE THEIR BRAIN AND SEE IF IN FACT THEY WERE TELLING YOU THE TRUTH. IT SOUNDS LIKE THE THREE OF YOU FEELS, BECAUSE YOU'RE ACTING ON IT IN SOME CASES, SOUNDS LIKE YOU FEEL THAT INFORMATION IS ACCURATE, TRUSTWORTHY ENOUGH TO DO SOMETHING WITH AND THINK B. I'M NOT TRYING TO PUSH YOU TOO FAR BUT I'M JUST CURIOUS WHY. WHAT IS IT ABOUT THE PROCESS THAT GIVES YOU SOME CONFIDENCE THAT WHAT YOU LEARNED WAS ACCURATE? 135 >> April Reed: WELL, I THINK IT WAS VERY ORGANIZED, THE QUESTIONS WERE VERY CLEAR. OUR EXPERIENCE WAS THAT FOR THOSE PEOPLE THAT AGREED TO DO THE SURVEY AND, YOU KNOW, WERE WILLING TO SPEND THE TIME WITH US, THEY GAVE US PRETTY STRAYED FORD ANSWERS. OTHERWISE THEY WOULDN'T HAVE AGREED TO DO IT. >> Michelle Wakeley: AGAIN, I THINK THE TRAINING REALLY HELPED. WE HAD SOME PEOPLE WHO MAYBE DID NOT UNDERSTAND THE QUESTION AS IT WAS STATED FOR A VARIETY OF REASONS. WE WERE TOMORROW NOT TO FEED INTO THE QUESTIONS TOO MUCH, TO TRY TO HELP THEM UNDERSTAND THEM BUT TO NOT PROMPT ANYONE FOR ANY SORT OF ANSWER OR ANYTHING LIKE THAT. I THINK THAT WE STAYED PRETTY TRUE TO THAT AND TRIED TO BE AS ONS BUT ALSO AS HELPFUL TO THE INDIVIDUALS WHO WERE HELPING US AS WE COULD. SO I THINK THAT WAS IMPORTANT. >> Daisy Feidt: I THINK FOR ME THE FACT THAT THE PEOPLE THAT WE WERE SURVEYING, IT WAS RANDOM SAMPLE. AND THE FACT THAT WE GOT A GOOD NUMBER OF RESPONSES. IT WASN'T A TON BUT IT FEELS LIKE ENOUGH TO GET GOOD RESULTS. QUITE FRANKLY, PARTLY WHAT MAKES ME BELIEVE IT IS THE RESULTS WEREN'T OVERWHELMINGLY POSITIVE. SO I FEEL LIKE THAT'S PROBABLY REALISTIC FOR THE POPULATION THAT WE'RE TRYING TO SERVE. IN FACT, PEOPLE HAVE A REALLY DIFFICULT NEEDS AND SITUATIONS THAT THEY ARE IN. >> Bob Michaels: QUESTIONS SORRY ANN, YOU ONLY GET ONE QUESTION. >> Audience Member: I'M ANN FROM 136 WEST VIRGINIA. YOU ALL KEEP TALKING ABOUT THE TRAINING REALLY HELPED YOU A LOT. SO REALLY MY QUESTION I GUESS, WITH BOB AND MIKE, IF WE ARE ALL ARE TO GO FORWARD WITH THIS, OUR STAFF OR OUR VOLUNTEERS THAT WE MAY WANT TO USE TO HELP US WITH THIS DON'T HAVE THAT TRAINING. SO IS THAT TRAINING, THOSE TWO 90-MINUTE TRAININGS THAT YOU DID FOR THESE PARTICIPANTS, IS THAT SOMETHING THAT'S GOING TO BE AVAILABLE TO THE REST OF US AS WE TRAIN STAFF AND/OR VOLUNTEERS TO HELP US DO THIS PROCESS? >> Mike Hendricks: BOB MAY FEEL DIFFERENTLY ABOUT IT. HE AND I DID THE TRAINING. THE TRAINING WAS -- MAYBE THAT'S TOO FANCY A WORD. MAYBE WE'RE SCARING YOU OFF WITH THAT WORD. IT'S MY RECOLLECTION WE WENT THROUGH THE TRAINING MANUAL. WE LITERALLY WALKED THROUGH THE TRAINING MANUAL, WHICH YOU HAVE, IT'S IN YOUR MATERIALS. WE JUST MADE CERTAIN THAT EVERYBODY REALLY UNDERSTOOD EVERYTHING IN THE TRAINING MANUAL, WHICH DID TAKE SOME EXPLANATIONS, RIGHT? SOME OF IT WAS NOT SO IMMEDIATELY OCCUPY. IT WAS MAYBE LESS A STAND-ALONE TRAINING THAN A TRAINING MANUAL. I'M SURE BOB AND I WOULD BE HAPPY TO PRODUCE SOMETHING, OR YOU MAY NOT FEEL ANYTHING ELSE IS NEEDED AFTER YOU READ THE TRAINING MANUAL. OF COURSE THERE WERE LOTS OF REALLY GOOD QUESTIONS THAT CAME UP, WHICH ARE HARD TO RETREE OF AT THIS POINT. BUT THAT WAS THE VALUE OF IT TO ME, THE REALLY GOOD QUESTIONS THAT CAME UP. 137 >> I THINK IN THE SECOND YEAR WE WERE MUCH BETTER AT PUTTING TOGETHER A TRAINING MANUAL THAT REREFLECTED THE QUESTIONS THAT CROPPED UP ALL THE TIME THE FIRST YEAR. SO I THINK THE SECOND-YEAR MANUAL IS REALLY -- I THINK MIGHT BE BETTER THAN THE FIRST YEAR, MUCH MORE COMPREHENSIVE. MAUREEN? >> Audience Member: I WAS CONFUSED. THE PEOPLE THAT PARTICIPATED THAT FIRST YEAR, DID THEY ALSO PARTICIPATE THE SECOND YEAR TOO? OR DID THEY JUST JOIN THE NINETY-MINUTE CALLS? >> 18 OF THEM DID. >> Bob Michaels: I THINK WE HAD 20 BY THE END OF THE YEAR A18 OF THEM DID. >> Audience Member: WHEN YOU SAID THERE WAS LESS THAN A 7% VARIANCE, WAS THAT HAVE THE CENTERS THAT PARTICIPATED FROM YEAR ONE TO YEAR TWO? >> Bob Michaels: WHAT DID HE SAY? >> Mike Hendricks: MAYBE I MISUNDERSTOOD THE QUESTION. THE OUTCOMES FROM YEAR ONE, WHEN YOU COMPARE THEM TO THE SAME QUESTION, WE COULDN'T DO IT ON EVERY QUESTION BECAUSE SOME OF THEM VARY. THEY GOT BETTER. BUT THE QUESTIONS THAT ARE OH COMPARABLE BETWEEN YEAR ONE AND YEARWAY, TO OUR GREAT DELIGHT, THE SCORES WERE VERY CONSISTENT FROM YEAR ONE TO YEAR TWO, EVEN THOUGH THERE WERE DIFFERENT CILs INVOLVED. THAT LET TO US THINK, THAT'S COMFORTING. MAYBE THIS IS REALLY CAPTURING WHAT'S HAPPENING OUT THERE, REGARDLESS OF WHO OUT THERE IS. >> Bob Michaels: JULIA? 138 >> Audience Member: I WANTED TO ASK A QUESTION ABOUT THE BOARD OF DIRECTORS. I HAVEN'T HEARD VERY MUCH SAID ABOUT THAT. WHAT WAS THE PROCESS BY WHICH YOU WENT ABOUT THIS? DID YOU PROPOSE IT TOO YOUR BOARD AND HAVE THEM EMBRACE IT AND THEN WENT AHEAD WITH IT? OR DID YOU SURPRISE THEM WITH THE RESULTS AND ASKED HOW THEY WANTED TO PROCEED? >> Daisy Feidt: WE TALKED TO THEM ABOUT THE FACT THAT WE WERE DOING IT. I DON'T RECALL REALLY ASKING PERMISSION TO DO IT. WE TALKED ABOUT THE FACT THAT WE WERE GOING DO IT AND THEN WE PUT ON THE RESULTS. >> Audience Member: WHAT HAS BEEN THEIR REACTION SINCE? >> Daisy Feidt: I THINK THEY ARE GLAD THAT WE'RE TACKLING THE ISSUE OF OUTCOMES. MEASUREMENT AND MANAGEMENT. I THINK WE POINTED OUT SOME AREAS WHERE WE WANT TO INFRASTRUCTURE OUR RESULTS OBVIOUSLY AND THEY WANTS US TO PROOF THOSE RESULTS, AS WELL, THEY ARE ON BOARD. >> Michelle Wakeley: I DON'T BELIEVE THAT WE HAVE DELIBERATELY SHARED IN-DEPTH THE RESULTS OF THE PROJECT WITH OUR BOARD. BUT I IMAGINE THAT AFTER THIS EVENT WE PROBABLY WILL BE PRESENTING THE FINDINGS TO THE BOARD AND GOING FROM THERE. >> April Reed: I'M NOT SURE, IF THE FINDINGS HAVE BEEN PRESENTED TO OUR BOARD OR NOT. >> Bob Michaels: WE'LL ASK MARY, SHE'LL KNOW THE ANSWER TO THAT QUESTION MEMBER MACHINE I'M MARY SLAUGHTER, I'M ON THE BOARD. I JUST MET APRIL AT THIS EVENT. 139 AND I DON'T RECALL IF SOME OF THIS INFORMATION CAME TO THE BOARD, QUITE FRANKLY. ONE OF THE THINGS I'VE BEEN WANTING TO TALK ABOUT A LITTLE BIT MORE AT THE BOARD LEVEL HAS BEEN EVALUATION OF DATA AND KIND OF UNDERSTANDING A LITTLE MORE ABOUT HOW WE'RE DOING ON THE PROGRAMS AND QUALITY IMPROVEMENT. BUT QUITE FRANKLY I THINK ONE OF THE THINGS EVERYBODY'S JUST SO BUSY DOING IS TRYING TO GET MONEY, TRYING TO STAY AFLOAT. I'M SURPRISED THERE'S A QUALITY IMPROVEMENT TEAM AT THE CENTER. I THINK THIS CAN BE USED AS AN I AM TUESDAY TO HELP ALL OF US SEE THAT NEEDS TO BE MORE GROUNDED. >> Bob Michaels: WHAT WE DID WHEN WE WERE TRYING TO RECRUDE, WE HAD NO IDEA IF WE WERE GOING TO BE OVERWHELMS. WE HAD CONVERSATIONS, WHAT IF WE GET 200? WE HAD NO IDEA. WE HAD NO IDEA. BASICALLY WHAT WE DID IS WE SENT A LETTER TO THE DIRECTOR AND WE SAID, OKAY. THIS IS WHAT WE WANT TO DO. AND WE EXPLAINED, WE SAID WE NEED FOR YOU TO APPOINT A PERSON AS PRIMARY CONTACT. AND THAT PERSON THAT HAS GOT NEIGHBORS AND YOU CAN HAVE TWO STAFF AND UP TO FOUR STAFF. IT'S JUST LIKE WHAT YOU HEARD. BUT THEN THEY HAVE TO JUST TRANNED, AFTER THEY HAVE PARTICIPATED. WE PUT SOME REQUIREMENTS ON IT. IF A CENTER WANTED TO PARTICIPATE, THEY HAD KNOW IT WASN'T GOING TO BE A BOX OF CHOCOLATES. THEY KNEW THEY WOULD HAVE TO MAKE A REAL DONATION TO DONATE. >>> SOME SENATORS WOULD, SOME 140 WOULD NOT. THEY WOULD HOLD THAT AGAINST THEM ONE WAY OR ANOTHER. I'LL DO QUESTIONS. NOTHING FROM THE RIGHT SIDE OF THE ROOM OVER HERE. WE'VE TALKED NOW FOR A COUPLE DAYS, AND I DON'T KNOW HOW MUCH ALL OF YOU HAVE SEEN. BUT YOU SAW ON THE LOGIC MODEL THAT THERE WERE DIAGRAMS UP ON THE LEVELS AND NOT -- AND WE DIDN'T LOOK CLOSELY AT THOSE. DID ANY OF YOU FIND THAT THE ONES WE DID LOOK AT WERE ABOVE OR BELOW WHERE YOUR CENTER IS? AND THEN YOU SHOULD HAVE BEEN WORKING ON SOMETHING MORE COMPLEX? I DON'T KNOW IF YOU UNDERSTAND THAT PART. >> Daisy Feidt: I FELT LIKE THE OUTCOMES WERE REALLY RIGHT WITH OUR CENTER. >> Bob Michaels: WE CAME BACK AND PICKED THEM BY GOING TO YOU AND SAYING, WHAT SHALL WE DO HERE. I'M HOPING THAT WOULD BE TRUE. IF IT'S TRUE IN ACCESS LIVING, THAT'S GREAT. NO? ANY OTHER QUESTIONS? PAT. >> Audience Member: ONE OF THE THINGS THAT WE WERE TALKING ABOUT EARLIER TODAY, FOR VERY LARGE PROGRAMS THAT HAVE MULTIPLE FUNDING SOURCES, THAT IT'S NOT CLEAR WHETHER OR NOT THIS PARTICULAR SET OF INDICATORS IS GOING TO WORK WITH ALL OF THE DIFFERENT FUNDING SOURCES THAT YOU MIGHT HAVE. AND SO I DON'T KNOW EXACTLY WHAT TO DO ABOUT IT. YOU KNOW, WE WERE TALKING ABOUT THE UNITED WAY GRANTS THAT SOME PEOPLE GET, AND IN MY ESTIMATION, HAVING WORKED ON MY 141 LOCAL UNITED WAY AND GRANT SELECTION AROUND OUTCOMES AND ALL THAT SORT OF THINGS A LITTLE BIT OF KNOWLEDGE CAN BE REALLY DANGEROUS. I FEEL LIKE A LOT OF THOSE UNITED WAY TYPES HAVE A LITTLE BIT OF KNOWLEDGE ABOUT THIS AND THEY ARE VERY DANGEROUS. I HAVE WATCHED SOME VERY GOOD PROGRAMS GET UNFUNDED BECAUSE THEY DIDN'T HAVE ANY HELP. WE WERE TRYING TO FIGURE OUT HOW TO DO LOGIC MODELINGS AND HOW TO DO OUTCOME MEASURES AND THAT SORT OF THING, IF THEY HAVE ALL DIFFERENT FUNDING SOURCES ARE GOING THE DIRECTION OF MEASURING OUTCOMES, OR ATTEMPTING TO MEASURE OUTCOMES, HOW IS THAT NEGOTIATED INTERNALLY? AS A BOARD PERSON OR AS AN EXECUTIVE DIRECTOR, WHAT ARE THE CHALLENGE -- HAS ANYBODY FIGURED THAT OUT? HAS ANYBODY FIGURED OUT HOW TO MAKE ALL THOSE FUNDING SOURCES TALK TO EACH OTHER, WORK TOGETHER -- [LAUGHTER] >> Audience Member: WELL, IF WE COULD HAVE AT LEAST A FEW COMMON VIEWPOINTS AND SO FORTH AROUND WHAT'S IMPORTANT, WHAT'S THE ESSENCE OF OUR WORK HERE. AND MAYBE IT COMES DOWN TO DECIDING WHETHER OR NOT YOU FINED STUDENTS WHO ARE WORTH IT IF YOU'RE CONSISTENT UP HERE. I WANT TO AT SOME POINT BEGIN TO HAVE DISCUSSIONS ABOUT HOW WE FIGURE THIS OUT IF WE HAD MULTIPLE FUNDING STREAMS THAT ALL WANT TO KNOW DIFFERENT RESULTS. >> Daisy Feidt: I TOTALLY CONCUR WITH YOU, IT'S A REALLY COMPLICATED PROBLEM. I'D SAY IT'S VERY CLEAR THAT THESE OUTCOMES IN THE FIELD TEST 142 ARE NOT SUFFICIENT FOR EVERY SINGLE FUNDING SOURCE FOR SURE AT ACCESS LIVING. THAT'S BEEN ONE OF THE REAL CHALLENGES FOR US, WHICH OUTOUTCOMES ARE WE GOING TO MEASURE. YOU KNOW, HAVING A CROSS-SECTION OF ALL OF OUR PROGRAMS, YET YOU CAN'T MEASURE EVERY SINGLE THING ABOUT EVERY PROGRAM. THAT WOULD BE TOTALLY OVERWHELMING. >> Bob Michaels: MIKE? >> Mike Hendricks: WELL, I DON'T HAVE THE DEFINITE ANSWER BUT I CAN TELL YOU SOMETHING I USED TO DO BACK WHEN UNITED WAY WAS FIRST STARTING TO DO THIS, I WAS INVOLVED AT THE NATIONAL LEVEL. IN ADDITION TO TRAINING THEIR AGENCIES, WHICH I DID A LOT OF, I ALSO WOULD SPEND TIME WITH THE BOARD OF THE UNITED WAY AND THE EXECUTIVE DIRECTORS. ONE OF THE THINGS I WOULD SUGGEST WOULD BE THAT THE UNITED WAY CONVENE AND CHAIR A LOCAL FUNDER'S COUNSEL I THOUGHT UNITED WAY WAS SEPARATE, AND INDEPENDENT AND HAD THE CACHET THAT THEY COULD PULL THAT OFF. I KNOW OF TWO PLACES THAT HAPPENED. THE COUNTY, STATE GOVERNMENT WOULD BE PART OF IT, ESPECIALLY IF THERE WERE ANY FOUNDATIONS IN THE AREA, THEY WOULD BE A PART OF IT, THE UNITED WAY WOULD BE A PURPOSE OF IT. THE PERP WAS, LET'S NOT JERK THESE PEOPLE AROUND OUT THERE TRYING TO DO GOOD WORK. LET'S SIMPLE FYE STREET LIEN CORD KNITS. IT CAN BE DONE WHEN YOU GET PEOPLE OF GOOD WILL AROUND A TABLE TODAY. I KNOW IT'S HAPPENED IN TWO PLACES. 143 FUNDERS COUNCILS. IF YOU CAN FIND SOMEONE IN YOUR COMMUNITY WHO WOULD BE THE RIGHT GROUP TO PULL THOSE TOGETHER, I THINK THERE'S SOME REAL POTENTIAL THERE. >> Audience Member: HAVE YOU EVER KNOWN AS PROGRAM TO CONVENE THEIR OWN COUNCIL? IF YOU'VE GOT LOCAL COUNTY FUNDS TO DO, SAY, HOME MODIFICATIONS. AND YOU'VE GOT ITEMS TO HOST AND THE UNITED WAY FUND, WHATEVER PURPOSES. TO GET ALL OF THOSE FUNDERS AND REPORTING POSTED IN THE SAME ROOM AND PUT UP ON THE SCREEN, HERE WHAT'S WE DO FOR YOU AND YOU AND YOU. HAVE YOU EVER SEEN ANYBODY TRY THAT? >> I HAVEN'T SEEN IT, BUT I'VE HEARD IT. YOU'RE ON THE WRONG SIDE OF THE POWER TO BE THE ONES CONVENING THAT MEETING. THAT'S WHY I THOUGHT THE UNITED WAY WAS SUCH A GOOD GROUP, THEY ARE ON THE RIGHT SIDE OF THE POWER. I'M NOT SURE YOU SHOULDN'T TRY IT OR IT WOULDN'T REFRESHING FOR YOU. ASKING BELT BE AWARE OF THE DIFFERENT THINGS WE HAVE TO DO. AGAIN, YOU'RE ON THE WRONG SIDE OF THE POWER TO BE THE CHAIR OF THAT. >> Audience Member: I HAVE A COMMENT TO WHAT PAT SAID AND SOMETHING ELSE TO SAY ABOUT FUNDING SOURCES. WHEN YOU START IN THAT, THIS IS WHAT YOU PAY FOR AND THIS IS WHAT YOU PAY FOR, THE FLIP SIDE IS I'M ONLY GOING GIVE YOU THE NUMBERS YOU PAY FOR. WE'VE HAD A BIG FIGHT WITH OUR ADRC BECAUSE WE DO WHAT WE DO, AND MOST OF WHAT WE DO IS PAID 144 FOR BY FEDERAL GRANT'S GRANT AND THE ADRC DOESN'T GIVE US ANY MONEY. I TELL THEM, I'LL GIVE YOU THE NUMBERS YOU PAY FOR. I TELL ALL MY FUNDERS THAT. THEY WILL SAY, HOW MANY DID YOUR CENTER DO LAST YEAR. I SAY, I'M NOT GOING TELL YOU. I'LL TELL YOU HOW MANY YOU PAID FOR. IF YOU WANT TO HAVE A BIGGER PERCENTAGE OF WHAT WE D GIVE ME MORE MONEY AND YOU'LL GET BETTER NUMBERS OUT OF ME. I DON'T THINK OUR FUNDING SOURCES OUGHT TO GET CREDIT FOR WHAT SOMEBODY ELSE PAID FOR. I HAVE A CONSTANT FIGHT, SAME THING WITH MY STATE INDEPENDENT LIVING COUNCIL. I'LL GIVE WHAT YOU PAY FOR BUT NOT EVERYTHING ELSE WE DO. WE FIND, AND WE'VE BEEN DOING OUTCOMES SINCE 1995 -- THE MAJORITY -- AND WE GOT LIKE 8 OR 9 FUNDING SOURCES RIGHT NOW. WE HAVE ONLY ONE OF THOSE FUNDING SOURCES THAT ACTUALLY UNDERSTANDS OUTCOMES. ALL OF THEM ASKS FOR IT BUT ONLY ONE KNOW WHAT IS THEY ARE ASKING FOR. YOU CAN READ SOME OF THE NOFAs IN THE FEDERAL REGISTER AND IT'LL SAY WE WILL EXPECT TO YOU GIVE US OUTCOMES AND TELL US HOW MANY RIDES YOU PROVIDE. THAT'S STILL AN OUTCOME, THAT'S AN OUTPUT. I SAY, I'M NOT GOING TO GIVE THEM WHAT THEY ARE ASKING BUT A THEY DON'T KNOW WHAT THEY ARE ASKING. IF I GIVE THEM THAT, THEY WOULDN'T KNOW WHAT TO DO WITH IT. WE START WITH GIVING THEM OUTPUT AND SEE IF THEY COME BACK AND ASK US MORE QUESTIONS. 145 EVERYBODY HAS BEEN SATISFIED WITH OUTPUTS EXCEPT UNITED WAY. THE SHUNNEDDERS DON'T KNOW AND I'M NOT GOING TEACH THEM HANDOUT HOW TO MAKE MY LIFE HARDER. I JUST GIVE THEM OUTPUTS, THAT'S THE THING THEY PAY FOR. >> Bob Michaels: LAST CHANCE OVER HERE ON MY RIGHT. I WAS GOING TO ASK ONE FINAL QUESTION HERE BUT THE I&R QUESTIONS. WHEN YOU CALLED, WHEN YOU GUYS CALLED THE I&R FOR THE I&R CALLS, DID YOU FIND IT WAS TOO FAR AWAY OR THAT THE PEOPLE WERE NEVER THERE? WHAT WAS YOUR SENSE, IF YOU HAD CALLED EARLIER OR AT A DIFFERENT TIME, IF YOU HAD TOLD THEM SOMETHING -- WHAT WOULD MAKE IT WORK BETTER? >> Daisy Feidt: I DON'T KNOW WHAT WOULD HAVE MADE IT WORK BETTER. WE DEFINITELY HAD MORE ISSUES GETTING I&R CALLERS WHO KNEW WHAT THEY HAD GOTTEN FROM US AND HAD MORE PEOPLE TELL US ABC, WE HAVE THEM IN OUR I&R LOG, BUT THEY DIDN'T -- THEY WERE AN ADMIT THAT THEY HADN'T GOTTEN AN I&R FROM US AND NEVER TOLD US WHO THEY WERE. >> Michelle Wakeley: WE HAD THE SAME PROBLEM. WE PROBABLY CALLED DOUBLE THE AMOUNT OF I&Rs THAN WE DID CONSUMERS. AND I THINK WHAT I WAS HEARING WAS, I WILL CALL SOMEONE AND I WOULD TRY TO SPARK THEIR MEMORY A LITTLE BIT, THEY WILL THEM WHAT WE D. I KNOW WHAT THEY GOT OR DIDN'T GET BECAUSE IT'S IN MY INFORMATION. BUT TELL THEM WHAT OUR SERVICES ARE, AND WHAT I'VE FOUND IS OFTENTIMES WHEN PEOPLE WERE CALLING AROUND FOR INFORMATION 146 AND REFERRALS THEY CALL MULTITUDE OF PLACES AND WE BLEND IN TOGETHER. MY AREA, THEY WILL SAY, ARE YOU THE COORDING COUNCIL FOR INDEPENDENT LIVING? ARE YOU THIS, ARE YOU THAT, ARE YOU THIS? THEY DON'T KEEP THE ACRONYMS STRAIGHT. WHEN I SAY I'M CALLING FROM THE CENTER FOR INDEPENDENT LIVING, THEY HAVE BEEN AROUND AND THEY HAVE CALLED SO MANY PEOPLE THAT IT'S A BLUR, EVEN IF WE DO GET TO THE POINT -- OH, YEAH, YEAH, I REMEMBER YOU NOW. I THINK THAT'S THE NATURE OF THE I&R. >> Bob Michaels: HOW COULD WE MAKE IT BETTER? MOVE IT CLOSER? >> A SHORTER TIME FRAME MIGHT HAVE HELPED. SOMEBODY'S MADE 10 CALLS AROUND AN ISSUE, EVEN IF THEY DID IT TWO WEEKS AGO THEY MIGHT NOT REMEMBER YOUR SPECIFIC PROGRAM. >> Bob Michaels: ANYTHING ELSE? OKAY. THEN IT'S ALMOST FIVE TILL NOW, SO LET'S GO AHEAD AND BREAK AND MEET AGAIN AT ABOUT QUARTER AFTER. AGAIN, THANK YOU TO ALL OF YOU. [APPLAUSE]