Campbell Collaboration: New Directions in Identifying What Works. Presenters: Dr. Chad Nye and Dr. Herb Turner. >> JOANN: Good afternoon, everyone. And thank you for joining the webcast on the Campbell Collaboration: New Directions in Identifying What Works. My name is Joann Starks, and I'm with the National Center for the Dissemination of Disability Research or NCDDR, based at the Southwest Educational Development Laboratory in Austin, Texas. I'll be moderating today's webcast and getting your questions for our presenters. First, I want to be sure to thank our partners at ILRU in Houston for helping with the technical side of the webcast. There are some materials accompanying today's webcast that can be found on ILRU's website. This includes a link to the article written by our speakers, a document of a plain language summary conducted by our presenters and a PowerPoint file with a corresponding Word version. The PowerPoint will be shown on the screen during the webcast, but you may prefer to use a printed version for taking notes. You can submit questions at any time during the webcast today and the presenters will decide whether to answer them at the end where we have set aside a question and answer period or during the webcast if some clarification might be needed. To ask a question, you can click the submit question button at the bottom of your RealOne or Windows Media Player screen or you can send E-mail to webcast@ncddr.org. If anyone has technical difficulties during the webcast, please call ILRU at (713)520-0232 and dial 0 for the operator to ask for assistance. This number is both voice and TTY capable. The NCDDR is pleased to bring you today's webcast that highlights the Campbell Collaboration also known as C2. The new scope of work of the NCDDR includes several activities that involve the Campbell Collaboration and its development of evidence-based resources such as systematic reviews of research evidence. The NCDDR hopes to work with C2 to establish a coordinating group or subgroup to focus on developing systematic reviews of evidence emanating from disability research, including high quality research supported through funding from NIDRR. We will also be supporting training activities on conducting high quality systematic reviews in the coming year. We're privileged today to have as our presenters two researchers who are very active with the Campbell Collaboration. They are Dr. Chad Nye and Dr. Herb Turner who serve as coeditors of the C2 Education Coordinating Group. Chad Nye is Executive Director of the Center for Autism & Related Disabilities and a professor at Central Florida University College of Health and Public Affairs. Dr. Nye has over 20 years of experience in the area of meta-analysis and systematic review of intervention evidence in the area of disability. Herb Turner is the director of scientific research for the Campbell Collaboration and is the project director for the Middle School Math Review team for the What Works Clearinghouse, a joint venture of C2 and the American Institutes of Research to produce systematic reviews on education interventions in the United States. Dr. Turner is also an expert in research design and analysis, web-based survey research and database design and development. So now I'd like to invite Dr. Nye to begin today's presentation with a brief overview. Chad... >> CHAD: Thanks, Joann, and I want to thank Joann and NCDDR for the opportunity to talk with you today. The Campbell Collaboration and the process of developing systematic reviews are kind of near and dear to the hearts of Herb and I, and we hope we can at least share some information with you that may be useful to you in your own work as well as kind of let you know what -- what's going on in the area of systematic reviews that can impact both professional and public policies, programs and so forth. So thanks again for the opportunity. >> JOANN: Chad, can I ask you to slow down just a little bit for the benefit of our realtime captioner. >> CHAD: Okay. >> JOANN: Thank you. >> CHAD: Let me just go through a couple of points of the presentation as an overview to tell you what's coming in the next few minutes. First of all, I'm going to share with you some examples of what I call unusual systematic reviews. They are reviews that you maybe haven't thought of or topics you may not have thought would be something that a researcher would be interested in, but I think they are kind of fascinating for their scientific basis. And then Herb is going to share a bit on the background of systematic reviewing from a historical perspective, particularly in terms of the individuals who have been instrumental in the development of the Campbell Collaboration and the organizations that are involved, engaged in systematic review activities around the world actually. And then he'll also walk through some of the steps in the systematic review, some fundamental, basic steps for the conducting of a review and a couple of examples we'll share with you information from a review that we have done in parent involvement in academic a achievement. And then lastly, I'll share with you some closing thoughts about how to get started -- topics, finding topics, dealing with some of the issues of resources and accessibility that may impact decisions about systematic reviews that you might conduct. And in the end we'll have time for questions and answered and certainly if you have questions during the process, feel free to E-mail those in. So if we could go to the next slide. Unusual systematic reviews -- I just have three examples here that I think are kind of unique. And let me say that the purpose of these is to sort of illustrate the breadth of systematic review and evidence-based research. There are many other reviews that could fall into this category. I have kind of a collection that I've been accruing over the last few years of ones that are just topics that I would have not thought of in my wildest dreams really. So slide 3, please. The first one I call it the walk, don't ride review. And as you might guess just from that, it's got something to do with transportation. So slide 4, please. This was a study on promoting walking and cycling. The purpose here was to try to find out what the research -- what research had been produced that could give us some indication from a public affairs sort of perspective, if you please, about how to get people out of their cars and involved in walking and cycling activities. So this was the review by Ogilvie and colleagues. Next slide, please. This is the objective. It's fairly simple -- promote walking and cycling, and then also to find out does it have any effect on health if we believe all that we're told, anyway, or at least some of what we are told. Improving our health will accrue as a result of exercise. Next slide, please. The authors summarized what they knew about the topic and here is the two points they basically were able to come to: That if you can get people out of the cars, it reduces congestion, traffic. They should have lived in Orlando, and, two, that there really isn't a lot of good evidence on which kinds of interventions are effective in accomplishing that task and getting that shift from cars to walking and cycling. So that served as sort of the basis for -- the rational for conducting the review. Now f we went no further, at least in my experience, I'd say it probably isn't too much research on this in any kind of scientific, systematic evidence way. Next slide, please. In fact, I found 21 studies, and they found five different categories of interventions. That's a fairly substantial database to work from. Getting people out of their cars and on bicycles or walking has a history of scientific investigation. Next slide, please. And the findings were -- we could say some would be expected but at least we've got an objective finding to it. And they tell us that, in way, if they can get about 5 percent shift, if there is some kind of subsidy, such as you build a new railway station or you subsidize the cost of travel, you can have some effect on getting people out of their cars, although I'm not sure how getting on a railway institutes walking, but I don't know, you have to walk up and down the train. But the interesting ones are the publicity campaigns and engineering measures don't have much of an impact. Interesting, too, is that they found no evidence about the health impacts, that those were outcomes that weren't measures. Well, walking or cycling versus riding in your car certainly would have an environmental impact and folks who are involved in the engineering or the development of road systems and transportation systems in large cities would have an interest in that. Next slide, please. The next one is my red light district one. And it's not quite as exotic as it might sound. Next slide. In this study, they are looking to see what's the probability of reducing traffic collisions, accidents, by using cameras. And you probably heard of it on the news or seen reports of trials of this where cameras are placed at an intersection and try to catch the red light runners. And where I live, I would find this a very useful study. I don't know about you, but it seems like I'm having to be more careful about crossing the intersection on those green lights turning to yellow. Next slide, please. The objective was to find out if the speed cameras reduced collisions and casualties. A simple objective, and certainly an important one. Next slide. I'll show you this one just to show you some of the sources that the scholars drew upon to do this review. This isn't a fairly small restricted kind of resource. Notice Medline, Cochrane group is in there. There are studies in the Cochrane library that you have some exposure to last week that bear on this subject. Transportation databases, social science indexes -- there are a variety of resources that are part of the systematic review process and that these authors drew upon. Next slide. The effect of the cameras -- they just wanted to see can we reduce collisions, reduce injuries and reduce death. Next slide. The results are kind of interesting. They found 14 studies -- none of them are randomized trials. 13 of the studies showed an effectiveness of over four years beyond the time of implementation. That is the result lasted, cameras seem to work and in fact you can see the reduction in the outcomes was pretty substantial. I think that that's kind of an interesting piece of data for public officials who are looking at regulations to consider when thinking about how to deal with some of the difficulties and experiences of red light runners and accidents that happen as a result. Next slide. What they found though was that the evidence is relatively weak. That what they are dealing with is observational studies, that there weren't any really good control studies. Point here being that the systematic review pointed to the need for additional research in a more controlled, scientific fashion. They could conclude, of course, that the cameras are potentially, useful, important, and can save lives and reduce injuries and so forth. So the call for more research now becomes a bit more focused on a particular kind of research that would answer the question. And the next slide. My last one is one that you might look at the title and say, well, duh, can't we figure that out without a study? That pool fencing prevents drowning in children. Next slide. The fact is that it is a highly -- a high frequency occurrence in deaths of children -- injuries and deaths to children. And it's highest of course in younger children. So it has a significant personal and public impact. Next slide. The objective is really simple. If you put a fence around the pool, do you reduce the number of children who drown? Next slide. They were looking at different kinds of fencing though where the pools were fenced versus unfenced versus situations where the pools only are surrounded by a fence or where there is a fence around a larger area such as the house that isolates kids from outside. And then they are looking at the risk percentage that needed to reduce the drowning rate. Next slide. Well, the result is not too surprising in that it did reduce significantly the risk of drowning; but more importantly now, they have some objective quantification for this and objectives measures for this that says the best fencing is the isolation fencing where you close off the pool independent of anyplace else whether it's the house or the yard or whatever. The policy for this is pretty significant. Look at the next slide. From this, a policymaker, a city planner, city commissioner who is faced with increased drownings of children looking to make policy without being intrusive as a government body, now has some evidence to say maybe what we should think about here is regulations that require pool enclosures with latching gates. Maybe we need some legislation that funds educational campaigns to address the issue. And maybe we need to deal with legislation of some sort or regulation of some sort that requires the fencing whether on new or existing pools or in some way make use of the evidence that says you can save kids' lives by simply putting a fence around the pool. Well, those are my three unusual ones anyway, I think they illustrate sort of the bigger picture of why you do systematic reviews and what are the bigger picture, the broader scope of evidence that can be use in the our society? We are all probably -- recognize that a lot of research just sort of gets buried in a journal and there is a few people that will read it and can have limited impact; but a systematic review has a way of being able to draw on the rich resources that are available in our scientific community on a topic, summarize that information and provide some important guidelines for policy making, for further research, for programmatic decisions. And so now we'll move to the -- talking a bit about Campbell and the systematic review in particular and here is where Herb is going to step in and take over and talk about the background to systematic reviews and the Campbell Collaboration. Herb... >> HERB: Thanks a lot, Chad. I always enjoy listening to you talk about up usual systematic reviews. It's a good reminder that there may be topics out in a particular field where one might think there is not necessarily evidence, but actually through a comprehensive and unbiased search, one might be surprised at the type of evidence that's being published on a particular topic. As you said, we certainly want to -- one of our purposes for the webcast today is to explore the Campbell Collaboration and how systematic reviews are conducted within the collaboration with a view towards informing policy. But before we do that, there is an adage that says in order to understand where you're going, it's important to understand where you've come from. And so with that in mind, we're going to explore a little bit about systematic review heritage. So we should be on slide 22 right now in terms of systematic review heritage and I'm going to ask that we now go to slide 23, which is the development of the field of systematic reviewing. And if you look at this slide long enough, you will begin to see some very interesting trends in terms of the field of systematic reviewing. For example, one thing that you'll notice from the slide -- and this comes out of the research that Chad and I are conducting on systematic review organizations and their growth and development and how one might want to think about organizing resources for systematic review production. You'll notice from this slide that prior to 1987 there were not organizations that were involved in producing systematic reviews. This slide indicates or shows that prior to 1987 there was little to no systematic reviewing activity going on, at least in an organized fashion. And although typically there is a tendency to think about systematic reviewing beginning in 1993 with the advent of the Cochrane Collaboration, in fact, there were two organizations -- the Swedish center for technology assessment that was conducting systematic reviews in 1987; and the center for the study of learning performance in Canada that was conducting systematic reviews in 1988. But of course most people that are new to the field of systematic reviewing or are just getting their introduction, it's rare that you have an introduction to systematic reviewing without learning about the Cochrane Collaboration which was founded in 1993; but also the epi center which is in the UK and focuses on systematic reviews in education also had its founding in 1993. It's also interesting if you look at this chart is that in 1994 the centers for review and dissemination was founded in 1994 and it was founded in part because of the UK's government desire to evidence and to learn about evidence from systematic reviews a bit more rapidly than the evidence that was produced by the Cochrane Collaboration in 1993. And so this was an attempt to form a systematic reviewing organization that was basically funded through contract work with the UK government. And then in 1995 the joining of Briggs institute was found and it was actually a group of people that were part of the Cochrane Collaboration that had a specific interest in better integrating qualitative research and conducting systematic reviews in nursing. And also you'll notice that most of the organizations -- this is between 1987 and 1995 -- in fact, all of these organizations that were founded whose purpose and primary mission is to conduct systematic reviews were outside of the United States. And in fact, the first organization that became involved with reviews was the blueprints for violence, this was in 1996. Although they didn't delve into systematic reviews in the strictest sense, they were looking at evidence in general, but evolved to begin to conduct systematic reviews. It was actually the first organization in the U.S. to get involved in systematic reviewing. There was a small evaluation shop that is run by Mark Lipsey, but the first organization that was founded along the scale of the Cochrane Collaboration at least in its early days was the Campbell Collaboration which was founded in 2000. And that in a sense has set off an explosion of interest in systematic reviewing. We have the What Works Clearinghouse that was developed in 2002 and there have been a number of federal agencies since then that have an interest in evidence. But I think the key point here is to remember that there is a tremendous amount to be learned from our colleagues across the pond in the sense that the origins of this field has its roots deep in countries outside of the United States or other than the United States. Next slide, please. I think it's interesting to note that when we looked at our small sample of organizations, we're conducting various rounds of data collection because as we identify organizations, there are other organizations that we're told about, so we're basically taking a snow ball sampling approach, but in the 11 organization that is were in our pool, basically most of those organizations -- eight of them were what we call contract organizations. So these are organizations that are funded and conduct contract reviews that have a project management coted with them. They have time lines and there are dollar costs associated with each step of that process. As opposed to what we call interest -- organizations that conduct interest reviews like the Cochrane Collaboration, the Campbell Collaboration and the Joann and Briggs organization where they review topics are driven by the interest of the reviewers and it's primarily while the organization provides a plethora of resources for assisting with conducting a systematic review, the actual funding for the review is left up to the reviewers to secure. Slide 25, please. So that is a brief history of the field of systematic reviewing and if we actually could chart the growth rate of this field, it seems that at this point that that growth rate is accelerating in terms of the interest and so we think it's also important to -- as we move to talking about systematic reviewing at the individual level, that we talk briefly about the accepted definitions of systematic reviewing within the field. Slide 26, please. And this definition basically comes out of an article published by Ian Chalmers. Actually it was Chalmers, Hedges and Cooper in which they defined a systematic review as the application of procedures that limit bias in the assembly, critical appraisal and synthesis of all relevant studies on a particular topic. A meta-analysis may be but is not necessarily part of the process. And we will delve more deeply into this as we move forward in the presentation, but this is very important definition because one key aspect of the definition has to do with procedures that limit bias which imply that one may want to consider having multiple individuals working on a review and it's limiting bias around and entire process in which one assembles, appraises, and synthesizes all the relevant studies on a particular topic. But also it's important to reflect on the fact that a systematic review is considered an umbrella process in which a meta-analysis may or may not be a part of it. Slide 27, please. Now, in the event that we have as a result of doing our review that we do have quantitative studies that are comparable, then a meta-analysis is defined as follows: The statistical synthesis of the data from separate but comparable studies leading to a quantitative summary of the pooled results. So here what this definition illuminates is that within a systematic review, if we have studies that are -- they are distinct but they have comparable outcomes, that there is an opportunity to try to quantitatively synthesize those studies so that we can make a statement about the effectiveness of an intervention, for example. Slide 28, please. And an organization that is dedicated to promoting the conduct of systematic reviews is the Campbell Collaboration. I think it's safe to say that the Campbell Collaboration is a fairly well branded commodity within the United States, and there is a lot of interest in what Campbell does and what it is. And basically, the Campbell Collaboration is an international and multidisciplinary group, and we'll explain more of that a bit later with the organizational chart in terms of how it's multidisciplinary. It's mission is to prepare, maintain and make accessible C2 systematic reviews of the effects of interventions. And this mission is important in terms of the distinction where we say prepare, maintain and make accessible because for example the unusual reviews that Chad pointed to -- there is a good chance that those reviews were published in a journal and oftentimes there is not a requirement to update those reviews. So, for example, if additional studies came out on the effectiveness of pool fences, the journal does not necessarily require that the authors come back and update the database and publish the result. They are certainly welcome to do that, but not required. Where the Campbell Collaboration requires an updating of the evidence, and that's where the idea of maintaining it and of course making it accessible has to do with providing access to the website. It's based on the -- in principle -- on the Cochrane Collaboration. It was actually sir Ian Chalmers who heard an address of the royal statistical society who after about five years after the tremendous success of the Cochrane Collaboration posed the question, what prevents us from having similar organizations that look at interventions in crime and education and social welfare. And so Ian Chalmers made a bee line to the United States to find Robert F. Berush after reading his book on randomized controlled trials and invited Robert to start the Campbell Collaboration. He convened a group of leading scholars and policymakers and practitioners in London and in 2000 the Campbell Collaboration was born. In terms of the objectives, the objectives are basically fairly straightforward, but in many ways complex in their implementation. The objective is to be transparent and require high standards with respect to the evidence, have international teams of collaborators to focus on current and emerging technologies and methods, provide worldwide westbound approach to information access and to continuously update its registries. Slide 30, please. What are the assumptions behind the Campbell Collaboration? Well, one assumption is that there is an increasing public interest in evidence-based policy, and seems that that assumption which was made in 2000 has been validated to a certain extent because of the increasing growth in organizations that are producing reviews and ancillary products around reviews. There will be increased scientific government interest in accumulation and synthesis of evidence. The increased use of R. C. T.'s, high end Q. E. D.'s to generate evidence on what works. Again, an example of how this particular assumption in 2000 seems to now be validated is in terms of -- take for example in education the institute for education sciences now requires its regional educational laboratory programs -- the programs -- each of the labs across the country are required to conduct randomized controlled trials as part of its task and that's in education and we are seeing evidence of a growing interested in funding of R. C. T.'s in other areas. And here we have increased access to information and the need to screen and related to this is work that was done by Fred Mostellar and his colleagues around their estimates that approximately 10,000 studies are published each year in education alone. And so with the increasing growth in publications in education, for example, there is a growing need to be able to systematically assess this evidence. Slide 31, please. And our target audience basically are policymakers, service providers and their professional organizations, public/private agencies, researchers and evaluators, university faculty and students, media people and corporations. Slide 32, please. Basically, the way that the organization is structured is there is a corporate board which is not shown on this slide that has fiduciary responsibility for the organization and then there is the steering group and the Secretariat which has responsibility for the intellectual property of the organization and the direction that it takes with respect to producing reviews and influencing policymakers and building partnerships and then there are the coordinating groups that are responsible for coordinating the review work that occurs within the collaboration. And those coordinating groups are basically denoted by various disciplines that the Campbell Collaboration has interest in seeing whether or not interventions work in those areas. So we have education, crime and justice, social welfare. There is a methods group and this is where it functions a little differently from how the methods group and the Cochrane Collaboration functions in that it really is a centralized clearinghouse for the methodological integrity of our systematic reviews. And then on the chart it says the international communication group but really that group has been changed to the consumers group and they are charged with what many would call the knowledge translation function within the organization. And the review groups below basically show the organization's vision towards having within education for example you could have various groups around particular educational topics that want to do reviews. And so the organizational structure makes room for those types of groups to emerge. And then finally, there is a C2 databases which we will talk about next, basically the products of the collaboration are deposited in these databases and consistent with its mission in providing worldwide web access to these products, these products are available through the web. Slide 33, please. So, for example, with C2 databases, there are three main databases. There is C2-SPECTR which is basically modeled after the Cochrane controlled trials register, although it certainly does not have nearly as many citations or studies, but this is an effort to move in that direction. There are 13,000 citations on randomized trials or possibly randomized trials. There are C2-PROT. There is a growing interest in identifying trials that are planning or underway. So, for example, again in education, there are a number of trials that the regional educational laboratory -- the labs are actually planning at this point. And so a register like PROT would have citations to those trials so that people that are conducting systematic reviews can know in advance that there are trials being mounted in a certain area. C2-SPECTR has actually citations to trials that have been completed and actually published either formally in journals or informal in reports. And then finally and most importantly is C2-RIPE which is the register of reviews of interventions and program evaluations and that's where our systematic reviews are housed. Slide 34, please. For an organization which builds on the enthusiasm of its participants, funding is always a challenge and over the years the Campbell Collaboration has been funded through grants from a number of organizations. I want to mention them because of their generosity the rock ea feller foundation, the Robert Wood Johnson Foundation, Smith Richardson foundation, the Knight foundation, the Jerry Lee foundation, and more recently we had the Campbell Collaboration has formed a partnership or an affiliation, excuse me, with the American institutes for research which have greatly helped the organization in terms of its infrastructure, building its infrastructure. And then there have been contracts that have supported the work of the organization, the U.S. Department of Education, in terms of the partnership between the Campbell Collaboration and the American Institutes of Research on the What Works Clearinghouse and that has helped to fund the work of the collaboration and its Secretariat. The UK home office has funded systematic reviews. The Swedish council social research and the Danish national institute of social research have all provided infrastructure support. Slide No. 35, please. Of course a Hallmark of an organization has to do with the products that it produces, and there are at least seven products. The Campbell Collaboration's product line is continually evolving, so I say at this point there are at least seven. That's the product No. 1 are the registries of the systematic reviews. The second product is the registry of the reports of trials. The third is -- and this is the intellectual property of the organization where we talked about the standards of evidence for conducting C2 systematic reviews. And there is a tremendous amount of thinking that has gone into what constitutes reliable and valid evidence for systematic reviews, what are the methods that should be used in conducting systematic reviews, and so the Campbell Collaboration's almost seven years of work in this area is valuable intellectual property and is embodied in the standards of evidence for conducting the reviews and also related to this, which we don't have on this slide, is that there are policy briefs. So there is a statistics policy brief. There is an information retrieval policy brief. There is a research design policy brief, all which provide guidance around how to handle that aspect of a systematic review. So the standards are very important product of the collaboration and are followed by people throughout the country. Then there is the annual Campbell colloquia a chance for people to come together and break bread together and to talk about methods and life in general and to get to spend time with one another. Training for producing systematic reviews, there are new technologies and methodologies, once again, our methods group are leading the charge in that area in terms of how do you deal with heterogeneity in systematic reviews, how do you control for various factors that can account for variation and affect sizes. And finally, there is the website and our url is there and we certainly encourage you to go to the website and take a look at what we're up to. Slide 36, please. Now, I'd like to get to the heart of the matter because I would suspect that a number of the participants in this webcast may be wondering, okay, I've heard about a systematic review. There is a great deal of enthusiasm, interest, momentum around systematic reviewing. What's really involved in this process? I think Chad and I have a unique relationship in that Chad is the person that -- that got me involved in conducting systematic reviews. I knew very little about reviews -- systematic reviews about five years ago, and it was Chad that invited me to be a part of a parent involvement review and I've been hooked on the process ever since. So I want to talk a little bit about that process and some of the benefits of it and what it means. So we'll talk about the eight steps in the systematic review. Slide 37, please. And although this is called the eight steps in a systematic review, in a C2 systematic review, excuse me, and again getting back do the organization's intellectual property, this reflects the work of our members of the methods group as well as our co-chair incoming up with this particular -- these eight steps which actually apply to most high quality systematic reviews in general. And those steps are -- to start with formulating a review question. It sounds a lot simpler than it is in practice in terms of -- because it basically involves not only developing the theory and the conceptual framework in which the question will be posed and how that framework will be used to investigate the question, but it also embodies asking the question, what's the available evidence on a particular topic. And then defining the inclusion and exclusion criteria that is determining the definition of the intervention, the types of outcomes that will be eligible for inclusion in the review, the target population, the type of research design that will be used, and so all those characteristics of a study or of the target population in the study or of the intervention are considered a priority before starting the review and are laid out in advance. And then there is locating the studies. Some have -- well, there is a considerable amount of research to show that this is the -- this is probably the most time-consuming aspect of the review in which one must be very transparent about search terms that are used, the databases that are searched in order to -- and the transparency is important in order to emphasize the comprehensiveness of the search and for people to be able to -- readers of the review to be able to evaluate whether or not the search was conducted in a way that minimized bias. And then based on the criteria, the inclusion/exclusion criteria that's applied to the studies that have been located and consequently one selects the studies, and then there is a process of assessing study quality. This is probably one of the least well-defined areas in systematic reviewing. I mean, there is a lot of work being done in this area, but right now, there has been a reliance on checklists because of -- we don't really at this point have a full grasp on the psychometric properties of quality scale. And so while -- people always talk about this particular aspect, assessing study quality, it's a very challenging dimension of a systematic review and there is still a lot of work to be done in this area to bring it up to the level of empirical rigor that you might have, say, for doing actually the synthesis, but nonetheless, it's still important to assess study quality by looking at basic characteristics of your study such as what type of research design was used, how was the randomization conducted, was there an intention to treat analysis done, i.e, people that started the study, are those people that were included at the end of the study, even though they may not have completed the intervention regimen, severe attrition, that is, have you had a tremendous loss of participants from the time the study started until the time study ended and so forth. And then there is extracting the data and this is again probably -- not probably -- it is a distinguishing feature of a systematic review from a literature review, per se, because here you're literally documenting the various characteristics of the study using a coding sheet. Some have made the analogy of conducting a survey of studies. Obviously this study cannot -- it's not an oral survey, but it's one in which you as the reviewer literally have a coding guide and you mark off the characteristics of that study and put it into a database. And then based on that database you analyze and you present the results of that study. And this is the phase where -- of the review or the step where people refer to it as the meta-analysis and then of course you interpret those results. Slide 38, please. And so all of this process-wise, the way it takes place in the Campbell Collaboration is that one registers a title with the organization and you provide a basic overview of the systematic review and then a title search is done to ensure that the reviewer has not been conducted previously because one does not want to duplicate evident. And once the title is registered with the Campbell Collaboration, as slide 38 shows, we basically have a protocol which is the research plan and that protocol has a cover sheet. It provides background information in terms of what is the problem, what does one know about the problem, and how is this review going to help address the problem or the research question. And then there are the objectives and then there are the methods. And these methods basically reflect the steps that we have discussed in the previously previous slide except one literally goes through the various decisions that one an advertise pates making about a literature review. And so the protocol really is, again, getting back to the transparency, the idea of the protocol is to be -- is to determine up front what the plan will be for conducting the systematic review ranging from the inclusion criteria down to the statistical procedures and the conventions. Slide 39, please. And then basically that protocol goes through a peer review process and also there is a requirement within the collaboration that at least two people will work on a systematic review. And this goes back to the idea of limiting bias. Now, of course this isn't the only way to limit bias and having two people working on a review does not guarantee limiting bias, but there is a good chance that when you have at least two people working on a review, and the requirement that two people code the studies and work together, one hopes that there is a counter balancing there if there is a bias on the part of the -- of a respective author. But once you've submitted the protocol is submitted to the appropriate coordinating group, so if you're looking at the effects of an educational intervention on a particular outcome that particular protocol would go through the Education Coordinating Group and then it's peer-reviewed. So it would be reviewed by two people selected by the co-chairs of the group in consultation with the coeditors, two substantive experts in the area would review protocol and then the methods would select a methodological expert to review the protocol. Once it passes peer review, the reviewers can then start on working on the review. They've completed the plan and then they implement the plan in the form of a review. Slide 39, please. Or we may already be on slide 39. But basically then we have the review and the basic elements of a review are a cover sheet, once again, the background and in a sense what happens is the protocol provides the core document and basically the protocol is a lot like the review, except the review will have the results and of course there are modifications that need to be made to the language of the protocol, but the protocol forms the core for the review. And with the review you have the cover sheet, the background again, the objectives of the review. You'll notice those two sections were in the protocol. The methods -- you'll notice also that section was in the protocol. And then what we should have here, but is not, is the actual -- the results. You'll have the results of the review and then basically there will be the time frame, the updating plans. Notice that's very important in terms of maintaining the review so this is a commitment to -- and an acknowledgment that the result of a systematic review, a Campbell systematic review are by no means the last word on the effectiveness of the intervention. It's an acknowledgment that we're dealing with, for the most part, an organic database that is constantly growing and so, therefore, the review needs to be updated. And then there are the other areas from acknowledgment to the actual tables. Slide 40, please. So now I'd like to give you just briefly an example of a review because I often find that people are curious. Particularly those that are new to the field of systematic reviewing, they are curious as to what do the results of a review look like? What does the results of a meta-analysis look like? When we talk about that, what is it we're referring to? Slide 41, please. What I'd like to talk about is the results of our parent involvement review and actually the order of those names -- I'm first because of the -- this is the presentation of the slide, but actually Chad is the lead author on this review and was the principal investigator on it. Essentially, you can learn a little bit more -- have a brief overview of this review in the plain language summary that was provided on the website. And essentially our goal here was to summarize the most dependable evidence on the effect of parent involvement for improving the academic performance of elementary school-aged children in grades K. through 6. And in a systematic review course at the University of central Florida that Chad and I, along with Jamie teach each spring when we presented the results of this review, one of our students said why do we need to do a review on something that is so obvious? That is, it was fairly obvious to this student at least that parent involvement should matter, but we actually found two newspaper articles, both of which contradicted each other in terms of the effects of parent involvement. There was one large scale study done out in California which basically there were large headlines in the L.A. times which had parent involvement does not matter or is not as important as other factors. And then there was another article that talked about parent involvement is important. And so it's not unusual to get conflicting findings reported particularly in the media about an intervention. And so we basically searched 27 electronic date bases. These databases were funded or maintained by publisher's in the United States, Canada, the United Kingdom and China. We basically in terms of our selection criteria -- well, as a result of our search strategy, we identified over 800 potentially relevant citations and in terms of our selection criteria, we basically defined parent involvement as those parents who were actively engaged in academic support activities with their child outside of school for a minimum of 20 days. And, again, going back to the emphasis of Campbell on transparency, I'm sure that there are people that would want to debate this definition in terms of whether it's broad enough, whether it was too narrow, and we defined it narrowly for a number of reasons, but the point here is that we're transparent in our definition. And so even if there are those that disagree, we can at least be transparent as to how we defined it and how that definition for the critic relates to the results. And so that was our selection criteria and basically it's also important to note that when we talked about parent involvement, we were essentially looking at studies in which parents and their students were randomly assigned to at least two conditions: One condition, parents were either enrolled in a training program that went over positive parenting practices, particularly around academic enrichment, such as reading games, playing math games with their child or reading for 20 minutes with the child in the evening at least three times a week. Or parents were given literature and information about positive parenting with respect to academic enrichment. So that's just an example of what the intervention groups would have looked like in terms of receiving training to be more involved with their children. Or parents were in the control group which meant they did not receive that training or they did not receive that literature or that material. So this is not to say that in the control group parents were not involved, but the point is the parents didn't receive the stimulus to be involved in the control group. And so that's the distinguishing difference, and we only included randomized controlled trials which meant that the pool of parents and their children were randomly assigned to either the intervention group or to the control group. And we ended up with 19 studies. And so if we could go to slide 42. Basically, the way that one -- when we applied our inclusion/exclusion criteria, so we had defined parent involvement. We focused only on outcomes that related to academic achievement because we were interested in how parent involvement affects academic achievement. And when we specified our target population as children in grades K through 6, and we focused only on randomized controlled trials defined as pools of parents and their children that were randomly assigned to a parenting condition or a control condition. And so with that criteria, when we applied that to the studies, and remember we started with about 800, that's when we ended up with the 19 that were in our final pool of studies. And so, remember when we talked about the definition of a systematic review in terms of procedures -- series of procedures that limit bias in the critical assembly, appraisal and the analysis of the data, and that it may or may not include a meta-analysis, in this case we had enough studies. We had quantitative outcomes to do a meta-analysis. And a meta-analysis is often displayed through a forest plot. There are other ways to display a meta-analysis, but we think that the forest plot is probably the most efficient and informative way to display the results of a meta-analysis and I'd like to explain why when we turn to slide 43, please. And the reason why we think it's efficient, because if you look here at this slide, it basically shows us the 19 studies. You can see them all in one slide. You can see that it shows the condition or the group. So you have all of these studies -- each study had a parent involvement group and a control group. That's what was being compared, the mean outcomes. And you can see we also have the outcomes for each of those comparisons for each study, and then you can see there are two columns which shows the sample size. That is, those are the number of students -- the number of children that were in the parent involvement condition and the number that were in the -- that were in the control condition. And also you'll notice that you'll see squares to the far right, and those squares essentially show us -- and there are different sizes, so those squares show us the larger squares have the larger sample sizes and the smaller squares have the smaller sample sizes and you'll also see there is a bar -- a horizontal bar and that extends out. And so the longer horizontal bar suggests that we have a wider confidence interval which corresponds to a smaller sample size, which means that our estimate -- that little square that we see -- those squares that we see, there is a lot of variability around that estimate. And then if you look at the bottom of the forest plot, you'll see that the right side to the right of zero favors the intervention group, meaning that there was a positive effect. And to the left favors the control group, meaning there was a negative effect. And if one of those horizontal bars extending out from the boxes, which represent the effect sizes, if they extend across zero, that means even though we have a positive effect, that effect may be due to chance. So, for example, if we look at the third study from the top, which is the Klegg study, there is an effect size of .776 or about seven-tenths of a standard deviation which is a fairly large effect size, but the horizontal bar touches the zero, and it's a fairly -- it's a fairly long horizontal bar. So it suggests that even though we have a fairly large effect, it's due to -- it may be due to chance. And it may be due to chance meaning that if we could theoretically replicate this Klegg study repeatedly, we may get a different answer each time in terms of this positive effect size, meaning it's positive this time. It could be much closer to zero the next time we would do the study or could even be negative. And so -- but there are also studies where it's positive and the confidence interval does not cross zero, which means that if we were to replicate the study, for example, the 1964 study which had about a 3.35 standard of deviation, that result is not due to chance. And we can be fairly confident that -- 95 percent confident that if we were to replicate this study, we'd get a fairly similar result. What we learned also from this forest plot, it really does speak volumes about what we can learn from the field of -- about the field of parent involvement. For example, we know from this systematic review in looking at this forest plot that a lot of studies in parent involvement are under powered. That is, they have very small sample sizes. The sample size ranges from as low as 18 participants to as many as 232. And we understand that because if we were to show you another slide, you would see that about 60 percent of the studies conducted on parent involvement are actually -- were dissertations. So you had doctoral students who are trying to complete their doctorate. They have a very limited budget. And so there was not an opportunity for them to invest large scale resources into recruiting a large sample so you could get a fairly large trial. And finally, because we can spend a lot of time on this forest plot in terms of the information it conveys. I'd like to show you if you go down to the very bottom of that forest plot where the column that says hedges G, the last line where we see random effects and we see an effect size of .482 and essentially that represents the effect size summed over the weighted average of all of the studies. And you look at this diamond at the bottom at the right, and this is really the essence of the meta-analysis in terms of the summary -- remember in that definition, it's a summary statement of the pooled results. That's what that diamond represents. It's the summary statement; but it also represents in another sense given this particular database -- remember we only including randomized controlled trials that in a sense it represents the results we would get if we had a large scale synthetic trial. That is, this represents a large scale -- in a sense it's a large scale trial, but I say it's synthetic because it was conducted with different populations and different settings and so we think that this particular forest plot also makes the case for the need for large scale trial to determine whether or not we can replicate this -- almost a half a standard deviation effect of parent involvement. And when you consider it's a very large effect, and you consider how much time parents invested in the method of this intervention, which was on the minimum was about -- the minimum was 20 days. And the maximum was a year. Slide 44, please. So we would like you to just -- as we move on here, there is just two more topics I'd like to cover. And then Chad is going to talk about actually how to get started on a C2 review, but in the event that I know that a number of you are actually -- a number of you are actually working on conducting primary studies, and we just wanted you to begin to think meta analytically that say in the 1980's you remember when there were very few organizations conducting systematic reviews to now 2006 where there are many organizations conducting systematic reviews, the probability that your study may be included in a systematic review is much higher than it was in 1980. And so reporting, making sure how you report your study because there may be someone who retrieves it and then is actually going to code the study, and it is in the best interest of science to have as clearly a reported study as possible. So another way to say that is in 2006, reporting your research can be almost as important as analyzing your research. And so there are some standards that there is the society for prevention -- for reporting primary studies. The American Research Association, the consort statement on trials and the consort extended statement for cluster trials in medicine has statements on how to report randomized controlled trials. A quorum statement on how to report a meta-analyses and there are probably a host of others out there that have flown under our radar screen. Slide 45, please. And finally, in terms of C2 futures, there are a number of opportunities that the Campbell Collaboration is working on in terms of growing the organization, but also advancing the field and advancing science. In terms of increasing its production of C2 reviews, that's quite frankly has been a tremendous challenge for the organization, Chad and I have had conversations with Joann about this very issue and I was having a conversation the other day with an aspiring systematic reviewer, and when we talked about the idea that systematic reviewing is hard work, but it's also very rewarding work, and it's very easy though for an organization to get -- to gravitate towards all of the ancillary products around the reviews, meaning the information retrieval, setting update a bases, and as opposed to focusing on the more challenging work which is the production of reviews. So we're hoping that the affiliation -- we're hopeful that the affiliation of AIR will help to ram am up the production of reviews within the collaboration. There is also work being done on a journal of systematic reviews, that is an effort is being made to provide an outlet for systematic reviews and I think this is an acknowledgment by the Campbell Collaboration that it wants to reward its reviewers because the average length of a review can range from six months to two years. And so the collaboration wants to provide an outlet that rewards its reviewers for all that hard work. There are other rewards that come out of doing reviews and maybe we can talk about that during the question and answer period, but this is one way. Speaking of rewards, another reward has been the Campbell Collaboration has set up briefings at Capitol Hill once its reviews are completed. This is an opportunity for policymakers to learn about and legislators to learn about results of reviews and an opportunity for reviewers to talk to policymakers about what they have learned. And then of course C2 is continuing to build its international network of partnerships. There is negotiations underway to form a partnership with a group in China that's interested in conducting reviews. Chad and I actually received an E-mail from a researcher in Thailand who after meeting with Bob Berush learned about our parent involvement review and got a copy of that review off the website and actually read the review and just now feels more comfortable in doing work around getting parents involved in the academic enrichment of the children in Thailand. So those are our futures, and at this point I'm going to hand the webcast back over to Chad who is going to talk about how to get started with a C2 review. >> JOANN: Let me interest erupt for a minute to say we've got about 20 minutes left. So not to cut you off at all, Chad, but we can go a little bit longer if we need to with our question and answer session. Thanks so much, Herb, that was very interesting. >> CHAD: Okay, slide 46, please. Getting started in a review is sometimes the most difficult task of actually completing it. It's figuring out where to start and how to get going. So the next couple of minutes I'd like to at least share with you basically -- primarily on the basis of my own experience, some ideas about how to do a review and how to think about getting started in a review. Slide 47, please. As Herb pointed out earlier, looking at the organizational chart for the organizations around the world that are doing reviews, you notice that there were interest topics and contract -- or interest work and contract work. Campbell review is an interest group. That is, many of the reviewers are doing reviewers based on topics that they have a particular interest or expertise in and background in in their own professional arena. That certainly is an acceptable and a good way to go. If you're not interested in it, it certainly can be a bit more of a challenge. On hot topics though can also be a very rewarding possibility. And in some sense that's how I initially started with parent involvement thinking about what is it that goes on in the school that could be value added, that isn't teaching reading in the classroom. My wife is a teacher in the public schools, and so I get frequent exposure to issues in the schools. And so I thought -- I did some reading and looking around in literature and came upon the idea of parent involvement. It turns out, of course, that the more I investigated it, the more I found that parent involvement is a critical target issue in the No Child Left Behind Act where the legislation has articulated the need for and the interest in parent involvement components for the educational system in our country. I also found a hot topic one -- came out in a publication, international reading association. My wife was showing me the article and said, look at this, and here was a chart with topics that are hot. Topics that are not hot, and then they had five reviewers who went through and checked off the ones that were hot and shouldn't have been, and the ones that weren't hot and should have been. We had 20 topics to choose from out of the reading association's own professional publication. That, by the way, we moved that into a review that is getting underway currently, in fact, as a follow-up to the national reading panel. Policy topics are ones that often come into play -- particularly in the contract area -- where an agency, a government agency, a private agency is looking for some answer to questions. Sometimes we see these referred to as rapid response reviews. And in the Campbell Collaboration, we don't have a mechanism for rapid response, but we certainly can deal with policy issues, policy topic specific such as the cameras for red light runners. It is a hot topic for which there could be significant policy implications, and if that becomes a priority and an issue at a government level, then that can be a care rot, so to speak, I guess, in the review process. So topics that interest you certainly are ones that, you know, come to mind first; but not the only ones to work with. We've been involved for a couple of years with student training and having students do reviews. And what I'm finding is that once we get going on a subject, even though it may not be a subject for which I have much professional experience and background, it becomes a topic of interest. Topic is the single most difficult activity in the process of the review, finding a topic. And so it's not one to just sort of jump into lightly. Some thought and time, preliminary work in library databases will help maybe to focus the interest on that. Accessibility -- we don't want to choose -- for the first reviews, we don't want to choose studies that are published only in Hungarian. Studies for which we can't get access to data, studies that are very restricted, studies, for example, that are of low incidents disorders kinds of things. There is a limited number of elective mutes to do a randomized controlled trial in that area is probably pretty unlikely and finding those studies can be a real challenge because they are located in so many different sources potentially. So can you get to the data, information? Resources is always an issue in doing a review. Campbell Collaboration is a volunteer organization. People may receive funding from other sources, but not typically from the Campbell Collaboration. We look at it in terms of can we recruit students to assist us either as part of their training program or as a resource that's relatively inexpensive, anyway, to help do some of the more day to day kinds of things such as retrieving literature information, making Xerox copies, can they be trained to do coding. Costs -- if you have to pay for databases, then that becomes an issue that you need to take into account because that's going to be expensive. It is a time commitment as Herb said, six months to two years is an average time. I won't tell you how long we spent doing ours, but it was longer than it should have been. Collaboration becomes key. Doing it -- doing a review by yourself is both unnecessary and probably not very efficient. It is that collective wisdom of two or more people working on a review that makes the review really a viable product in the end for which you can hold it up as being an objective summary of intellectual property of research and the wisdom of the discipline. All of those issues and topics, accessibility and resources, are things you need to think about. They don't need to be hindrances to stop you from conducting the review, but you do need to sort of think of them on the front end. Lastly, I'd say from sort of a personal point of view, systematic reviews -- I try to say forget for a moment that in my case I'm an academic. I work in a university. That I'm a professional. My background is in special education, speech and hearing disorders. As a taxpayer, it makes a great deal of sense to me that systematic reviews become an important part of the evidence that's generated from research. Making decisions as a taxpayer means I should be getting sort of a bigger bang for the buck here. I should be able to expect that the scientific community provides evidence that supports decisions, that supports programs or that leads us away from programs, from interventions that don't work, that directs us in identifying the needs of research that advances the knowledge base in our area of interest. So I think systematic reviews has a big role to play in our society beyond whatever academic publications vit tas and so forth that we may find ourselves working to contend with. Systematic reviews have a role in our society at large and worth the evidence have effort in my book, any way. With that, that concludes at least the formal part of our presentation, and Joann, I guess we turn it back to you for any questions you may have. >> JOANN: We do have a few questions and some of them I think you've already covered so I'm going to jump down to some that you haven't touched on that I'm sure on you audience would want to have you comment about. For example, this question is disability research often involves small sample sizes without controls. Can this be used in a systematic review? >> CHAD: Herb, you want to respond to that? >> HERB: Sure. It can be used in a systematic review. The challenge that one will face is the conclusions that can be drawn from a review that includes such evidence. And the issue is not as much the small sample as opposed to the lack of a control group. Now, there is work being done -- because then the issue is the internal validity of the designs and so if you synthesize a set of designs or a set of research that has low internal validity, then by extension you're going to have conclusions that have low internal validity and that's not good for making policy. There is work, though, being done around the single case studies. The singed case studies are acknowledged by methodologies as having very strong internal validity. The work that's being done now is how do you synthesize them and come up with an effect size. >> CHAD: and you can do the synthesis, and I've been working on a couple of them over the last several years. It may be that that's really the only evidence or not the only -- it certainly is the majority of the evidence that's available. For example, I worked in an area of autism, much of the research in autism is single subject research, single subject case study design. The problems that Herb articulated are certainly the quantitative issues related to it. There is a need to synthesize that research, but we really just don't have the methodological rigor to be able to do what we would like with that and to be as confident in our outcomes and interpretations of that data. >> HERB: but that's speaking to the meta analytic portion of trying to summarize those single subject designs, but all the steps that precede the meta-analysis, those seven steps from problem formulation to locating your study through a comprehensive search, through establishing the exclusion/inclusion criteria, to coding those studies. All of those steps which are designed to limit the bias are very important steps. And the other thing that we didn't touch on here in this webcast which is a very important function of a systematic review is also the mapping feature of the review in terms of being able to make some qualitative statements about what types of studies have been conducted, what types of interventions have the studies been conducted on. So there are other dimensions of the systematic review besides just the question about whether an intervention is effective or not that can be important. So for example, sometimes there are interventions out there for which there is little to no evidence on, and to be able to document that is very important. So I think it's important to also keep in mind the distinction between the systematic review and the systematic review that includes a meta-analysis. >> JOANN: That relates I think to one of our other questions which is can you conduct a systematic review of qualitative reach or must it always involve randomized controlled trials? >> HERB: That's a good question. And I think, first of all, to set the context for this presentation of what we just did, it depends -- it depends on the question. So, for example, the Campbell Collaboration is focused on the question of whether an intervention works. And so in that case, to answer that question, one really needs to use -- to synthesize or to evaluate quantitative research. If you think about in a research methods class and sort of the taxonomy of research, qualitative research in its broadest sense was really developed to help understand process and to lay the ground work for developing hypotheses which can be tested which one can draw some conclusions about did significance or whether or not you can reject a particular hypothesis or whether you can test it rigorously. So there certainly is a place for the synthesis of qualitative research, but when you synthesize qualitative research, oftentimes it's done to address a different question. Now, with Campbell, there has been work done in terms of the integrating -- not necessarily synthesizing, but integrating qualitative research into the systematic review so that one can understand the processes that are leading to that summary statement about the effect of the intervention. And a good example of that is there is -- okay, maybe I should stop. >> JOANN: We are getting short on time, but we do have a couple of other questions I want to see if we can squeeze them in. One is how would I go about getting my research study included in a systematic review? >> HERB: Chad? >> CHAD: On one point you could register -- I'm assuming what you mean here is you're conducting a primary study. You could register that with the C2-PROT so that folks would know it's out there. The review of Campbell is the summary of studies that have been completed. So if the data is finished, even if it's not in a published form, that is data that can be integrated if it's specific to the topic and so forth. It can be integrated into a review. You could look at the Campbell website in the C2-SPECTR or RIPE and see who has got a topic that is in development. If you go there on the topics, click on the link, it will tell you whether there is a title that has been registered, whether there is a protocol that has been completed registered and then if the review is finished. If the title and the protocol are still active, you can contact the authors directly and let them know about your work and see if they have interest in including it in the study if it meets their criteria for inclusion. >> HERB: I think another way -- it's unusual for people to sort of lobby or do a study to get it into a specific systematic review. I think one of the reasons why we presented those standards for reporting is one of the best ways to ensure that your study is included in a systematic review is to publish it and publishing it is to report it well. Because a well reported study with a well-defined intervention, a well-defined target population, a clearly written method section, a complete reporting of results will have a much better chance of being included in a review. For example, in our parent involvement study, our parent involvement review, there were 19 studies, but there were actually 25 that met our inclusion criteria, but at least six of those we had to exclude because of missing data or a lack of reporting of data for the study. >> JOANN: Okay, you had mentioned -- I think, Chad, I'm not sure which of you mentioned the colloquia as part of the activities of the Campbell Collaboration. What can you tell us about the upcoming colloquia, when and where is that going to take place? >> CHAD: It takes place in May. I believe it's May 14th in London this year. It alternates every other year from the U.S. and non-U.S. sites and so this year it will take place in London. It's a three day meeting that includes some training programs, training courses, orientation to systematic reviews, orientation to Campbell, as well as some of the more detailed and information about methods, about information retrieval, summaries of studies that have been done and should typically includes folks from around the world which is part of the mission of Campbell. >> JOANN: Okay, great. Well, I think we've come to -- we're actually a little bit over on our time, and I haven't seen any new questions come in. So I think we should go ahead and wrap it up. I want to thank you very much, Chad and Herb, both of you, for your very informative presentation this afternoon and thanks to everyone who did participate with us. An audio file and a transcript of the webcast will be available on the ILRU website within a couple of days. So you can find at www.ilru.org. I want to thank the National Institute on Disability and Rehabilitation Research, NIDRR, that provided funding for the webcast. And I also want to thank especially the staff at ILRU because without their efforts, the webcast would not have taken place. Those include Tanjauna Arnold, Marj Gordon, Sharon Finney, Dawn Heinsohn, Vinh Nguyen, Maria del Bosque, Rob Dickehuth in charge of the technology and Marie Bryant is the realtime captioner. I want to thank you all. Herb, Chad, do you have any final words for us? >> CHAD: Yeah, real quick. If you're interested in a review, feel free to contact us any time. I'll be happy to talk to you about Campbell and the review process and our web links are on the Campbell review. You can find us on the web pretty easily I think. >> JOANN: Sure. And send in any questions you might still have to webcast@ncddr.org and I'll make sure they get to our presenters. We will hope you'll able to join us for our next webcast on November 16th, and again thank you so much for your efforts today, Chad and Herb, and thank you for joining us. Goodbye. >> CHAD: Thank you.