Roundtable Participant: Teresa Wilson, American Institutes for Research/What Works Clearinghouse

MS. WILSON: Hi. The What Works Clearinghouse was established in 2002 by the US Department of Education's Institute of Education Sciences to provide policymakers, researchers, educators, and the public with a central and a trusted body of scientific evidence of what works in education.

The What Works Clearinghouse is a joint venture between the American Institutes for Research and the Campbell Collaboration. What Works Clearinghouse gathers studies of effectiveness of education interventions, and that's on programs, products, policies, and practices. And we review those studies that have the strongest designs. And we report on the strengths and weaknesses of those studies as measured against the What Works Clearinghouse evidence standards. And this is to help consumers make well informed decisions.

We look for studies through a comprehensive search. We do electronic searches. We do hand searches to find all citations and abstracts, and even unpublished reports that appear relevant to a topic or an intervention area. And we also encourage public submissions.

DR. GAMORAN: Thanks. Let me pose the next question to Gina and to Terri. We heard this morning some concerns about whether abstracts might constrain the kinds of research that is carried out. And so, that might stimulate our thinking about whether different users of research are looking for different kinds of things when they search for information.

So, a question I want to put on the table then is whether different users have different standards of evidence. This might be implicit or it might be explicit, but if there are different standards, then that might result in different search strategies. So, let's turn that first to Gina.

MS. BURKHARDT: I don't know very many of you here, but those of you who do know me, know that I never answer the question that is posed, so I'm going to continue my reputation and not answer that question, but something very similar to that.

I think that the debate -- if the question on the table is will structured abstracts allow for access and efficiency to those of us who need to get to the research to translate it to those people in the field who actually use it, apply it, then the answer in my mind is absolutely yes.

That right now the process of searching for research either in the education field or outside the education field, and really we do look at sociology research and we do look at anthropology research when we are trying to bring the entire look of the issue to those people who have to make critical decisions, right now it's a very difficult task.

And it's very difficult in terms of time and efficiency, because if you are a policymaker, you can't wait six months or nine months or two years to get a synthesis of the research to make your decision. We are really on-call to bring the research to our constituents who are often chief state school officers, who are often decision-makers at large districts like in Chicago public schools, for practitioners who are being held accountable to pretty high standards right now, and need information to prove that they have evidence-based or research-based practice.

So, in terms of access and efficiency, having a mechanism that allows organizations like ours, and there are many more, AED is another one, who can take the research that is out there and make it usable, applicable, it really is about knowledge utilization, then the conversation again is put the question to design architects and figure out what needs to be in that kind of design so that the access is available to those people who need this research. That's a moral obligation.

If the conversation is about inside the research community, what abstracts are doing to change the conversation around research, and around how people conduct research or about how people report, that's a very different conversation. And separating those conversations, and answering them in very different ways is something that you might consider, because the research community has a reputation for being able to debate issues like this for a long time, just like every other community has.

And in the essence of that debate, you are going to figure out exactly what you need in terms of accessing the research to enhance the knowledge domain, because basically that's what researchers do. You find out what's already been done, and extend, expand the research. You synthesize it, you repeat it, make sure it's valid.

It's not what practitioners use research for, and it's not the demand they are under right now to use it. So, consider this, if I'm an elite athlete and I have already won five triathalons and four marathons, and I'm looking to enhance my endurance.

I'm going to be looking at the research around endurance, and the tools that are available, and the health clubs and all the support in a very different way, based on my expertise, my experience, and the community in which I exist than if I'm Gina Burkhardt who runs five miles every day, and now has been told that by the end of this year I have to complete a marathon. And if I don't complete the marathon, they are going to take my 13 year old daughter away from me.

But in a way schools are right now being held to continuing support, federal dollars, continuing getting all kinds of resources, keeping highly qualified teachers based on how well they can prove what they are doing works. In order to prove what they are doing works, they are being really pushed to show evidence-based or scientifically-based research to support the practices that they are using.

Giving them access to all of the research in a way that lets them use the research to their advantage, again is a moral obligation of my organization, and I believe you would agree that that's a moral obligation of doing the research that you do.

Figuring out how that access is given is only the beginning, because access doesn't really determine usability or applicability of it. So, the second part of that is in terms of translating the research is one piece, but translating it so that people understand what it means, whether it's applicable to their specific situation is another aspect of it.

So, to make a long story short, or longer, the structured abstracts in a way that allows access and efficiency for organizations like ours, but even more for schools and districts, who themselves are now being pushed to access the research and make decisions from it is pretty critical.

MS. WILSON: The WWC does use evidence standards, and they are rather rigorous. First, we screen studies to determine whether they meet the criteria for inclusion within particular topics for review activities. And we look at the relevance of these studies. We look at the relevance of the intervention of interest. We look at the relevance of the sample population of interest. Sometimes that may be limited by grade or by age. We also look at the recency of the study. We are looking at studies over the past two decades. And we also look at the relevance and validity of the outcome measure.

Then we look at the studies to see whether they provide evidence of causal validity. And those studies are rated according to the strength of their causal validity. Studies with insufficient evidence do not meet WWC standards, and are not included in our reviews.

Studies that place students randomly, or randomized controlled trials without notable design or implementation flaws are classified as meeting evidence standards, and those receive our highest rating of two checks, and are included in our reviews.

Other studies that use comparison groups or quasi-experimental design studies, and randomized controlled trials with some notable flaws are classified as meeting evidence standards with reservations and receive one check.

All of these studies are included in our review. And then we lastly look at the studies again to further review and describe them and rate them on other important characteristics such as intervention fidelity, outcome measures, people and settings, statistical analysis, and reporting.

It's really interesting, because part of what we have been doing is trying to translate this and make it clear to the public how we rate studies against our standards. And so, we have struggled with how to make that transparent through providing what we call right now, study reports. They have gone through several iterations, and we are more and more closely towards something that looks like a structured abstract. So, I think it's interesting that we may be providing that service in the future.

DR. GAMORAN: One thing I'm taking from both of these, and I would like you comment if I'm correct, in both cases there is an assumption that the decision about standards of evidence is not left with the practitioner, with the end user. In your case, a big part of the clearinghouse's mission is to identify the standard of evidence so that the practitioner doesn't have to decide is this study informative about a causal process or not? It's either rated as causal or not by your standard.

And in the case you described, Gina, I had the impression that the big pressure for the practitioners is time. They don't have time to go scour the literature for effective ways of teaching reading. They need a translator like you. And that puts a big onus, a big burden on you and your group to hold standards of evidence yourselves in deciding what information to disseminate. Have I got that right?

MS. BURKHARDT: That's why we are really happy for the What Works Clearinghouse, because it will be blamed on them.

DR. GAMORAN: So, is there an assumption then that end users are not in a position, or lack the time to determine standards of evidence?

MS. WILSON: I think we are just trying to help the users and provide them with some guidance.

MS. BURKHARDT: I think that's exactly right, but I think the assumption that they should have those capabilities are maybe not the right assumptions. That practitioners in the field are not trained to be consumers of research in the same way that researchers are trained to be that way.

So, having the added support of those kinds of categorizations or syntheses or reviews or even support of the research is really important for schools and districts. Teachers in the classroom are now being asked to make decisions that they are not trained as researchers to review.

DR. GAMORAN: Steve or Debbie, did you want to add anything about standards of evidence?

MR. OLSON: It will come up when we get to the next question.

DR. GAMORAN: Good. Then let's move to our next question, which I will invite Steve and Debbie to respond to, which is how well do current systems serve the very needs of the different user groups? You heard Terri and Gina refer to a variety of different types of users, from the teacher in the classroom, to the school learning coordinator, to a district curriculum director, to a superintendent, to a state policymaker.

So, there are many different constituents for this information. And so, we would like to know how well the current systems serve the needs of these different user groups, and whether abstracts in fact are used in these processes.

Debbie, do you want to go first?

MS. VIADERO: I find that often times abstracts are lacking for me. And it's not so much a problem when I'm reviewing journals, because I can turn a page and look at the relevant section of the study that answers those questions that I have unanswered.

But it is more of a problem when I am using a database like ERIC, because up until recently you had to pay to get the full text of ERIC articles. And I found after taking many wrong turns, that it wasn't always worth it to just get everything that looked vaguely relevant to what I was writing about.

Sometimes what would happen would be an article would say this is a comparison of this intervention and that intervention, and it looked useful, but then when I got to the study itself I found that we were talking about something that involved maybe 10 kids and two classes. And that was not something that I felt would merit mentioning in a larger article that I was writing.

Inadequate abstracts also tend to be a bit of a turn off when I'm reviewing articles as well, because if they don't tell me what the conclusions are, and that happens often, they will say we looked at this, and that's all that they say, then I don't have much time. And I might just skip on to the next article.

So, I think that structured abstracts could be useful to me, and could help me do my job more efficiently as well. And when I'm not looking at the abstracts myself, what I'm doing is I'm calling researchers in the field, and I'm asking them what do they think are the key studies in this area. And I'm also looking at the reference list at the back of studies.

But when I talk to researchers I realize that often times they may be giving the names or referring me to studies that echo their own thoughts or findings. And I think that opens up the possibility of potential bias in whatever my end product is.

DR. GAMORAN: What do you do to combat that?

MS. VIADERO: Well, I'll ask people are there studies that disagree with what you have said? And people don't always give you a straight answer when you ask for that.

MR. OLSON: This has been an interesting process for me to listen to people, and it's been interesting to actually prepare for this panel, because it made me think a little bit about how I do research. So, I'll just very briefly describe it, and then think out loud a little bit about what a structured abstract would be.

As I was doing research I realized I sort of need two things as I dip into a field, whatever that field happens to be on that particular day. I sort of need a point of entre. So, I need a book or a person or a listing on the Web of resources. Or most often these of course what I'm using is an electronic database to try to find out as much as I can about a particular subject.

And the other thing that I need, and it's related to what Debbie just mentioned, is I need some sense that in any given discipline I have discovered all the references that are important, or at least the references that sort of frame the issue. So, if there is a spectrum of opinion, I would like to know that I have at least looked at the ends of the spectrum, and at a few places in between, rather than seeing everything that's on one side.

So, often it's best if you can do that through a person, but if I'm working through an electronic database, I have to use the material that is available to me on the database. I think later on this afternoon you will be talking a little bit about PubMed, and PubMed is certainly the database that I use most often. I sort of think of it as the gold standard of databases. I use PubMed in sort of an idiosyncratic way, as I imagine everybody uses the database.

But with regard to abstracts, the value of an abstract to me is dependent upon sort of the context of this electronic database in which I'm using it. So, the structure of a database makes a big difference as to how I will use the abstract, or whether I'll use the abstract, or how much information I can get out of the abstract.

So, when people design databases, I think it's a consideration what kind of abstract is going to go in there. Now, PubMed has all kinds of abstracts. They have will have structured abstracts and little things, and no abstracts whatsoever. And the one thing I can say about it is in using these databases I prefer my abstracts to be short, just because of the way I hop around within a database in this effort to try to find a comprehensive listing of resources.

Now, PubMed has a feature which I always advocate for anyone who has anything to do with databases, and it's called related articles. It is simply the most incredible research tool I have ever had. And anyone that constructs databases, I encourage them to look at the link called related articles. It must take an incredible amount of work to make that link work, but it is the most valuable thing that I have as a researcher.

And the way related articles works is that if I find an article that looks interesting from a title, I can then click on the title, and I will go to the abstract. And what I want from that abstract is a sense of how representative that article is as a summary, or as an example of research in the area that I'm interested in. And I will then hit related articles, and up will pop 500 pieces. And those 500 pieces are sort of organized in the relevance to the article that I started from. It's really how I do all my research these days, or at least 80 percent of it.

DR. GAMORAN: Reflecting on the medical field, which is as we heard this morning, part of where the stimulus for this whole conversation comes from, is there a different expectation, norm, and perhaps ability or capacity on the part of end users to read abstracts and make judgments about research?

We heard that a relatively small proportion of the readers of Education Week are teachers. And I don't know if, Terri, you have figures on what percentage of the users of What Works Clearinghouse are teachers, as opposed to district and state level people. Whereas, I'm guessing that doctors, physicians are using these medical databases to keep abreast of their field. Am I right in thinking that that's an important difference between medicine and education in the consumption of research information?

MR. OLSON: Regarding PubMed, as I say, it does have all different kinds of abstracts on it. So, you will encounter structured abstracts, as well as regular abstracts or conventional, traditional abstracts. And so, it's actually an interesting way to compare the two formats, and see the best ways in which they work.

I can really only speak for myself as to how you use it. In other words, I wouldn't be able to know how doctor or practitioners or other people use them. We researchers can be trained in a lot of different directions, and as the advantages of structured abstracts were made clear to us, I think we would perhaps not react the way we do now.

When you are using a database, you are moving quickly from topic to topic. And when I see a structured abstract pop up on PubMed, I'll sometimes groan, simply because it's a little bit longer than a conventional 120 word abstract, and I want to get the gist of that article as quickly as possible. But then again, as I got used to structured abstracts and could derive more information from them, I think I would get additional value out them.

DR. GAMORAN: Terri, you have also worked in the medical field. Your own background is as a medical practitioner, as a nurse. I wonder if you would comment on this assumption of mine, whether it's right or wrong that medical practitioners and educational practitioners are very different in whether it's normative first of all, and second in their capability of using research information directly from the research source.

MS. WILSON: I'm a nurse. So, I think relating health with education, I think it's very similar, nurses and teachers and how they use the information, and how to provide it to them in a easy way for them to understand, because it's difficult. So, to me, I think there are similarities in the users.

DR. GAMORAN: And so in that case, the kind of broker or translator role that this panel is really about becomes crucial.

MS. BURKHARDT: As a simplistic example that I hope isn't just simple, when you think about if your spouse has breast cancer or prostate cancer, and you bring that person to the doctor, you expect that the combination of the people treating your spouse would have the knowledge from the research about what's the best practice in treating this.

When you take your 5 year old to first grade, you ought to have the expectation that the person in the classroom knows how to teach your child reading based on what the research says.

The kinds of access that the medical field has to information, and how they use it I think is structured a little differently than the access that education practitioners have, and the training that they have in interpreting and using and applying that.

So, if you go to your children's school and you ask that school, how did you decide on the reading or math program, the curriculum that you are implementing in this school? Depending on where you are in this country, and how resource rich that school or district is, you are going to get a different answer. And it really falls back to the ability of those schools or teachers or districts to access and understand and use the information, and the training that they have had to do it.

So, I think the medical field is not a good comparison here. It should be, but I think right now it really isn't a good comparison for education.

DR. GAMORAN: That leads us to our last set of pre-arranged questions for the panel, and I'll ask all the panelists to comment on this set of questions. We have heard that structured abstracts would perhaps be useful for these translators and brokers of research knowledge. And so, let's ask what are the strengths and weaknesses of adopting a policy of structured abstracts?

And of particular interest to us is your views about what the content and format of an ideal structured abstract might be. And this is your chance to comment explicitly on the strengths and the weaknesses of the Mosteller et al proposal for structured abstracts.

So, let me ask each of you to take a few minutes to respond to these questions about how you would like to see a structured abstract, both in the sense of responding to your own needs, and in response to the proposal that was put forth in the first session this morning. We'll start down at the other end with Terri.

MS. WILSON: Well, I think these would be helpful for us in doing our research, particularly the components that we're interested in with regard to research design, and how the research was done and conducted, as well as the population participants, especially if we are looking at a specific grade level, or a specific population. So, I think most of these would really aid our search and evaluation to determine if these studies would relevant.

DR. GAMORAN: But is it helpful because it's more efficient, because the information is organized in a certain way? You still have to read the article, right, to determine whether you accept their statement that it's a randomized trial. And you are making various judgments about whether it's a good randomized trial or a bad randomized trial. Whether it had an appropriate statistical analysis methodology or inappropriate methodology. So, where is the time saving?

MS. WILSON: I think with our first screening, the relevancy screening. Knowing if it's a relevant population, knowing if the intervention is relevant to the topic of interest that we are looking at is the population that we are interested in. So, I think it would help cut time in screening, because sometimes we have to pull articles and look at them, so there would be some time savings in that.

As well as having this available, and then being able to look at the standards of the What Works Clearinghouse, and very clearly understand how a study may measure up against those standards.

MS. VIADERO: I agree with Terri. I do think this would help me just screening out things that I'm not interested in reading. And I'm interested in all of the items that are mentioned in the format. I need to know background. I need to know methodology, setting, et cetera.

Since we are being asked to make a wish list, I will add one more item. I would like to know what the significant contribution of this particular study may be. For example, is this the 30th randomized, controlled trial on class size, or is there something really completely different about the study as well.

And there is one thing that these don't help me with though, and that's cutting through the both education and research jargon. And I have become used to the language now that I have been doing this as long as I have. I do think it's a problem for my colleagues at mainstream newspapers. I think rather than take the time to read further when confronted with the jargon, that they would be more tempted to move along.

MR. OLSON: Well, it's very rare that I get enough from a single abstract to write what I need. In other words, I almost always have to go back to the full article. And so, I haven't decided whether structured abstracts would help me, though I suspect upon certain occasions they would.

When I encounter a structured abstract I'm almost certainly going to start skimming through the abstract to the core of what would have been in a traditional abstract so that I can get the information that I need even from that larger abstract, then make further decisions that I have to make in the course of a literature review. So, I really could go either way as far as I'm concerned, regarding this issue.

MS. BURKHARDT: Well, I think that Felice said the most brilliant thing all day today in the very first session, and that was it really is a decision about whether you are looking at these structured abstracts to be she said a vehicle to accelerate research and knowledge, or a communication mechanism for practitioners and policymakers.

If it's a communication mechanism for practitioners and policymakers, I would say absolutely yes, we need it. It would make our life more efficient. The What Works Clearinghouse was designed to do this, because there isn't a way to get more access to research in a way that we need to.

If it's a vehicle to accelerate research knowledge, then the debate continues in your community about whether or not the example that was put up gets you the information that you need to expand the knowledge domain. But on the side of the practitioner use and the translation piece, I believe it would make our lives and the lives of the people we serve a whole lot easier and better.

DR. GAMORAN: So, this general response I'm taking is neutral to positive, which contrasts on some of the conversation earlier. So, I would like to bring up some of what I heard as concerns expressed in earlier panels, and ask this panel to comment on them.

First of all, we heard a concern that space in journals is precious. This would at least triple the amount of space used for journals, maybe add a half a page or a full page. And is that worth it? So, that's the first one.

The second was the argument that structuring the abstracts would structure the kind of research that is done. And so, I would like your perspective on whether you think there is a danger in that? And if there is, would that be a good thing or a bad thing?

And then the third concern is whether a structured abstract would let the reader think that he or she gets everything there is to know out of the abstract, and make it less likely to read the journal. So, that's the third thing. Would having a better or a more structured abstract -- forget better, that's not my job to judge -- if it's a more structured abstract would that make you less likely to read the whole article? So, I'll throw that open to whoever wants to comment.

MS. VIADERO: It wouldn't make me any less likely to read the article, if the article were something that I would want to write about.

DR. GAMORAN: In other words, if you screened it out, you wouldn't read it. But if you weren't going to screen it out, you would be no less likely to read it.

MS. VIADERO: And I do think that's a potential issue for education writers at mainstream publications. But they are not going to be reading the research anyway. So, if they are just looking at the abstracts, they are getting a more complete picture of the research with those abstracts than they would in a press release or by skipping the article altogether.

MR. OLSON: Space considerations, I think a little bit about the size of the computer screen too. If anyone is familiar with PubMed, you know there is a big difference between abstracts that are on a single screen, and abstracts that go to more than one screen.

MS. BURKHARDT: And I understand in the research community that reading the abstract doesn't necessarily stop you from reading the journal article, especially if it has a specific person's name on the article. So, I think there is some danger, but I think Debbie is exactly right, for those people who aren't in the research community, they are getting more information up front, right from the start that will help them screen things.

DR. GAMORAN: Well, we heard this morning that some journals more than others have broader theoretical, conceptual, philosophical, a variety of concerns aside from issues of is this an effective program. And there is a concern that the specific content in this proposal for a structured abstract would move us away from those kind of more theoretically driven articles towards a more practice or policy-driven article.

Is that a concern from where you sit? To what extent are you looking for connections with the broader social movements or theories or understanding of the way the world works? Or to what extent is that an irrelevancy that you would just as soon do without anyway?

MR. OLSON: I certainly need both of those. So, I agree that a personal essay is shoe horned into the form -- taking an extreme example here of a structured abstract wouldn't work. There does need to be flexibility.

And current abstracts are flexible, and that flexibility I think is responding to the different formats of the article, and the needs of the reader of that article. So, I don't know how to retain that flexibility under a system like this, although there may be a way.

MS. BURKHARDT: Well, my guess is that the brave folks who proposed the structured abstract in the beginning weren't expecting that your format would be the final one used, and that there is some flexibility in that. And the debate about whether or not the current one drives the kind of research that would be done, or has that kind of impact on the research is one that still needs be debated.

But again, it's sort of a straw man on the table about how you think through those harder issues to find something that works. And it may be that it's find some things that work, rather than one specific format.