DR. DICKERSIN: Thank you very much. All right, let us go to the questions. I just wanted to take the Chair's prerogative and add one thing about how we got the changes at Medline for the new indexing terms and that was by doing some research showing that you couldn't find what you were looking for and that things weren't indexed properly. So, it is putting another agenda item on there to do some research but that is how the change came about.

Okay, questions?

Joe?

DR. TOBIN: Hi, I am Joe Tobin. I would like to pick up on Judy's last point about this goal or dream of having an international rather than just a country-by-country set of standards.

It strikes me that research in the US doesn't take advantage of research in England much less in non-anglophone(?) countries for various reasons including terminology. I am in early childhood education and if you don't know to look up reception rather than use the term pre-school you won't find anything.

In the kinds of databases which are being compiled in England how is that being dealt with? The examples you gave us look like they came mostly from publications in UK journals but how do we get over this provincialism?

DR. SEBBA: I couldn't agree more with you. Certainly in the epi system it uses international literature mainly written in the English language but not exclusively. Some of the groups like the modern languages group have members of the group who are able to read articles in all sorts of languages. So, they have included a much wider selection.

It seems to me that certainly with the epi key wording system it is deliberately trying to overcome these international differences and include the different versions. When we go to the research summaries on the web we have had that problem that it has gone beyond just looking at English publications and has needed to expand its key wording in order to do that.

I think we found that to be a huge problem, first of all the problem with current databases being restrictive and B, with current databases not having consistency, and I can't see that a country like England, we can't get away with it. You probably can, but we definitely can't because we simply cannot restrict ourselves to English research and expect to get away with it. So, we have to look to some sort of international system, but it does seem to me that is not, it wouldn't be that difficult to overcome that problem because even if the education systems use alternative terminology ou could have that, you know, both versions or three versions of that terminology all included in any kind of key wording system. So, I wouldn't have thought it was that difficult to overcome.

DR. TOBIN: Maybe Hannah from the US side, do you have any thoughts on that?

DR. ROTHSTEIN: I think just very practically if we could get editors within similar disciplines together from different nations and get somebody to fund conferences you could put together something and then the first step is knowing what all the different variants are.

DR. DICKERSIN: Gary, did you want to respond to that?

DR. VANDEN BOS: More a comment about the structured abstracts. Every good advance has its own curse and we are looking at what medicine has put forward and saying how we should do that, but we also need to look at the fact that they have been doing research on their structured abstracts and there are two major problems with it. One is from a research side and one is from a practice side.

Research on structured abstracts has shown that one-half of all abstracts do not contain the same information that the journal article does. So, if you are just harvesting those abstracts and then doing meta analyses based on it you now do have truly garbage because you haven't captured it. It may be a good finding tool, but one of the standards in terms of integrated reviews has got to be that you go to the actual article and verify the data. You cannot accept the data in a structured abstract when of it is wrong.

DR. ROTHSTEIN: You never use the abstract alone to extract the data. You are using it as a screening device.

DR. DICKERSIN: There are two separate types of abstracts. You are talking about the one that goes with a full-length paper not the kind that is in a conference proceedings?

DR. VANDEN BOS: Yes, and the second thing is that to the extent that we are trying to make the educational research accessible to practitioners, whether that means a schoolteacher or a school administrator or whatnot one of the things that the medical profession has discovered and they are horrified about is that the practitioners are reading the structured abstract and then trying to base their practice based on that missing very, very important qualifications and limitations stated in the research articles and we can anticipate the same thing will be true.

So, we have to find ways to deal with that both on the application of the research and doing research on research.

DR. DICKERSIN: I can't help but comment on that. I don't think that that is interpreted as a problem with abstracts. I think that is a recognition of what has been discussed here that clinicians are just going to read short excerpts. They are not going to read the whole paper and so different things have emerged to try to deal with that problem. It is a real problem. It is not a problem with the abstract per se.

DR. SEBBA: Just one point on your very first point. The research I quoted from England by James Hartley is not medical research. It is in psychology of education almost exclusively and that led to the British Journal of Education Psychology adopting structured abstracts and that work does suggest that practitioners find those more accessible. I am not saying that overcomes the problem that you just mentioned about looking up all the qualifications but there is certainly plenty of research about structured abstracts not just in medicine.

The second point about leaving out crucial information in half the abstracts, I would just wonder what percentage we would get if we looked at traditional abstracts on that same issue.

DR. SCHNEIDER: I certainly feel that in terms of my own experience perhaps I didn't spend enough time looking at the abstracts especially since two of those were from our journal and that was Ranella's(?) actually. For anybody here from the University of Wisconsin, I recognized that one right away. I think the other one was actually written by one of my students, but I am not sure. Anyway the more important issue here is that we used abstracts for other reasons as well as the search process.

For example, one thing that we use abstracts for is when we send them out to people and say, "Would you be interested in reviewing this article?" So, it was a way to kind of flag to reviewers information about an article.

The second one was that we used the abstracts also to identify reviewers looking for key concepts and ideas but then we went to the databases to search for reviewers.

In light of these three purposes which are two additional ones to the one that you mentioned it seems to me that the whole issue of the structured abstract and what we are doing with the abstract is really pretty vitally important.

So, I think I am making a statement more than asking a question, but I don't know to the extent in how you have been using abstracts in other kinds of ways other than just for the meta analysis per se.

DR. SEBBA: I was simply raising the issue that in order to promote systematic review work neither the funders nor the researchers are very happy with having to do what they perceive to be a lot of extra work or fund a lot of extra work that they feel would not be necessary if the abstracts were more informative.

That is not to say that those same abstracts aren't going to be used for a whole variety of other purposes and I think what we have to look at is a bit like the issue that Gary raised. What I think we have to look at is the implications of using structured abstracts and whether in fact your reviewers would be just as happy with those.

Now, I believe one of the studies that Hartley did was on reviews but I don't recall all the papers. He has written a whole series.

DR. DICKERSIN: Okay, there is a bunch of questions from the panel, and I am going to go to the Committee first, but I just want to clarify something if it is all right because I think maybe we are all so used to doing systematic reviews that some of the questions we are hearing are because people don't know some of the problems that we encounter doing it and that is that indexers, studies have shown that indexers rely on what is in the title and abstract even though they are supposed to read the whole thing.

So, if the title and abstract don't have those key words like randomized field trial or whatever, then you couldn't tell it and the same with the people then going through 20,000 citations from their search.

Am I picking up on what you were saying?

DR. SEBBA: It is not so much methodological with epi because epi is looking at all methodologies but that is absolutely right and where it often falls down is on characteristics of the sample because the study might only be looking at ethnic minority peoples or might only be looking at what you would call elementary age pupils and the abstract doesn't tell them what the age group is or whether the study included ethnic minority peoples and so they then have to request the full article and if they are paying for the article and if they are waiting months for the article which in some cases they are then that either might mean that the article doesn't get included and therefore we are missing or might be missing a crucial study or that the delay and the additional work involved in then reading it and finding it was on secondary age pupils anyway, high school pupils anyway. It has just wasted a lot of time and my real concern is that it is actually putting people off undertaking this process and putting funders off funding it, and I think that we don't have time to discuss it today but I could give you lots of evidence that this is making an important contribution to improving quality in educational research and we can't do that if people get put off doing it.

DR. DICKERSIN: I think Margaret was next.

DR. EISENHART: Margaret Eisenhart. I think this is a point of clarification. Both of you mentioned in different ways that qualitative designs and quantitative designs are both covered by the process that you are using. So, Hannah, you mentioned that you were talking about quantitative, but these things applied to qualitative and you mentioned, Judy, several times that your epi process includes a variety of research designs.

My question is about is the idea here that there will be one standard of reporting that applies to everything or is it that within designs there will be different standards of reporting? How are you thinking about that issue?

DR. ROTHSTEIN: I think that the specifics of the reporting would have to be at least somewhat designed and study purpose driven. So, I don't think you could have one universal document but I think that the notion of transparency of what was done is a constant across all the study designs and I think that the important features that you would want to know about to be able to evaluate a study are probably pretty consistent from study design to study design even if the specific details aren't. So, I think you could work off one general model but it would have to be customized for specific types of designs.

DR. SEBBA: The criteria list that I put up, the ones that have been developed thus far which are not that dissimilar to the ERIC categories including things like sample or design or whatever, it seems to me those criteria apply to virtually all studies. It makes it quite clear, those systems make quite clear that if a paper is a theoretical discussion for example, not empirically based at all then obviously those categories won't all apply but nevertheless it seems to me that the things that we are missing at the moment out of the abstracts are things which apply to most empirical studies, whether they are in-depth qualitative ethnographic work or major large-scale quantitative studies.

So, I still think at the moment there is a set of criteria which need to be adhered to which would make everybody's lives easier and which probably exist and are completely accepted in biological sciences which is why the gap between submission and publication that was mentioned in that other session is that much shorter because people are writing to a standardized format without even necessarily articulating it.

DR. DICKERSIN: We have two more questions up here and that is about all we have time for, I am sorry.

DR. KLAHR: I have a couple of related questions. One of them is about whether you have thought about who uses the two classes of possible users of these kinds of meta analyses and reviews and I can think of two fairly distinct categories. One of them is researchers. Researchers use these reviews. I think their goal is to find a niche or a secret weapon or to stake out an unexplored area that nobody else has looked at yet or to see that there is a consistent weakness in an area and then to go ahead and do some more science whereas policy makers are looking for support for the authority of science behind some agenda that they are trying to pursue, certain advocacy goals and so on and I think they are quite different goals for a researcher to look at a really good meta analysis and then go on and do their research versus a policy maker and I wonder if you think reviews need to be tuned for those two different uses or if you even accept that dichotomy.

The second question is whether you can point to examples in the policy area, the second of those two categories where a major meta analysis has really advanced policy beyond where it would have been without that meta analysis?

DR. ROTHSTEIN: With regard to your first question in some quick and dirty research, a bad word for meta analysis person to use, that I did for an article I was writing on 25 years of meta analysis in industrial psychology we actually did find getting back to this issue that primary studies that were developed directly from analytic results had a higher citation rate and more impact than primary studies that were initiated the ordinary way, so that from the research point of view it is carrying out that purpose that you mentioned, that it is providing a niche and a better foundation for future primary studies. I don't think that it is the role of a meta analysis published in a scholarly journal any more than a primary study published in a scholarly journal to draw policy implications. I do think that there need to be mechanisms similar to those mentioned earlier today to disseminate the findings of meta analysis to policy makers, maybe even more intensively than the findings of primary studies but I know one of the issues in Cochrane and I think we are going to have to deal with that in Campbell, too, is whether within the systematic review itself there should be an implications section.

I know Cochrane has come out very strongly against that. They published their results and then the implications are not drawn by the reviewers because that is felt to be I guess at that point advocacy and they are not advocates but it does point to what the necessity for a mechanism that is in place to do exactly that.

Judy?

DR. SEBBA: I think you missed an important audience which is funders because the policy makers are not always the funders of research, and we have a lot of independent charitable foundations in England and the research councils who regard themselves as independent of government who are very interested now in systematic reviews and in the outcomes.

I would need to distinguish to answer your second question between systematic reviewing and meta analysis because most of the epi work doesn't involve meta analysis because as you will be aware the number of studies that are available to meta analyze in education are fairly limited in the topics which the epicenter are addressing. That is not to say that the system hasn't been developed to allow for meta analysis. It has and some of them, there are two I can think of that have already been published that include a meta analysis component but they don't all include meta analysis.

The others are just reviewing the studies. They are not actually if you like re-analyzing the data and in terms of the impact they have had there is one, I mean I think this relates to a lot of the discussion we have had today about how you know when impact is happening. In terms of impact on policy ministers do not always immediately change the policy when they see a piece of research. That doesn't mean it is not having an effect, and one of our first systematic reviews to be published showed that testing demotivates children and as you can imagine that wasn't really an easy one to present to the ministers in one of the first four that we were doing and expect to get more money for systematic review, but nevertheless we did it and I know only because of my position in government that that has affected discussions and although it hasn't reduced overall the amount of testing as of the next Monday morning it certainly led to an increase in the status of teacher assessment, particularly with younger children where the effects were shown to be more dramatic. So, that systematic review and it is one of several has certainly influenced the discussions amongst policy makers.

DR. ROTHSTEIN: If I could provide just a few examples of meta analyses that might have actually even changed the lives of people sitting here, if you have taken aspirin after a heart attack or been administered streptokinase after a heart attack then that was probably based on meta analytic findings, both done by people in the Cochrane collaboration.

If you are using the cognitive ability test across situations to predict employee performance that was probably based on a meta analysis done in the validity generalization area of industrial psychology and if you no longer believe that the DARE programs work or that Scared Straight is an effective strategy for reducing juvenile delinquents that was definitely based on meta analysis.

DR. DICKERSIN: Thank you.