DR. ROTHSTEIN: Yes. Thank you very much for inviting me here today. I am going to be addressing the relationship between journal policies and practices and the development of high-quality systematic reviews.

I will be talking about primarily quantitative reviews or meta analyses. I don't think that anything that I am going to be saying would not be applicable to systematic reviews of qualitative research.

So, I want to make sure that both are understood to be covered. The validity of a systematic review depends in large part on its ability to retrieve an unbiased sample of the research that has been done on a particular topic.

Ideally we would want to have the retrieval of all of the research that has been done. However, we want to make sure at least that to the extent we can guarantee it the research that we retrieved is an unbiased sample of the research that has been done and I am going to talk about how journal policies and practices can contribute to our ability to retrieve an include an unbiased sample.

Because I only have 20 minutes there are going to be some issues that I am not going to address that are related. I would like to mention them just to acknowledge that they exist and so that you don't think that I have accidentally left them out.

One is the gray literature also known as the fugitive literature or the semi-published literature in other words those things that do not appear in peer-reviewed journals that might be decent research on a topic but they are not published in a format that really makes them accessible to be retrieved easily.

The second thing that I am not going to talk about is the existence of registries of trials or studies that have been proposed but have not yet been completed.

Third, I am not going to be talking about the idea that by changing some of the copyright ideas or the granting permissions to use copyrighted abstracts and other material might be relevant. I think that might be addressed in the session after this one though.

So, I just told you what I am not going to talk about. You probably want to know what I am going to talk about. So, the questions that I am going to address are two. How can journal policies and practices affect systematic reviewers and others' ability to locate and retrieve potentially relevant studies and once we have retrieved them, how we understand and assess the design, conduct and the result of those retrieved studies.

So, I would like to address the location and retrieval process first. You have to think about how to look, where to look and what to look for. One of the biggest problems is that social science and educational terminology is variable, inconsistent and changes frequently. This has been alluded to in some of the earlier sessions.

A single concept or a single class or interventions can be described in a variety of ways and just based on some notes I took in the last session the notion of labels changing and how that might contribute to the reinvention of wheels that already exist. The most recent example that came to mind was the notion of disadvantage becoming at risk, becoming under served, becoming poor, becoming underachieving.

So a variety of terms will exist to describe the same concept or the same phenomenon, often without reference to each other. That leads to inconsistent indexing and it really complicates information retrieval very greatly.

You need to know all of the terms that could be used to describe your phenomenon. Then you need to now how to search for them and you need to hope that they were all somehow recorded or key worded or indexed in a similar fashion and basically that almost never happens. So, you are at great risk of losing some of the studies that you need to know about.

Furthermore and this is a point that I cannot emphasize heavily enough social science and education databases do not index by methodological terms such as randomized experiments, also, known as trials, okay and there again is another example of a labeling problem or time series design.

What happens if we don't index by methodological terms? Well, if for example as in Campbell or in the What Works Clearinghouse we are interested in randomized experiments or certain other types of high-quality quasi-experimental designs we can't find them because they are not listed as such. Again, that makes information retrieval a lot more complicated and a lot more time and resource consuming than necessary.

Highly specific search strategies which have been developed by among other people Kay Dickersin in the health care field are much harder to develop in social science and education because of this methodological indexing problem and that leads to the need for multistage searching. At this point if you wanted to search something like Medline which I will talk a little bit about more later there is a very well-developed search stream that exists that facilitates information retrieval very greatly.

Instead I will show you what Campbell suggests now. This is Campbell's interim policy about databases. This is what we tell people developing an electronic search strategy and again I am only talking about electronic searching for right now is an iterative process in which the terms that are used are modified based on what has already been retrieved.

People by and large cannot do this by themselves although you must work with an information retrieval specialist or a librarian because it is simply impossible as a researcher to know all the ways that you should look. You have to search for multiple terms that describe both the condition and the intervention of interest and then you have to go back and try those terms out on studies that you know that exist to see if it retrieves all of them and if it doesn't you have to keep modifying it and again since databases containing studies of interest to Campbell or to education researchers do not index by study design we can't even recommend searching using methodological terms until you have gotten the substantive part of your strategy straight, okay?

Once a search strategy has been refined to the satisfaction of the reviewers people can then apply methodological search filters recommended by IRMA, that is the Information Retrieval Methods Group to the studies retrieved.

So, you can see just by the description that we are talking about a very long and probably unnecessarily complicated process but given the state of indexing and key wording in the major journals and databases in the field that is what is necessary. It makes it more time consuming than necessary and it makes it expensive.

So, what can we do to improve information retrieval? The single biggest suggestion I could make is to convince publishers and journal editors to introduce methodological or study design index terms and then to go back not only and prospectively index studies that are going to be published but to re-index existing records and key databases.

You might be scratching your head at this point and saying that is absolutely impossible; it is a gargantuan task. I can tell you that the people at the Cochrane(?) collaboration which is the international collaboration that produces systematic reviews in the area of health care has succeeded in doing this with Medline. They have worked with people from the National Library of Medicine. It took several years but they succeeded in introducing methodological search terms and they have had a retagging process that has been going on also for several years now so that virtually every article that appears in Medline is indexed not only by substantive key words but also by methodological ones and I think that within social science and education that the three major targets should be ERIC Psych Info and Sociofile(?) although clearly there are more candidates for that kind of work.

I don't think that it is impossible. I think that it requires some targeted persuasion but I think that the results to the cumulative knowledge base in the field will more than repay the efforts that should be put in.

Second, encourage authors, journal editors, publishers and database producers to take key wording more seriously. I know that before Gary VandenBos referred to being able to plug in key words and pull out articles and then authors to facilitate the reviewer assignment process.

I use Psych Info a lot. I enter in key words. I am not sure who actually assigns the key words to the articles. I know it is not the author because I have authored enough APA Journal articles and no one has ever asked me to key word my own articles. I don't know who does it at APA in particular but I can tell you that my research on key wording in general has told me that most key worders are underpaid free lancers who often don't have good knowledge of the area that they are doing the key wording in although it does vary from place to place.

So, not all journals have key words. The key words are not applied consistently or with any kind of standardization. There is no crosswalk of any sort produced across publishers or across databases that would allow you to know that term A in journal A or discipline A is equivalent to term B in journal B or discipline B.

There needs to be some kind of consistency across journals and there really has to be more attention paid to the hiring and training of key word specialists. It just cannot be left as a sort of a random process it seems to be now.

It is also complicated by what happens when databases are acquired by different database publishers. They merge. They become acquired and you could think that you are searching consistently within a database and find out that there are actually subdatabases within it that need to be searched differently and on the front end you would have no way of knowing that unless you worked with a very skilled and up-to-date information specialist because often those changes are not published anywhere.

Am I being discouraging enough yet?

PARTICIPANT: Realistic.

DR. ROTHSTEIN: Realistic. Thank you. I had the absolute horrible experience just last week to use the ABASCO(?) academic search premier and I don't remember who the author was. So, I will use my name as an example. Let us say I put in Rothstein, H. R., and then I get a bunch of articles back and then if I put an H. R. Rothstein I wouldn't get the same set, and now if things are that bad at such a simple level, well, enough said.

Then the third thing that I would like to mention but I know Judy is going to talk about this is to encourage journal editors and publishers to adopt a policy requiring structured abstracts. She will tell you more about that in a few minutes, but I wanted to get an early plug in because I think it is that important.

Step two, if you remember I said that there were two parts to this. One was actually retrieving the relevant articles. The second was to be able to understand and assess them and here is where the role of journal reporting standards come in.

I would like to show you a quote from a document called The Trend Statement that was recently circulated in draft form about meta analysis and systematic reviews and the role of reporting standards.

Meta analysis requires full reporting of methods and outcomes to enable assessment of comparability of different studies. Inadequate or non-transparent reporting may make it difficult to understand the variables that impact intervention outcomes and the central elements in intervention success or failure over multiple studies.

Currently there is little consistency in what is reported in articles even within a single journal during the term of a single editor. The situation worsens across editors and across different journals in the same discipline.

Across disciplines even if the topic is entirely the same reporting practices may be entirely different, again making retrieval and assessment of information incredibly difficult.

I suspect that many of the people in the room serve as peer reviewers, even the ones who are not editors. How many of you even pay attention to the abstracts, for example? We tend to cut right to the introduction. So, abstract reporting and Judy again will get into this often has nothing to do with what the article actually says. What the article actually says has a variable association with what was actually done and that is the part I am going to be talking about.

I don't know if any of you have ever attempted a systematic review or a meta analysis, but I am willing to place a bet with anybody in the room that nobody reads an article as carefully as someone doing a systematic review,not the author him or herself, not their diligent graduate students, not the editor, not the peer reviewers. Until you are trying to pull information back out of the article you are not going to examine it as closely. I don't know why but it seems to be the case and I use APA journals for the most part since I am an industrial psychologist and I can tell you that there is despite the very wonderful editorship of most of the APA journals, despite the existence of the APA standards which I think really make a huge difference in quality, there is still a tremendous amount of A, inconsistency, and B, either non-reporting or mistakes, tables that tell you one number of subjects and then a result narrative section that tells you a different number.

I think that somebody mentioned before it might be just bad editing, but I think it is more than that. You just can't read that closely when you are processing this huge quantity of articles and I think people need more guidance.

I would like to tell you about some initiatives in other fields to improve study report quality. Harris Cooper mentioned some of these. The first is the Consort statement is consolidated standards of reporting trials. It was developed in health care and it is how to report trials, randomized controlled trials in the field of health care in medicine.

I mentioned TREND. It stands for the Transparent Reporting of Evaluations with Non-randomized Designs and then there is also another one from health care called STRDA(?) Standards for Reporting of Diagnostic Accuracy.

Since Consort has been around the longest and the other ones are modeled on Consort I would like to tell you a little bit more about Consort.

Consort provides a 22-item checklist for the transparent reporting of randomized clinical trials. It covers specific aspects of the background, the methods, the results and the discussion sections. It, also, has a flow diagram. There is a typo in there. I am sorry. That shows the progress of all participants in the trial from the time they are randomized until the end of their involvement. So, you really do know how many subjects are there at any part of the trial.

It has been adopted since its start in 1996 as a framework for the reporting of randomized clinical trials, also, known as randomized experiments by more than 150 mostly medical journals including and I just put up some names you might recognize, the Journal of the American Medical Association, the Lancet, the British Medical Journal and the Annals of Internal Medicine and essentially all of the journals that use it publish these standards and they are just there in the journal and they have it consistently across all of these journals which is wonderful.

Okay, I hope you can see this. This is an example of the type of information that they would require to be reported in the methods section, eligibility criteria, the details of the intervention, what the outcomes were, how the sample size was determined, how randomization was generated; was there blinding; what statistical methods were used? It is very thorough and I think very useful.

Trend does a similar thing. It is for non-randomized trials. It is consistent with Consort but more focused on behavioral interventions. It is not relevant to all non-randomized studies but to evaluations of interventions with certain kinds of quasi-experimental designs. These are the lists of the people who have supported at least the proposed trend statement and you can see that it does cover some of the journals that were put up originally by Barbara although not that many.

I know there are page limit problems about how we can have all of this stuff reported.Some of the solutions are to either have journal web sites or central repositories for data which I think Gary VandenBos mentioned APA was taking some kind of initiative in.

There are also reporting standards for systematic reviews. I didn't want you to think that we systematic reviewers were exempting ourselves and only demanding this of primary studies, and there are ones for both randomized trial and for observational studies. I think that these are definitely applicable on the areas for which they were developed and I wanted to point out that they were developed in part after looking at systematic reviews in ERIC and in Psych Abstracts. So, they should be relevant.

Okay, key factors for success in doing this, consensus development of standards by an internationally respected group of editors and other experts, a very participative process where people are getting feedback to see if it fits or not, getting sign on from professional associations, funders and other key stakeholders and then broad dissemination of the standards, publication in key journals, supportive editorial statements, freedom from copyright restrictions for the dissemination of the standards. Let me restrict it to that so you don't misinterpret it and making sure that it is published in languages other than English.

This is what worked in health care to publicize Consort and there is empirical research showing that Consort has made a difference in the way articles are reported in journals that have adopted Consort compared to those in which it has not been adopted.

Okay, so just to sum up journals can contribute to systematic reviews by improving key wording, by working with databases to improve indexing and by adopting and promoting reporting standards both for primary studies and for reviews.

Thank you.