DR. SCHNEIDER: Good morning, everyone. Thank you all very much for coming. Before I begin I would like to acknowledge the person who actually did all the hard work on this which is the analysis that I am going to show and could you please stand up so everybody can see you and if you have a question Ann is probably the best person to answer it.

Let me just say a little bit about my experience with journals so that you will understand where I am coming from on a lot of these issues.

For 3 years I was the editor of Education Policy and Evaluation Analysis. This is one of the AERA journals. I am going to show you the other journals as well. As the editor of EPEA we tried to do a number of things, changed a few of the ways that things were in the past some of which we were very successful and others in which we weren't so successful.

One of the things that we really wanted to do was to encourage both junior people to submit as well as senior people in the field.

We thought that there were many different things that were going on with respect to random assignment, change of administration, No Child Left Behind, and we really wanted the journal to reflect the changes that were going on in the field, and so consequently I called a lot of people and really begged them to submit manuscripts so that the journal would be in fact more timely.

The other thing that we did was we really wanted to move to three reviewers and no matter what happened I stayed with three reviewers. One was always methodological. So, if the work was qualitative or if the work was quantitative there was always someone who in fact could make a decision about the methodology, but just to give you a sense about what happened during the time the associate editor was James Blaine who is at Northwestern University and fortunately the two of us got along famously. I think it would have been a problem if we didn't. Jim had a special issue on leadership that will be coming out this winter, but we were able to contact 818 reviewers and you have to realize that this is 818 reviewers, some of whom you have addressed twice. We kept a database on each of those reviewers because remember we had three reviewers for every manuscript and how many new manuscripts did we process? We processed over 345 new manuscripts.

It is hard to hear. I will try to speak right in here. Sorry. That doesn't count the revised and the resubmits. So, you can just about imagine the scope of what we were trying to accomplish.

We moved with a relational database. We updated it and we kept it as current as we possibly could. We, also, if we ever got back a review that was just a paragraph we didn't use that unless we could encourage the person to write more. We always tried to make sure that we gave every person at least three pages of reviews you know, from three different people.

Anyway, those were the successes. Some of the not-so-successes part of it were that we were never able to get off an online manuscript submission process. This is something that I tried very hard to do and that didn't work at all and like all other things you know as an editor unfortunately we had one manuscript that kind of went through the terrible thing of somebody got back a review with the reviewer's name on it, but one out of 818, I felt well, that wasn't too bad although that was of no consolation to the person who got it back and so you know that to me was one of our mistakes and I am sure that the staff probably kept from me all the other mistakes which probably they were screaming and yelling on the phone but they were nice enough to keep that away from me.

Anyway what happened with EEPA and I think this is very important to stress because we tried so hard on this there were three pieces. Bob Lind had a piece in EEPA that was cited in the New Yorker. Martin Carnoy and Sussanna Loeb had a piece that was cited in the New York Times and then most recently there was a new piece that was cited in the Washington Post.

Now, I am not saying that this is just that work has to be relevant but if we are going to create usable knowledge that means something to people it is nice to know that these kinds of work that is being represented in the journals are in fact being picked up by the press and the people that did this work I think did a excellent work and they are excellent manuscripts.

That is a little background, a little longer. I see people getting a little tired, but I just wanted you to have a little sense about where we were on all of this.

So, now, accumulating knowledge through the journals. So, what have Ann and I been up to? Essentially what I am thinking about is that the journals represent the mechanism through which knowledge is reported and used to generate, one, new knowledge and it is of course the journal is what builds the profession and how does it build the profession? Well, one, journal citations are used to rank departments, and journal citations are also used in tenure decisions where you know this is not just something that we do but in fact this is something that makes a difference with respect to our field.

Okay, so, if we then say that we also recognize that there are not basically standards throughout the field on the journals, so how do you determine what constitutes the most important and influential research reported in journals? We know that quality is relative and that because of the fact that there aren't standards we have to rely on something. So, what do we tend to rely on? Mainly in the social sciences we rely on the Institute for Scientific Information, that is ISI, that claims that it provides a comprehensive database of the world's most important and influential research and that is a quote by Testa 2002 and then there is a very important caveat here that comprehensive does not necessarily mean inclusive.

So, it is important to say that while we don't have standards this is kind of the closest to where we are and this is actually what is the thing that works so to speak in place of a universal standard.

Now, Garfield is really the person who has set up the Social Citation Index and kind of how it works and the kinds of analyses that are used in it.

We really had wanted him to come to this meeting but unfortunately we weren't able to make that happen.

Now, he feels that it is very important to be selective and what he has stated is that there are about 150 journals that account for what is cited and one-quarter of what is published so that you really get the sense that knowledge is concentrated within a certain set of journals, that there is approximately 2000 journals that account for about 85 percent of published articles and 95 percent of cited articles. So, these are the core of what gets cited in the field.

So, now what are the education journals that are listed in the Social Citation Index? Now, the Social Citation Index categorizes 152 journals into one of its three educational categories, education and educational research, educational psychology and special education. So, these are the education journals that are in the Social Citation Index.

So, what do they look like? We tried to make this as big as we possibly could. So, these are the journals that are considered education by SSCI and what we have done for you is we have highlighted the AERA journals in yellow. I am not going to read this list, God forbid.

So, please if you all will just, I hope you all can see. So, these are the ones that the Social Citation Index considers to be education journals.

So, if somebody could just sort of raise your hand when you think you have kind of perused it enough?

Okay, thank you, Brian.

DR. KLAHR: Is this going to be on the quiz?

DR. SCHNEIDER: Yes, absolutely. There you are, Bob. Bob is one of the editors of RER.

Educational Resource Information Center, ERIC, lists 1016 education journals, and if you compare the Social Citation Index to ERIC you can get a sense of how many more journals are listed in ERIC.

Now, the following are the journals considered education by the Social Citation Index but not included in ERIC. One-third of these journals excluded by ERIC are international journals and nearly two-thirds are psychology and developmental journals.

So, now I am going to show you a list of education journals that are in the Social Citation Index but not included in ERIC. So, these are the ones that don't go in ERIC.

You will see some of the ones at the bottom in fact that are international journals. It does appear to be a little scattered and fragmented in terms of what gets listed and what doesn't get listed. I mean I don't really quite know how ERIC does this, but these are the ones that kind of fall out.

Then there are the next series that there are 153 journals in both the Social Citation Index and ERIC that are not considered education by the Social Citation Index. So, now these are journals that are both in ERIC and the Social Citation Index but the Social Citation Index doesn't consider them to be education journals. They are for the most part at least from my perspective disciplinary based.

So, they would be things like the American Historical Review, the American Sociological Review and Cognitive Psychology.

So, this is that list, and here you will see the American Journal -- because of sociology I am going to just point out like the American Journal of Sociology and then you will see some of the ones that it is interesting because the Society for Research on Adolescents and the Society for Child Development, I mean some of these journals which many people in schools of education submit to, but yet these journals are not considered education, at least from the perspective of the Social Citation Index.

So, if you just went to the education part you would miss these. So, that then asks the question what is the evaluation process for the Social Citation Index. Now, I am just giving you what they say they do.

First, they rely on publishing standards, citation analysis and expert review. So, what are the publishing standards that they use? First is the timeliness of publication. The journal must be able to adhere to its stated frequency of publication. The second is peer review and we have had a workshop on peer review but clearly peer review is indicative of a higher quality of research presented in a journal and the next is international editorial conventions, that is articles in the journals must provide retrievable sources, full bibliographic information and address information for authors and informative titles for both journals and articles.

Now, I think that this idea about retrievable sources, also fits a little bit into some of the discussions that we have been having on questions of data sharing and data access that have, also, come up in some of the other workshops.

Then the issue about citation analysis, so how do they do citation analysis? Citation analysis, first of all there is the overall citation rate. That is the number of times the journal has been cited.

Then they have something called the impact factor. That is the number of times a journal has been cited in the past year which is then divided by the total number of articles published by the journal in the past 2 years. Now, the reason they do that because journals vary by their page length so that some journals have a much smaller page length than others and then they have something called the immediacy impact which is the average number of times in the year of its publication an article is cited so that these are average article rates and that is different than the impact factor and then the overall citation rate.

So, then what we did is we calculated these for the 153 journals that SSI considers education. So, the average for that in terms of the citation rate is 471.

Now, if you look at the average citation count for the AERA journals we are doing pretty well. In fact, the Educational Researcher is not considered a journal. So, that is excluded from this analysis, but if you look at AERJ, EEPA, JEPS, RER and RRE we have a pretty good citation rate.

Now, if you look at the average citation count for the top five education journals that looks somewhat different, and you will look at these. These are child development, psychological review, the American Psychologist, Developmental psychology and the American Sociological Review.

So, these are the ones that are the top five. The AERA Journals do not make the top five and the psychologists on the panel are going to be very happy because they are the ones that really make out here like -- yes?

PARTICIPANT: Why is Educational Researcher not considered a journal?

DR. SCHNEIDER: I am telling you what is in the Social Citation Index. I can't tell you why that is the case. I can only tell you what gets reported. I don't know.

Then there is the impact factor, remember? So, this is the time that the journal is cited in one year, by the pages in the past 2 years. So, here again we see the impact factor and then when you look at again the AERA journals do better, you know, better than average but not as well as the top five education journals and then here again you see the average impact factor for psychological review, the monograph series in Child Development, the American Psychology, Cognitive Psychology in Child Development.

Now, the immediacy index is where we are below the average and the immediacy factor of course for the American Psychologist, Counseling Psychologist, Educational Psychologist, Psychological Review and the School Psychology Review do much better on getting their work out right away and getting it cited within their fields.

Now, there is something that they also use which is called expert review and this is an editorial board t hat is made up of professionals from various fields, reviews the journals and submits it for inclusion in their databases. The editorial board also asks subscribers to the journals and people working in the field for feedback about the quality and the usefulness of the journals.

Now, like SSI and Kay could probably talk to this the medical field has its own SSI and its own database. So, the SSI is a Social Citation Index and then there is one that is done for the medical field.

Now, the medical journals also look at circulation and acceptance rates. So, we thought that it would be worthwhile if we did some of those analyses for you as well.

So, now we are looking at the AERA journals the acceptance rates and the circulation rates. So, look at the circulation rates and I think that this is really very interesting that you know here is the Educational Researcher. It goes to 22,000 people. The AERJ Journal goes to 19,000. EEPA is small. JEPS is small. RER is very large and the Review of Research in Education again is small, and then these acceptance rates we got from the publications committee, Linda at AERA and so this will tell you basically what their rates are.

Now, I thought that it would be worthwhile to look at this in relationship to some of the other journals as well. So, we took the top five journals by citation count in the Social Citation Index and did the same analyses. So, you get Child Development has a circulation rate of 10,000.

Now, look at the American Psychologist which is really amazing how many people they are able to reach and then also think about the immediacy factors for all of the psychology journals and it explains to you that they really are circulating their information among a much wider, larger circle.

Then the American Sociological Review and we have Glenn Firebaugh who was one of the editors of ASR and somebody else from Child Development so that the idea here is that we would hear from these journals in the next panel and then this is by impact factor.

So, first we did it for you for circulation and acceptance. Now, this is impact and then here is immediacy.

So, basically what we wanted to do with this is to give you a sense at least from the Social Citation Index what is going on with the journals, what they use, how they cite things, what these citation counts look like and the reason that I am kind of pressing on this is because this is not a benign thing. This is something that is clearly used by people in departments when they make very important decisions on issues related to tenure and also they are considered when people do things like rankings of departments.

So, then the question is well, what is our workshop going to explore. One, it is going to look at how policies and practices promote or inhibit the development of high-quality journals, second, the promotion of coherent standards among the journals. This is particularly the case in Britain and what are they doing right now in terms of trying to put together some kinds of standards with respect to the journals and then some of the quality issues that are facing online publishing.

So, that is kind of my quick overview, and I am ready for questions.