DR. KLAHR: I think our format is the Committee members ask the first questions and then we open it up to the floor, and I guess since I have the microphone in my hand I will ask the first question.

I think we will have a good time here because people love numbers. There are lots of numbers up and people are going to try to figure out how to make sense of those. So, I will ask one number kind of question, and I have a whole bunch, but I won't ask them all.

One of them is that this immediacy impact measure in which you compute the number of citations in a year of publications is really an informal way of assessing how well connected the authors are who publish in that journal because in order to cite something in the same year it is published you must have known about it well in advance.

So, there is a lot of informal sociology going on here reflected in the journal review process and the journal evaluation process and I wonder if there are other aspects of that besides sort of well connectedness and visibility that you think some of these measures might reveal?

DR. SCHNEIDER: I think you are absolutely right about that, and it isn't just that people know that things are coming out. Another way to kind of help promote careers is basically to cite the work of junior colleagues, to, also cite the work of other colleagues.

In fact, I remember I had lunch one time with Don Campbell when he was at Northwestern, and I was a very junior person at the time and he said to me, "The first thing you do, make sure you cite yourself and your colleagues," and so, it seems to me that I am not saying that these are the criteria that the field should necessarily adopt. I am just telling you what is being used among people that do this kind of work, and I think it is important that we know about it and we understand it and we also understand what some of the pitfalls are and what this really represents.

On the other hand, it still is important to recognize that people that are doing work are at least trying to tell their colleagues that they are doing this work, sharing their papers and setting up their own social networks so in fact this work does get out, and we know in fact on the basis of the workshop that we had last time that when in fact we have controversial issues that these are the kinds of things that also will ratchet u the immediacy factor.

DR. KLAHR: A while ago I used to have to compute salary productivity measures every year, and I would look at the Social Citation Index but I computed a narcissism index for each of my faculty members which is the proportion of their citations that were self-citations.

Use the microphone and say your name every time you speak.

DR. TOBIN: I am Joe Tobin, a member of the board. Barbara, do you have any sense of the average number of citations per article in different disciplines? I read anthropology journals and some child development journals and my sense is in an average article in something like Anthropology and Education Quarterly you will have fewer citations than you will in a psych journal where people tend to have a separate literature review with a long series of citations and it seems to me that may explain one, that may be one explanation why the psychology journals have so many more citations, and if that were true what would be the implications of that? Do we tell anthropologists that you really need to load up your article with references to everything that has ever been written in the field including everything you have done before?

DR. SCHNEIDER: That is kind of the reason why we did the average citation count for the education journals which was the first line across. We didn't go through all the fields mainly because of the fact that we don't know all the journals in each of the fields. We could in fact I suppose if people were willing to give us that information we could certainly calculate the average number of citations in each of the representative disciplines that people work with but I think for our purposes and for the issue since this was about scientific research in education that is kind of what we did there.

I think that the issue about loading up, you know, your reference list in an article is something that we really want to think about. We are in the process and I also wear this other crazy hat. I am the Director of the Data Research and Development Center at University of Chicago where what we are trying to do is a social network analysis of people that are funded under a certain kind of federal program to be able to look through a relational database to understand who they rely on and how much in fact things move from one area to another.

In the briefing book there was an article that talked about mapping kinds of analyses that people had done to look at who cites you and how that moves around in a particular kind of social network series.

I don't know to the extent that this will be helpful or not helpful. Some people are more social than others, but on the other hand this is at least one objective way of looking at things.

DR. LAGEMANN: Ellen Lagemann. I was struck by how much more authoritative in a sense the disciplinary journals are than the education journals are than the education journals and if that is a fair implication from what you were saying then clearly those of us who are concerned about our junior colleagues tell them to publish in disciplinary journals not in educational journals, right, which kind of feeds a downward spiral that we really don't want.

I mean what do we do to in a sense break out of this because clearly this has to do with the status of education relative to the disciplines?

DR. SCHNEIDER: I would agree with you. I mean I think that is definitely the case and I know that people that are for example in sociology of education are told right away to try to publish in ASR, AJS and Social Forces before they move into trying to publish in Sociology of Education and in the AERA journals and that is the advice that people give people in that particular field.

What we can do in education and now I am just speaking personally I feel that we really need to really deal with these issues about standards and quality within the journal process and in the publications committee of AERA and I just don't think that this is necessarily the most appropriate place for me to start to say what I personally think how that might be changed.

DR. HENLEY: Ernest Henley. Barbara, I have two questions. The first one is how often in the social sciences does it happen that a publication is not appreciated for several, it could be 4 or 5 years which wouldn't show up in the citation index at all?

DR. SCHNEIDER: I think that is true for everybody. I would like to say it is true for myself but going back to Don Campbell and the Cook and Campbell piece, Don Campbell told me that he could wallpaper his office with rejection notices from journals and the number of times that the piece that you know kind of his most cutting edge work was basically rejected over and over again before in fact it ever got published and I also know through conversation that I had with Jim Coleman that after he published his work on busing he could not get anything accepted in a major refereed journal and that he just basically stopped trying.

DR. HENLEY: My second question was how many of the journals are available on the net before they appear in print.

DR. SCHNEIDER: I don't know. Maybe Gary Natriello may know.

DR. HENLEY: That would affect the immediacy factor.

DR. SCHNEIDER: Gary, maybe you could answer that. I am sorry, I don't know.

DR. NATRIELLO: I think it matters a lot. Most journals, and this is different for individual authors, a number of individual authors have personal web sites. Most commercial journals have at least an embargo period.

DR. KLAHR: Please use the mike. I know it is a little bit awkward but if you are starting to speak, that is the only mike we have out there, but we need to record this.

Thank you.

DR. DICKERSIN: Kay Dickersin. It looked like from your data although you didn't explicit say that that the vast majority of articles are only cited once or never and that is true for medicine, too.

So, what is the implication for for journals? I mean one implication of that is that these articles aren't making a contribution. That is not quite fair but there are some elements. If nobody ever cites them maybe they aren't making a contribution. So, what is the implication? Do we have too many journals, and also there is a very low acceptance rate. It looked like to me the highest was close to 30 percent, but the majority of what you put up there was less than 10 percent, and I don't know if there have been any studies as to whether those articles that are rejected ever reach publication in some journal but is there an implication there between those two sets of data you presented about not only focusing on journals but focusing on the research and the quality of the work? I know that they are talking about that in the health field. I don't know about in education.

DR. SCHNEIDER: We did have Don Jacheti who presented at our first workshop and he did give some research related to the peer review process but that information that he gave was really pretty much in the medical field.

We have contacted Don to try to see if in fact we could replicate his analyses and I was willing to do it with EEPA but I have to say that I got busy and I didn't do it. So, I am sorry I didn't do it but I know that it is possible to calculate, do the same kinds of calculations that he has done for medicine about the peer review process in education.

The issue about are the journals being ignored, I think that people that are editors of journals who are really thinking hard about these issues are trying to make it so that they are not ignored.

That is why I went so far as to say and I am not saying that just because a piece gets cited in the New York Times or in the New Yorker or the Washington Post that it is the most wonderful work in the world but I feel that I can stand behind those pieces and say, "Those were really excellent pieces. They were very important and they had a lot to do with what was going on in education at the time."

To the extent that most of the journals don't even get cited once, well, the mean level of publication at least when I did these analyses 15 years ago for a faculty member in the school of education was zero.

So, that meant that you didn't have a single publication in a refereed journal within a single year. So, I think that this issue, you know the question is how important are the journals as a mechanism for accumulating knowledge. If you believe as I do that the journals are important for accumulating knowledge, how then can we implement or think about standards in the journal field so that it in fact can enhance the quality of research in education, and I think that that is a question that I would pose back to the panel. It is certainly one that I am thinking about all the time.

DR. LAGEMANN: Barbara, have you thought at all about how influential journals are as opposed to books because I am deeply aware that for example in history books matter, journals don't? I mean you wouldn't even count a journal article toward tenure really.

So I wonder whether we don't need to worry about reviews for publishers, too, publishing houses.

DR. SCHNEIDER: I think we most definitely need to worry about reviews for publishing houses. I can only state from experience and my experience is that it is very hard to get tenure today without a book. So, you expect at least one book within the 7-year tenure period as well as articles. The review process of the book, I have done that for Yale and Harvard but I don't know that you want me to go on and you could probably say a lot more about that than I could, and I think that the issue about journals and books is that they are both being considered so that I think in education it also varies by specialization within education on who counts for what, but I think that that is another topic that we probably need to talk about.

DR. FALKENBERG: This is Karen Falkenberg. Barbara, I wanted to ask you a question based on the information that you started off your presentation with. You pointed out that there is not a one-to-one correspondence between the different databases about what is considered an education journal and I wondered if you had a sense of the criteria that are used to make that selection for each of those and why they may not be the same.

DR. SCHNEIDER: No, I don't and in fact that is why we did the analysis because I was just really curious. I mean if ERIC has 1000 and the Social Citation Index only had one hundred and some odd, who got left out and who was included? So, we just did that to kind of look at that.

I mean statistically i suppose I could do factor analysis or some other kinds of things to see how they group together and if there are any particular kinds of themes that are consistent about them but we didn't do that at this point. We really just did this descriptive work and I don't know, sorry.

DR. FALKENBERG: Do you think they will be able to tell you that?

DR. SCHNEIDER: No, at least when we talked to them we weren't able to get any kind of information on that. When we talked to people at the Social Citation Index on that we weren't able to get any information.

DR. KLAHR: I think it is interesting we have been talking about journals in really two distinct senses. One is in the sense of knowledge accumulation but the other one is in the sense of career advancement and I think it might be at some point interesting to look at the way these two are related. In terms of alternative formats that Ellen asked I know that in an area, I am a psychologist but a lot of my colleagues are interested in human-computer interaction and they go to kind of technical meetings in which the accepted paper in a refereed conference is the coin of the realm and if you look at their vitae of quite prominent successful people there are very few journal articles but they go to the meetings and they present their technology, their particular package they put together and to the extent that a lot of education is with a deliverable ultimately there might be quite another form of knowledge accumulation that we are missing completely with both journal articles and with books.

DR. WISE: Barbara, you presented a very rich set for the quantitative indicators of the productivity of journals and so on. Do you think any of the journals actually look at this kind of information as part of trying to improve their own publication standards or review their processes by which they make decisions about publications.

DR. SCHNEIDER: Okay, I will give my one comment about the publications committee at AERA. This to me seems to be one of the most important questions that needs to be looked at by a publications committee. I mean I think that is the kind of thing that a publications committee needs to be thinking about and considering and looking at those numbers and talking through.

Let me just give a for example? One of the things that I noticed much to my chagrin after it happened is that we published something in EEPA, a very close, almost exact was published in RRE and then I saw something in AERJ that was almost the same thing that we published in EEPA. So, I think with so few opportunities for people to be able to get their work through the review process that it seems kind of problematic if nearly the same article is being looked at in two journals, and I think that these kinds of discussions are really discussions that people need to have.

DR. KLAHR: State your name and try to speak into a microphone.

DR. DE HAAN: Bob DeHaan. I wanted to follow up on David's comment about the distinction between reviewed journal articles and presentations at meetings. In what we refer to as the hard sciences tenure decisions at least in our experience, tenure decisions and hiring decisions are generally made under circumstances where if an applicant has too many presentations listed rather than journal articles that is a negative. That is a very distinct negative and in fact one of the kind of general reputational issues is exactly that, that is that we see in education this tradition I guess of seeking to expand the number of presentations one gives without having backup in terms of a reviewed article in a reputable journal and I think that sort of cultural difference is one that really has to be explored.

DR. SCHNEIDER: I would agree with you but I also feel that that might be some of the kinds of things that we might want to talk about in the workshop tomorrow because of the problem and maybe the advice that we are giving to junior people about making presentations and the size of the AERA program and the numbers of times that people are presenting, all of these kinds of questions.

So, I would definitely agree with you.

DR. COOPER: I am Harris Cooper, and I want to make a suggestion about making sure we have a clear distinction between what it is that SSCI does with citation indices and the notion of quality.

I think what they do really has to do with influence and while influence may be related to quality there can be very important ways in which it is different. So, I am not sure we should look to SSCI to look for quality. I think somebody has already mentioned the notion that if you are in a small discipline you are going to have fewer citations, lower impact factors than if you are in a large discipline and obviously the American Psychologist showing up there with over 100,000 subscribers, a large percentage of those folks are actually clinical psychologists who have only a tangential relationship to the research enterprise.

Some of the ways in which it might be difficult to translate influence into quality, for example, we might ask the question what would be the impact on citation counts if the quality of every education journal were increased equally, in other words we had a broad-based intervention that raised the standards for every journal in education. The answer, my hopefully not too provocative answer would be that there would be no impact then at all.

Another important thing to recognize in the influence measures is that articles can get cited a lot because they are really bad.

(Laughter.)

DR. COOPER: So, somebody puts out a piece of junk that makes its way mistakenly through the peer review process and leads to an enormous amount of comment, turns it into a citation classic for all the wrong reasons and then other folks will argue on some other things about some of the more sensitive distinctions in here and not that this is a bad thing but obviously review articles, review of education research as a purpose is to replace other articles. They typically have anywhere from 40 to 80 references in them and they are meant to replace those articles which have fallen behind the research front,

They will get cited a lot for that purpose and methodology is the same way. So, there are lots of different ways in which looking straight at the influence factor and taking it as a direct proxy for quality might lead us down the wrong path when we think about those kinds of things.

DR. EMIHOVICH: I am Catherine Emihovich. Listening to the dialogue here it strikes me that education researchers are caught in a terrible bind.

On one hand with the formation of the new Institute for Education Sciences the emphasis is shifting to evidence-based practices which presumably are intended to influence what happens in K-12 schools so that there is a greater use of research to guide practice which I think personally is a good thing. However, at the same time Ellen has identified the fact that the prestige that education researcher get in terms of where they publish comes when they publish in more disciplinary oriented journals rather than education-based journals which in turn are less likely to be read by the very practitioners they want to reach in order to ensure that evidence-based practice makes its way into K-12.

I just wondered if you had any thoughts about how they get out of this bind?

DR. SCHNEIDER: No, I don't,not at this point, but I would like to say something to Harris because I think that he made a very important point that we really can't say that the Social Citation Index necessarily citation counts mean quality and when I started I thought I underscored that point.

On the other hand one of the reasons that I did use this is because of the fact that it is used and it is one of those kinds of catch -- as you would say, you know, you can't ignore it because of the fact that it has such a strong presence in our departments and in the process of not only knowledge accumulation but really more specifically the advancement of the field through the promotion of our young people.

DR. WILEY: I am Alex Wiley with the National Science Foundation and I am struck by the parallels between acceptance rates in peer-reviewed journals and award rates in funding programs that we run at the National Science Foundation and one thing that I worry about is the issue that we worry about all the time in funding programs which is the really quality research proposals that don't get funded but almost make the cut, don't quite make the cut and in some cases those are every bit as good as the ones that do get funded because there is an imperfection I think in the peer review process. There is an element of chance is all I mean.

So, from a sociology point of view the question is what happens to those proposals or from the journal point of view what happens to the almost acceptable articles that don't make it; do the people persist and get them eventually published like Don Campbell did with his landmark article or do they give up or not get tenure and the stuff never reaches the public eye?

In my own personal experience I once had the privilege 25 years ago of turning down Richard Florida for a grant and his work was at the center of a large article in the Washington Post over the weekend on regional economic development. He obviously found other ways to get funded but I still remember how angry he was at me when I called him up with the bad news.

I think that sort of thing goes on all the time. So, in terms of sharing knowledge I know there are lots of other avenues besides formal journals and I think all these are probably very important.

DR. ROTHSTEIN: Anna Rothstein. I am an industrial psychologist. So, I can only speak indirectly to the relationship between impact within journals and dissemination to the field, but in my field, Sara Rimes of the University of Iowa did a very interesting study where she compared several well-known, well-established research findings that were widely subscribed to by researchers and then she compared that with practice by HR practitioners in the field and in most of the cases the practice was exactly opposite.

So, I think that in many cases journal authors are really speaking to other journal authors. Your experience with getting things into the popular press is three out of how many, and we really have to think more carefully about not only talking to each other when we think about impact but talking about how to influence the thinking of the people out there in the trenches.

DR. KLAHR: We have time for one more question.

DR. JONES: Loretta Jones and I have a little bit different thought on research influence in K-12 education. K-12 education is certainly increasingly diverse in this country and it is true that the questions that one is asked as a researcher are influenced at least in part by the background and experiences that they bring to their field.

I think for instance of research on health and so on among women certainly being influenced by the number of women researchers in this field.

I was wondering if there has ever been any thought about a standard for journals that would relate to diversity not just in the international way but diversity among researchers involved in both the articles being brought and the peer reviewers and a variety of other things. You can see how this might go on and the extent to which this influence positively the impact of research on an increasingly diverse education field.

DR. SCHNEIDER: I can only tell you my experience through the publications committee at AERA and I know that AERA make a commitment. I can't say that it became a standard but a commitment to diversity both with respect to the editorial shifts within the journals and to try to increase the numbers of people that were submitting and getting work accepted in the journals to reflect a more diversified population.

DR. SEBBA: Judy Sebba and I am from England as you can probably tell from the accent, professor of education at Sussex University.

I just wanted to first of all say that we laughed at Harris' point about poor quality findings being cited extensively. That is a real issue which I hope we will take up further in this workshop because in the work we have done in England that has been from Silverbert onwards a real issue in terms of publicity given to poor quality work and citation figures I think can be quite distorting in that effect.

The other point which I think also follows on one of Harris' points and a point made by one or two other people is that impact is a very complex concept and often confused with dissemination. The citations in themselves don't tell us very much about the impact on policy, practice or even theoretical advancements and I think that even if it does tell us a lot about subsequent research funding or promotion of individuals we need to get somehow behind that and work out how we can sort out the relationship between quality and the journal publications because for example in England and I guess it is true here but I don't know the American press as well, the highest circulation of newspapers that those newspapers would carry research which was A, sexy and B, had been well promoted by the researchers. It had nothing to do with quality actually and that is a real problem for us.

So, I raised those issues and hope that we can address some of them in the workshop.

DR. SCHNEIDER: Since we are coming to the end I think I would like to end on a more positive note which is the fact that my understanding is that the panel is moving toward trying to understand the relationship of quality, what quality indicators might be, what mechanisms we can use to accumulate knowledge and that is where we are and this is one and hopefully the rest of the day we will be able to look at some others.

So, thank you all.

(Applause.)