DR. MC KEOWN: I would like to start with actually something that is almost that but in getting reviewers the hardest thing we have and I know just recently having a meeting with other editors of AERA and it is not only our journal is getting reviewers.

I mean we think we know how to pick high-quality reviewers. We have an editorial team of four editors. So, we know the field pretty widely. We know who is doing the work and the papers that are related to us or have bunches of sources of getting them.

It is actually getting people to agree to review that is incredibly hard and I think one of the things that we could do in boosting quality of journals is to somehow boost the review process.

We have tried that in AERA by actually honoring the top reviewers for each journal but if that could be a more important part or a more visible part or I don't know of people's careers because we have such a difficult time that we end up sometimes not getting the right people to review papers so that they don't know for instance the background of the papers so the kind of issues that Barbara mentioned about catching somebody who had already published a very similar article might go unnoticed by a reviewer and it just wastes an enormous amount of time as you have to ask again and again to get reviewers for journals.

So, I just want to make that plea as we start.

DR. FLODEN: Is that a problem for other people, getting reviewers?

DR. LIBEN: Let me just add to that because it has certainly been an issue for the journals in psychology. One of the things that the Child Development Institute as part of my editorship, I was concerned about that having been an editor for the Journal of Experimental Child Psychology before and I knew that problem, one of the things that I instituted was actually an attempt both to increase the reviewer pool in terms of the workload but also to get people who had not already been participating perhaps under represented in the sense of different methodologies, different populations, different research areas and so on is we actually sent out a request on e-mail to all of the members of the Society for Research in Child Development, so 5000 people roughly inviting them to go visit a reviewer site, and what I did was to create key topics or the topic list in which people could check off what their areas of expertise were and purposely inviting junior people to review, too, and there is also a section that allows people to say what kinds of experience they have had that they have published so that you don't, as an editor you don't end up selecting three green reviewers that you have no idea what their track record is but if you have two very well- established people and another person who maybe is very junior and has not been a part of the process before and does a good job then they can enter the pool, and I guess I just would urge that.

The second thing is the guilt induction process where I try to use as much as possible because I have found that some of the most senior people who are publishing and submitting fairly continually then refuse because they are so busy that they can't possibly do it.

So, I usually try to send a little personal e-mail to somebody when I know they have something under review with some wisecrack to remind them and I say out loud that this is meant to be a guilt inducer and I hope you are going to respond positively and it helps.

DR. FIREBAUGH: I think one of your, at least I have found one of my most time-consuming jobs as an editor has been that I am continually upgrading and adding to my reviewer pool and I can't tell you what the number is right now. I am actually a past editor. So, this is a few years ago, but I inherited from my predecessor a list of about 2000 reviewers broken down by categories and this is all computerized and the way we do it in ASR we try to be very careful about it is not just a matter of getting people to review but obviously getting the right people to review and we had certain sort of rules of thumb and principles that we followed.

One was that we did not knowingly send a manuscript to someone who was a mentor or a colleague or someone who has published with this particular person, and unfortunately in sociology there are certain areas that are fairly small and sort of have an old boy, old girl network and if you are going to do something in certain types of social psychology for example, practically everybody in the field either has an Iowa pedigree or a Stanford pedigree. So it is very difficult to come up with those reviewers, but when we did, we would send it to three reviewers, sometimes four and the biggest problem I have is not necessarily people who refused. In fact sometimes I wished they would refuse. My bigger problem was people who said that they were going to review and then they wouldn't do it, and you send them reminder after reminder after reminders. Those of you who are asked to review my perspective as an editor is I don't mind too much if you send it back to me immediately and I can get another reviewer.

What really bothered me was we were trying to cut our turnaround time from the time that an author submits a paper to the time that they get their usually with ASR rejection letter down to about 3 months and when you get too good reviews in and you keep waiting and waiting and I would so the same thing that Lynn would do and often send a cute little e-mail and "Oh, I am going to get to it this weekend." Well, that weekend never comes. So, one of the biggest problems we would have is getting reviewers to live up to their word, but if I could say one more thing in this database which we had of reviewers we would enter anytime we sent a manuscript to a reviewer we would enter when we sent it. Then we would enter when they actually responded, when they actually reviewed. So, we would have some idea of who our slow reviewers were and who our fast reviewers were. We also wanted to know, and I think this is very important, we wanted to know who our tough reviewers were and who are easy reviewers were. Basically the choices reviewers have are reject, accept and revise and resubmit and then within our form that we send out within each one of those we had two categories of each, but we would have like a box score for each reviewer and the box score for a real tough reviewer would be twenty-two, one, zero. The first number is the number of times they recommended reject. The second number is the number of times they recommended revise and resubmit and the third number was the number of times they recommended accept, and a really easy reviewer might be zero, ten, ten, something like that okay? Because again in selecting reviewers we didn't want some people to get really easy reviewers and get kind of a bad paper slipped in and other people just because of the luck of the draw to get very nasty reviews because of notoriously nasty reviewers.

DR. VANDEN BOS: In terms of the question of reviewers I don't think it is necessarily a problem of getting reviewers. I think it is a question of getting the right reviewer at the right moment in time; you know, whether or not they are available at this moment is a key thing.

We felt it was essential to provide editors with tools so that they could find the absolutely most right person and one of the tools that we are in the final stages of building and making available to our editors is a large database, actually it will be a database of over 750,000 empirical articles which any of our editors will be able to insert key words of what the research in the paper that they are looking at is about and then it will do research and it will tell them about all of the articles that are on that but different than the psych info database itself what is highlighted is the authors' names, and then you know you don't just pick an author because they did one article. You can click on the author's name and what this tool will now do is it will give you all of that author's publications so that you can see did this person do two articles on this topic or has this person done 43 articles on this topic and then you can sit there asking yourself why I don't know about this person if they did, but that is another question.

Across the APA journals over the course of the last 18 months 15,000 people have written reviews for the 42 APA journals so that I think we in general are able to get fairly wide diversity, etc.

In the last 18 months 60 percent of the people who identify themselves primarily as a researcher have reviewed for one of our journals and we think that this new tool is going to help in the task of finding the even more appropriate reviewer because we are going not on the basis of who does the editor know, you know, who is mentioned in the reference list, who has done research like this; you know, they will now be able to use a research tool that will take them and help them to identify the 20, the 30, the 40 people who are the most involved in that particular research question.

DR. EMIHOVICH: I want to go back to the question of how do you get to be an editor because people always wonder how do people get selected for these positions or how do they wind up in these positions. The Anthropology and Education Quarterly is managed by the Council on Anthropology and Education which is a subunit within the American Anthropological Association similar to the way APA has divisions and AAA has units or subunits.

The editor, you put in when there is a call for the new editor to be changed, there is a call put out and people send in their vita. They send in a letter saying why they are interested in becoming the editor and the appointment is made through the board of directors for the council which consists of approximately 25 people and that would be the president, the treasurer, all the officers, plus each committee, we have subcommittees that deal with different topical areas each committee has a chair and they vote on whom the next editor is going to be and that is how the appointment is made.

I want to go back to the point about reviewers. One of the things I very often did when I was sending out papers for review and I would do that to send papers to senior scholars but I would also ask them if they were mentoring some graduate students to have their graduate students do a separate review which I would then have indicated as a separate review because one of the things that I was always thinking about is how do you build capacity not just from the pool of people that are known for their work but how do you begin to bring in new scholars and show them what is the process by which this kind of work gets down because to many graduate students and even to brand new colleagues this is a very mysterious process. People don't know what it is and by showing them what is sort of the behind the scenes look and working, matching them against somebody who is very well known in the field and who is very familiar with the process you are in a sense teaching them how to become a good reviewer by matching them or asking them to mentor people that you already know who your good reviewers are. So, you ask them to mentor new people so you are building the pool from the ground up and that was a very successful approach and it also worked very well not just for new scholars but it was very successful particularly with scholars of color, many of whom did not often get this kind of mentoring in many places. So, it was a way for them to learn the ropes so that they could become more familiar with the process and it also increased their ability to submit better papers themselves.

DR. FLODEN: Bridget, do you want to say something from the point of view of your journal and how things in biology are similar or different?

DR. COUGHLIN: Sure, let me first give you a quick, very quick rundown of the Journal of PNAS so yo know how my comments are tainted or to be normalized.

PNAS gets a little over 8000 submissions a year. We publish 10 percent of that. Of the 8000, 60 percent ar rejected without review by the editors and when they do go on to review anywhere from two to three referees will look at a paper.

Our RSI impact factor is 10, just over 10 to give you some indication of where we lie within Barbara's numbers that she presented this morning.

Beyond our impact factor though we look at other indicators of how not impact by citation but impact by application which is something that was hit on this morning. We look very heavily at web traffic, how often a paper is downloaded, not necessarily cited but read,how often it is read internationally because by IP addresses you can tell if it is coming the United States or from abroad and then sometimes you retroactively look at if a paper isn't ever read or isn't ever cited who were the referees and where did the referee process go wrong and giving them grades basically at the end.

So, to elaborate on what you said basically are they easy or hard and then if they are easy did it actually influence the quality of what was published in the journal long term, and if they were hard maybe they were too hard and we missed something. That is a harder indicator to track.

After you secure the referees let us just assume we have the editorial board lined up and we have the reviews how do you get good reviews and that is something that we deal with constantly. How do you ask the right questions to get meaningful reviews with which to make a meaningful editorial decision on?

DR. FLODEN: You mentioned the large number or large percentage that are rejected without review. Can you say something about what the criteria are for deciding not to send a paper out for review?

DR. COUGHLIN: The review questions we ask the referees are is this paper in the top 10 percent just sort of an arbitrary line that we have established for the journal. If it is not close to that, if it doesn't have a snowball's chance we kind of don't waste everyone's time.

So, it is really a gut feeling by the editorial board which is made up of Academy members. There are 108 Academy members with the National Academy of Sciences that are on our board in a variety of disciplines, two anthropologists, but a whole lot of biologists.

We do try even when it is rejected without full review to provide the author some indicator as to why. Sometimes it is a little too formulaic and there are lots of appeals, whether it is an insufficient advance, too preliminary, unsubstantiated, you know all the buzz words. So, we do try to give some feedback even though it is not a formal review.

DR. FLODEN; Do other people also reject large proportions without review? If so, what is the basis for doing that?

DR. FIREBAUGH: It was difficult at ASR because we are a journal of the American Sociological Association. I remember my first luncheon as a new editor of ASR with my board. We have a once annual meeting at the American Sociological Association, a luncheon, and we discuss business and I asked them what am I doing wrong, and the first thing that somebody said was you are sending me too many bad papers and I said, "Well, do you think that I have the right as ASR editor to not review, not have some papers externally reviewed because there is some dispute about that?" and it was interesting among the board and we had about 45 board members. Not all of them were there but of those who were there probably 80 percent thought that yes, as the editor I do have the right if something really is obviously a bad paper I have the right not to review it or not to send it out for external review.

Now, the minority though, 20 percent said that I do not have that right, felt very strongly that because it is an ASA journal some of them even made the distinction if this person is a member of the American Sociological Association and one thing I really hated about the way ASA decided to do it, when you submit a paper to ASR I think there is a $15 submission fee. Well, that is peanuts in terms of the cost of the operation but it makes people think that therefore they are paying for my time and so some of them would send in their $15. Obviously we send their $15 back if we didn't review it but that is kind of a long answer to a short question I realize but I very rarely did not send out papers for external review even papers that I knew did not have a snowball's chance and in a few cases where I did that it sort of backfired on me.

DR. LIBEN: I really feel very strongly that the review process is an educational process as well as a gatekeeping process and so in general I approach the submissions for child development as kind of everyone deserves a review, but I was just looking at the numbers. Probably about 5 percent I have been rejecting as they cross the desk because what happens is I look at them and again, a snowball's chance. Thee is absolutely no way, but I do provide some kind of review. It is only maybe a paragraph's worth of substantive comments, typically talking about match with child development, not usually saying that we only take good stuff, but very often the reason for the rejection will be that maybe it is an application that is totally not discussing anything theoretical, conceptual, empirical. It is just saying, "Okay, now, I have done this with this group of kids," and so it really is not a good match for the journal and I try to say which I really believe that it is in everyone's best interests that we get an answer very quickly so that way it goes out within a couple of weeks of when they have submitted it. They are not wasting, authors are not wasting a lot of time, but I do it very, very sparingly because I do see this as an educational process, too.

DR. EMIHOVICH: When I did edit at EQ I did turn back some papers without review and I did tell people why. Most of the time it was because they were just not suited for the focus of the journal, and I had two criteria that I looked at in particular. One was the issue of methodology because we are field based and there is a considerable debate in the field about what constitutes good methodology when you are doing field research and one of the things I look for is time spent in the field because in order to be able to make good not generalizations but draw good conclusions from what you are seeing or interpretations, one of the things that you have to have is depth of time because somebody that would send in a paper and say,"I spent 3 weeks in a school," sort of like this is blitzkrieg ethnography would get turned back.

I mean there is no point in reviewing it because it is not fitting the standards of good anthropological research.

The other criterion was really and this was something more in terms of and I would be interested to hear the other panelists talk about it because it is more of an editorial judgment, is around the issue of focusing around issues of culture and really looking at what the anthropological notion of culture is as opposed to what a lot of people, a lot of different fields because it got very trendy to talk about cultural issues in education, what really do we mean by that term and anthropology has a very particular way of thinking about it. It is not the only right way but it is an anthropological way and if this is an anthropologically oriented journal people need to see that that is distinctive from another journal that talks about it from a sociological perspective, talks about it from a humanities perspective or a linguistic perspective and so it was really talking more about the kinds of concepts of what constitutes a cultural analysis as opposed to sort of somebody saying, "I went in and studied the culture of the school," and of course they usually say,

"And I did this in 3 weeks."

So, those were some judgment calls. and sometimes you do take flak over it but that is why you are an editor because if you weren't then you would just be a flow through. You know you would just sort of be the person that just walks the manuscripts and sends them out and brings them back in. There would be no reason to do editorial judgment.

The second part of that though is there were papers I would get where I would do what would be called developmental editing. I wouldn't send them out for review but what I essentially would do is review them myself and then I would write back the participants and then I would give them a letter that was usually around four or five pages long, and I would say, "You know, I don't think if I sent the paper out right now that it would probably get a good review, but I do think there are some really good things that you have got going in this paper and there are some interesting points here. I would like to see you develop these further," and then I would list out or talk about all the different things that they could do that I thought might help in the review process.

Now, it still had to go through review, but it was very interesting because a lot of the papers that I did that with typically wound up getting accepted down the road after they had gone through another round of reviews but I knew that if I had sent them out just as the way I received them the likelihood that they would have been rejected right off the bat would have been much higher and again that was a developmental kind of process.

I did that very often again with new scholars in the field because again it is a teaching function, and I guess I see editing as very tied to teaching in the sense that you are not just doing the review for the quality of the field but you are also teaching people about what is the process and your real goal is how do we make the research better and more meaningful and how do we help people understand what is the criteria for good research. So, I saw that as an important piece of my work as an editor.

DR. VANDEN BOS: To this point I think we have been focusing on quality in relationship to peer review and no matter what media you ar working in whether it is print or electronic I think that most scholars agree that you have to have some kind of peer review process for quality control, for a stamp that this is good research and what we have been talking about so far is getting appropriate and high-quality reviewers to participate in that process, but I think that there are more things that you can look at, more things you can pay attention to and do to assure quality than appears in any particular outlet.

One of them has to do with the definition of what this journal publishes, the publisher, the association that publishes it, the sponsoring organization whoever they are can do a lot to define the quality of the work that is going to appear in their journal.

Do they define the journal as an archival journal which means first publication of these findings? Do they define the journal as an empirical only journal or do they publish other stuff, case studies, you know ideas about what to do and so forth?

To what extent do they value that the research is theory based? That is going to influence the quality of it. Do they have separate statistical and methodological reviewers who they review on every manuscript, people who may not know the specific content area per se but who are going to take a hard look at the methodology used, the nature of the data that is collected, the best way to analyze it and so forth so that how you go about defining what you are even going to review and giving yourself some definitions about stuff that the editor may reject out of hand and I would say that Lynn captured what APA does, too. There are very, very few APA editors who reject more than 5 percent. They do it very, very cautiously and most APA editors take the attitude that has been articulated here which is that journal editing has two purposes. One is education and training and development of future, you know the person's next research, the person's next article and one part is gatekeeping and that they are two co-equal functions in many, many people's eyes which is why the rejection rates are not that high, but organizationally what I am saying is there is a lot that you can do to define the quality of the material that will be published in a given journal and then over a fairly short period of time a journal gets a reputation in any area in terms of what it covers and it doesn't take very long before you can talk about in this particular area what are the A journals, what are the B journals and what are the C journals.

In terms of selecting the editor, the editor is incredibly important and I say that in part from watching the selection of well over 200 editors and seeing the impact that the editor has on the material that comes in, both the number of manuscripts that come into a journal and the quality of them too and the kind of focus that they have.

So, for APA, I mean our selection process is a nightmare. The selection for an editor with an APA journal starts 18 months before the person is going to become an editor. We form a search committee. We announce that. We beat the bushes. The typical journal for an APA journal even the very smallest ones start with at least 25 people being nominated. The largest one I have seen recently has been about 85.

All of those people get letters saying, "Would you be interested and willing?" The good news or the bad news depending on how you look at it is about 75 percent of those people say, "I couldn't possibly do that at this point in my life."

So, we end up with a much more reasonable number that you can now start to think about and look at in depth and in most searches it is somewhere between 8 and 15 people who are seriously looked at; their resume is obtained. We look at their publication list. We get comparative reviews, you somebody who knows four out of nine of them, compare and contrast them on many more dimensions than you would ever want to imagine.

The search committee does a ranking. Then the P&C board does the final ranking. The chair of the search committee makes a presentation to the P&C board. I have seen some of these presentations take 2 hours why they go through the top and usually they say, "You know, after the sixth ranked candidate I refuse to say any more."

They talk about the top candidates, the strengths and the weaknesses of them. The nine people on the P&C board then all have votes. They rank order them. We weight them, etc., and then through magic we come up with a list and I get to start calling from the top and go down and I am happy to say that 90 percent of the time the top ranked person says, "Yes," and I can't remember the last time I went past the No. 1 ranked person.

They are always well known in the field. Then you get to the reviewers which we have been talking about extensively and they are critical and one thing I would note, so what I am saying is that there is what the sponsoring organization does and how you define the journal that influences quality, then the editors, then the reviewers and one thing I did notice on here had to do with the question of bias and I would just say that one of the policies that APA has in part and now there is a variety of things; we socialize the editors and they meet every year and talk about a whole host of things but one of the things that APA determined really probably before World War II was one of the best ways to control for bias is to limit an editorial term.

You know, before World War II psychology journals had a lot of editors for life. They had a variety of biases and it looked to me like looking back the longer that they were in the more biased they became and one of the ways that they control that APA limits editorial terms to 6 years and then we go on to some other person with their own set of whether you call them emphases or biases. So, that is one of the ways we deal with bias.

Thank you.

DR. MC KEOWN: I wanted to weigh in on the question of rejection without review. At ARJ we reject without review probably about 20 percent and it is because we get a huge amount of articles that are just not even, it is so clear that the person has never even opened the AERJ Journal. I mean seriously. We get papers that are five pages long. We get papers that are just simply essays, someone's opinion. We get things that are clearly a course evaluation or some internal institutional report that people have just said, "Oh, you know I think this is educational research," and they submit it which is a big problem because it can be very time consuming if we had to send all those out for review, plus we would have no reviewers left. I mean they would just refuse to review for us if we were sending them out those kinds of things. We do still send a letter explaining why it is not appropriate for the journal, but I also think, and this kind of relates to what Gary said about getting the word out about what a journal wants to publish and looks at. I think that is very important and I think AERJ could probably do a better job with that because the long version of that only comes out once a year at most and that is AERA's decision but we should probably bug them more to do that, but how do you account for people who clearly have never read the journal and they are just clearly going down a list of journals and I think because we start with A, and because it has got a broad title like educational research they think, yes, this applies and they toss it in, and I am very interested in the education part of this and mentoring young scholars. I have been in several initiatives to do that but it can be also very frustrating when you get these papers that are so out of the field that again it takes time and I don't know how to approach that and get people to say, "Is your paper really anywhere in the ballpark of what this journal will look at?"

DR. FLODEN: One of the things when we had a panel on peer review in our research proposals, a topic that came up was the training of reviewers and helping people to pay attention to the right things, to write good reviews and this has been mentioned a little bit. I wonder if any of you do things specifically when you are working with reviewers, what sort of guidance do you provide to them; what sort of examples do you provide of what a good review, a helpful review would look like. Is that something that is done in any of your journals?

DR. VANDEN BOS: I think what Lynn just said in case you didn't hear it is what most people do which is you assume everybody knows. Obviously you have to do a variety of things and it is typically done in graduate school in terms of orienting people towards professional obligations and stuff like that and one of the things that you attempt to do in graduate school is to say that part of your obligation as a researcher is to participate in the peer review process of journals and the better programs I think typically find ways to make sure that a student gets exposed to and has an opportunity to do a real review. Catherine has mentioned some ways.

We do allow somebody who has been selected based on their own expertise to involve a graduate student. We also prefer that the graduate student write a totally separate review and it be identified. Some people co-author it, have the student do the first review and the major professor basically adds to it and rewrites it and so forth.

Most journal editors for APA journals I don't think that I know of do a lot to educate or train reviewers per se themselves. They assume that people know.

Some of the things that I do you know, I and several people on my staff both at the APA convention and at many of the regional conventions do sessions that are both on how to get your article published in a scholarly journal but that are also oriented towards telling the person how to participate in the review process, the kinds of things that they need to know, how to introduce themselves to an editor and say that they would like to review for that journal, how to seriously evaluate themselves and think about whether or not they in point of fact have the right expertise to review for a given journal and so forth, so that there are a variety of things that we do and we count on the universities to do some of them.

APA does some stuff and most editors are willing to involve some junior reviewers. I know as an associate editor of the American Psychologist one of the things that I do is I usually use with any one of the things that I am reviewing what I call a random reviewer and one of the reviewers and obviously I don't weight that one particularly high you know I send it to somebody who is a subscriber to the American Psychologist who does not necessarily know anything about this content area and the questions that I am asking that person are very different. Would you like to see this thing? As you read it does it make sense to you? Do you understand it and do you think that this would be helpful to some other people to read, and while that is not particularly developing a new set of reviewers one of the things that it is doing is it is starting to address the question of getting input from if you want practitioners or non-specialists about the communication style and effectiveness of communication of empirical research which I think is one of the things that we do have to be involved with.

DR. LIBEN: I would like to add something and that is two kinds of feedback that I have given to reviewers. One is I sometimes reject the reviews by which I mean I have gotten some that are just really inappropriate, hostile, nasty, obnoxious, arrogant, etc., and I actually have written back to the reviewer as nicely as possible because of course the reviewers are very precious, too and you don't want to insult them but I have said things like this really, remember that this really needs to be constructive for the author, I mean in the letter asking for reviews we always say that these are going to authors, please be constructive and save your ranting and raving. I don't say that exactly but the message is save your ranting and raving for the separate note to the editor, but occasionally people really don't do that and so I have asked people to revise their reviews. This happens very rarely.

One case for example, I just did a word document track change, suggested ways of just toning down. So the message was still there substantively but I couldn't have sent that to an author.

The other kind of feedback that I have given is actually the opposite direction. I have super infrequently written back to especially a junior colleague who has prepared maybe two reviews of this type which are the 10-page incredibly scholarly citations. I mean it goes on and on and yes, it would be very helpful for the author but I have this image of this poor junior faculty member never getting tenure because he or she is putting unbelievable amounts of time and more than is really necessary or appropriate.

So, what I have done in those cases is to write to say that this is a fantastic review. It is very scholarly. Your comments are really helpful but I am a little concerned about how long this is taking you and take a look at the other reviews that you are getting back because that is part of the educational process and while your comments are really helpful just think about if you are going to do this a lot you are really going to be in trouble yourself.

So, I think at both ends, I mean sometimes the review is almost too good and sometimes the review -- the one-paragraph ones I mean we don't actually get very many of those and typically don't use those people again if they continually write these two-sentence reviews. They are not terribly helpful and sometimes informally I have told people.

I don't think I have every written to someone and said that this is too short, but that is another kind of problem.

DR. FIREBAUGH; I think for the editor one of the tricky parts is when you get a review which is maybe not nasty or inappropriate but is not helpful and it is for a paper that you have decided to R&R, to revise and resubmit because I think ASR probably does it like most journals that the reviewers get all the correspondence, of course, blinded, that the author gets.

So, if you write a review for ASR and the paper is rejected then you will get a blinded copy of the rejection letter and so you are learning a little bit and you are also getting the other reviews as Lynn just mentioned.

So, you should be learning, plus, of course, we are sending them forms. So, I think the way it is structured maybe people are getting more instruction than we are giving ourselves credit for, but in the revise and resubmit letter it is very difficult but I think I got somewhat good at it, at figuring out a way in the revise and resubmit letter to the author to make it clear to the author that they could not pay much attention to reviewer B without reviewer B getting too upset.

DR. EMIHOVICH: I was just going to say that I am really fascinated by Gary's concept of the random reviewer which I have not heard before and I think that that is an interesting idea to try out.

I think it would be certainly more applicable perhaps to the AERA publications because the big issue that again I go back to what we have been hearing from Washington with the Institute of Education Sciences is that the work we do doesn't reach the field, that we are intended to try to address problems of education and practice and making sure that indeed it becomes research based practice.

So, it would be an interesting idea to try to see just what kind of first run effect we might get. I didn't see, for example, going back to AEQ, I don't see it as directly tied to the field of practice in the same way because it is addressing a somewhat different audience in that it is really rooted in an anthropology journal and it is really to talk about how anthropologists deal with issues of education which is very broad by the way.

We would publish papers that would be focused on issues of education but it might be a comparative analysis of something that a ministry of education was doing in Thailand for example. So, it is not really tied to influencing practice or it would not be tied to some kind of intervention. I mean it would just really be talking about sort of the comparative issues of policy in a ministry of education but I just think that is a really fascinating idea and I wondered if it would even work because one of the things that we talk about medicine, education is often compared to medicine and sort of like how come you do education research and you don't improve schools the way medicine does medical research and the improve health and then the question is okay well, whether doctors in the field are reading the medical journals and influencing their practice and just right before I came here there was a study published that suggests that they are not, that doctors don't read their medical journals and/or if they do read them they are not necessarily changing practice as a result of what they read which is a whole interesting issue for us to be considering when we talk about impact on the field.

DR. COUGHLIN: PNAS when I said that they use two to three referees, the third referee is usually always a junior or a phantom referee or some random referee. I say, phantom because we rank their comments differently than the two known reviewers who are really developed in the field.

Some biological journals are also taking referee's comments not only and distributing amongst the referees in a blinded fashion which like you said is a great way to self-educate but also going a step further and publishing the referee's comments and sometimes they are blinded and sometimes it is without referee anonymity but from my colleagues really ups the ante if you have a chance even if it is ghosted to be published as a referee's note and some journals think that this is shameless self-promotion because the referees are just highlighting how great this article is that the journal published but when presenting referees with that thought that this needs to be in such a cogent fashion that you wouldn't mind it being distributed broadly even if it is anonymous that you will be forever linked to this manuscript should it go to press.

DR. FLODEN: In the prior session we were looking at the impact of journals through things like frequency of citations. Is this something that as editors you think about and consider the citation impact of your journal and how your practices might affect that?

DR. LIBEN: I would like to say that it definitely is. I think at the publications committee meeting the board of the Society for Research on Child Development, the fact that I am curious to check these numbers because Barbara, at least the number that Blackwell showed for child development had it first in impact on ed psych which was interesting and the monographs which is another SRCD journal was shown as the first for the developmental journals and we had quite a discussion about how that impact is done and how the monographs is a small base rate because it only publishes four issues a year and so on.

We certainly do that but the other aspect that I wanted to address and I am not sure if this is the time but it sort of feeds into it is the question of impact on the public and especially from the perspective of educational research.

One of the things that I instituted when I became editor of Child Development which I think is working rather well is we are now insisting that after an article is accepted for publication, so it is not part of the review process but it is before it will go into print that authors have to provide a 300-to-500-word lay person's summary and the idea is that those then will become useful to the society, to PR groups within universities as well as to SRCD's publications and we are still working on how those are going to be used, posting them on web sites for example. It is also the case, I am sure no one will be surprised to hear that many people don't understand what a lay person's summary, 300 to 500 is. It is not so much the numbers of words. It is the lay person part so that by no means do all of these authors do a great job with them and so we are also working on how to edit them, what kinds of processes, what kinds of person power you need to create them into usable distribution, but I would say that it has actually been very, very successful.

Many of you may have seen in the New York Times there is a couple of articles, a big spread on the child care issue that comes from another perspective that I instituted, but part of the reason I think it got into the Times and CNN and Newsweek, I mean it got everywhere was because we had these summaries prepared. We had someone in the SRCD office prepared and actually spoke with a number of reporters ahead of time before the embargo so that things really were set up and I think apart from having the raw material to use for that kind of dissemination it also has the educational value for authors telling them you have got to put this into language that normal people can understand and we asked them to talk about implications. So, it could be clinical implications. It could be educational implications. It could be theoretical implications.

So, even for someone who is doing infant perception work in a laboratory that they don't necessarily think of as applied it might have application or at least we try to force them to think about that application and some of the best ones have actually been people who do kind of what might be viewed as this boring memory research. I can say that because that is what my dissertation was on, you know, this very boring laboratory memory research and somebody writes up a public summary that really helps when you are coming out of the shopping mall and you are trying to find your car and just those kinds of little vignettes which grab normal people, not like us, I think are really helpful.

So, anyway I think that there are ways to think about dissemination and impact that are useful.

DR. VANDEN BOS: I guess we are sort of moving into the second part. You know, we have talked an awful lot about the review process and that is fine for editors and we are, also, authors and so forth but we are not talking about the content. You know, the content counts. The quality of the research that was done, the quality of the questions that were asked, the quality of the methodology, the quality of state and federal funding and the review of that, we have got to have good quality research before the journals can do anything. Then the journals, you know, journals to magazines are going to array across a variety of areas from the hardest most rigorous research, perhaps theory driven to research that is at least experimental and is looking at comparing some approach to some other approach to something that is at least empirical. You have got some numbers about something you did and then there are high-quality reviews and then there are low-quality reviews and then there is description of programs and there are ideas and essays and whatnot, and whether they all get published in the same journal or in an array of journals and how we grade those journals all relates to what we think is quality work.

At some point we have to become concerned about the translation of research into practice. My own feeling is that it is not the obligation of a research journal and an editor of a research journal to have as their prime or even the fifth down priority the translation into practice although I am lobbying for having an implications section in articles where in point of fact it is asking a question of so what and how would somebody use this research.

The translation into practice, you know I think we achieve that in a variety of ways and the journal may not be the primary vehicle that does it. Certainly the Internet is causing a lot of people to look at a lot of stuff.

Some of it is good stuff, and some of it is really bad stuff. That is why I think the kind of media stuff that Lynn just got done describing is important for scientific organizations to do .

One of the things that we are doing, and I don't necessarily think it is going to change the world or change what any state department of education does or any school district but it is part of educating a public about the fact that there is knowledge and there are somebody's ideas, and that knowledge is something that is examined and tested and debated and it appears in various publications based on peer review and a variety of things like that, and I think that when professional societies, research societies, etc., engage in this kind of media information, you know, we are doing one very big service to the public. We are bringing high-quality research to the attention of the public and indirectly we are teaching them about how to evaluate information and what is scientifically based information versus somebody's popular idea and those kinds of programs by scientific societies I think also serve to at least slightly undercut the self-promoters in the world who are promoting their idea whether it is empirically based or not, whether they are doing it through media contacts of their own, whether they are putting up glitzy sites on the Internet that attract a lot of attention. We have to get quality research into hearts and minds and media campaigns are one of them, but we have to look at the other avenues, too.

We have to look at the avenues whether it is various programs to collect and do integrative reviews, whether it is developing books with an innovative way of coming at it and in the area of education we have a book series that is as far as I am concerned it is translating empirical educational research into practice.

We have about 20 books in the title and part of the way that we are doing it is that in this series we team an educational researcher with a classroom teacher who is interested in writing and who in fact grapples with colleagues in practice-oriented organizations in their school district and other places, has become in some way known to others and they become a co-author of a book and the teacher is always asking the "so what" question and the researcher is trying to say why this variable is important and that variable is important and the teacher often can overrule the other co-author by saying that that may be really important to teasing out this little thing that you have told me about, but it has no practical implication that I can figure and I have been arguing with you about this for an hour and you can't come up with a practical implication either.

So, I think we leave it out and move on to the next thing, but what I am saying is we do have to look at the avenues. How do we want to translate the information, and what are the vehicles that we want to do that with?

DR. EISENHART: Could I just ask you one question? Could you name that series, please?

I am sorry, I didn't mean to put you on the spot.

DR. VANDEN BOS: I think it has no more formal of a name other than the Division 15 Series. You know we were working with Division 15 which is the educational research.

DR. EISENHART: And then is it published as an American Psychological Association publication?

DR. VANDEN BOS: Yes.

DR. EISENHART: Or does it have something like Sage that then publishes?

DR. VANDEN BOS: No, it is one of our APA books. APA publishes about 85 books a year.

DR. EMILHOVICH: You know one of Gary's comments raised an interesting issue that we haven't talked about which is how do you draw boundaries if you need to between evaluation and research because when you start to get people who begin to move into the policy arena and we had a whole big issue around this in AEQ, for a long time we had people who did micro-analyses of classroom discourse and very good research, very focused, very fine grained. People would write 30 pages on what was essentially 5 minutes of classroom tape and videotape and do a fine-grained analysis with the idea of really talking about some of the linguistic issues or cultural, social linguistic issues around that in terms of how teachers conveyed messages.

At the same time we had a whole series of people who began to do work, and particularly I am thinking about the Chicago public school system and how anthropologists become involved with helping a large urban school system begin to think through school reform and what can anthropologists contribute to that discussion with their understanding of what are the cultural issues that come into play when you begin to do that kind of large-scale reform and there was a long debate in not just the journal but in the association itself about what is the difference between research and evaluation because you are using much of the same methodologies. You are posing very interesting questions or you are helping policy makers think more analytically about their practice but you are doing it in the context of trying to really begin to get your hands on a very complex problem that really bedevils many of the urban school systems around this country in terms of how to begin to address issues of reform, student achievement, community involvement. I mean there is just a whole host of questions that come with that and then where does this show up in the journals and what place? Are the journals, the lead journals the places where it does get published or do journal editors say, "Well, no, that is not research; that is really evaluation, and here are some evaluation journals you can go to, and there are very good ones to do that, but we are only going to be publishing sort of basic research"?

So, those are really questions that start to feed into the concept of impact on whatever public you are trying to reach, whether it is the immediate context of the classroom or you are really working at a system-wide level which then becomes an infinitely larger question of how to do that and where do you get those findings published and where do people go to look for information that could guide them as they think through those kinds of issues.