DR. KLAHR: Yes, I would like to return to something that Gary said earlier. I think it was kind of a blockbuster that we should pay more attention to and that is that we are no longer a journal-based science but an article-based science.

I think that has, to the extent that it is true, and I think it is, tremendous implications. I mean I get a stack of journals like most of you do and I put them on the shelf and they turn out to be a very expensive form of wallpaper, you know with all the red ones on one shelf and all the green ones on another shelf, but what I do is I subscribe to these various abstracting services like Ingenta(?) that tells me every month what might be interesting to read and what might not be.

Then I will go and get a particular article. So, I really don't care about journals very much. I care about the set of articles that are of interest to me and I wonder what all the journal editors think about that because it means that my loyalty to journals is just about zilch and what I care about is a set of articles.

DR. VANDEN BOS: I think you should still love your journal. The journal based efforts are still very well placed and they are part of our quality control process. What I think is different is that the role of the journal is different. The role of the journal I think is much more the case of collecting the highest quality information for a particular audience. The utilization of the material, it is, the journal gets broken down into pieces and is accessed electronically on the article basis.

So, the journal is still very important. If we were to put all print journals out of business tomorrow we would have to create a new quality assurance content selection mechanism and would involve all the same people and we would do something else, but the journal is essential for that and essential for a core community who will always read it because so much of it is relevant to them.

DR. EMILHOVICH: I was just going to say that coming at an interesting junction in 2 weeks I am going to AAA and we are going to be considering the whole issue of digitizing and what does that mean. You asked me earlier does that mean we are going to keep the print-based journal and I am not entirely sure that is the case. I mean I assume we will but there is a whole huge expense about publishing in print that is eliminated by going to the digital version.

DR. EISENHART: First of all, I want to say that I am in awe of anyone who wants to be an editor or is an editor or has been an editor. I can't imagine the logistical details that you have to deal with, but I am struck by the earlier conversation that focused an awful lot on the logistics and the management issues around handling manuscripts and finding reviewers and encouraging reviewers and so forth. I am struck by that in comparison to what I didn't hear very much of which is the question of what are you proud of as editors of journals that has to do with this question of knowledge accumulation; what are you proud of in terms of the articles that have been published or the work that you have done as an editor or the way that you have organized the issues that you put out? What have you been doing that you feel good about with regard with regard to accumulating knowledge in the fields of your respective journals?

DR. FIREBAUGH: Neil Senelger(?) was the editor of ASR, one of my predecessors, and he had the idea of asking all the living ASR editors to pick out three or four articles that were published during their tenure that they thought were the best articles and I think I picked three or four articles that I think really did have a major impact on the reader and I guess also some of the articles I am thinking of, they were not necessarily the greatest articles but they were articles I worked very closely with the authors and in some cases in ASR papers might go through three or four revisions before they are finally published and I guess what makes me feel good is I felt like there was some value added.

DR. EMILHOVICH: I think there are two things I could say. One would be the editing process because it is not so much the accumulation of knowledge as it is the idea that knowledge can be found in many places, but sometimes it has to be worked on to bring it into being so that it opens up opportunities and challenges for people who might not otherwise have gotten their knowledge out and spending a lot of time with different people in different places you see that there is a lot of really good work going on but sometimes people don't always know how to articulate it well. So, helping them to get it articulated well so that they can move it to a place where people would say,"Yes, that is something I really learned from there."

The second one is I started a feature called Reflections from the Field and it is not so much contributing knowledge as the knowledge base as we define knowledge but it really was people's way of writing about their experiences in the field much more from the narrative standpoint but what I found and what people have told me since I left the journal is that is one of the first features they go and read is people's reflections from the field because people have attitudes and views about the kinds of dilemmas they face, whether they are going to be questioned on the methodology, working on a difficult question and how to assess it, how to think about it and it was opening up a space for that apart from the traditional review articles and these were reviewed, too, but in a different kind of way because they had a different purpose and as I said that feature has remained very popular in the journal since.

DR.COUGHLIN: It is interesting that all the previous answers have addressed sort of value added content or specialty articles which get to your point of an article or a journal-based driven society and I think obviously to be solvent journals are adding that value-added front matter of editors' picks classic articles type thing and so no one has mentioned raising editorial standards.

Something that I think PNAS has done fairly well over the last few years is to really promote international synthesis. So, we spoke earlier about synthesis of ideas and it is very easy to say especially in the biomedical literature that it is really not that important for developing countries to have this knowledge because they could never really apply it in their research environment because it is incredibly technical or requires special equipment. It is a very sort of passive and esoteric way to look at.

There are many,many ways and very inexpensive ways that we have found that journals can participate in distributing their content free to developing countries and it has really been a boon for us in sort of promoting international synthesis as well as furthering sort of the scientific endeavor and research in a global sense and it was much easier than we thought it was going to be and much less expensive, happily than we thought international synthesis promotion would be.

DR. MC KEOWN: I guess one of the things that I am most proud of during my editorship and it is a team of editors, Elise Foreman, Kevin Crowley and Randy O'Connor are associate editors and I think that having been able to publish a range of articles that show the different ways that can constitute contributing to knowledge and understanding and learning, and for example we have articles that are based on very large databases and we also have articles, we have one article that is based on interviews with three teachers. We have an article that is a math article that is actually all about discourse based on 15 minutes of a classroom transcript and we have other, you know, longitudinal articles. So, I think that that is kind of exciting to show a real range and how they all really do have something to contribute.

I guess the other thing that I am proud of is being able to provide the kind of review process that in a couple of cases like what Glenn said made me think of this of being able to, a couple of articles that I can think of that came in that clearly had something very good in them but they were hard to work with and because of the time that the reviewers put in they had some fabulous reviews and I was able to see what was of value and able to explain that to an author which I think is really important being able to lay out in detail what an author has to do to get an article in good shape, and I am proud of having done that and especially in a couple of specific cases I can think of.

DR. VANDEN BOS: The best editor is an editor who rewards himself and I really enjoy editing and I find it personally rewarding and by and large nobody knows what I did. So, if you can't reward yourself you are going to feel pretty miserable and alone as an editor because what are the things that you do? You know you quietly make decisions in terms of what you are going to publish in a well-developed research area where you are really looking for is there something new here and if there isn't something new it is another look at the same thing and you would have predicted the finding before they did the study and yes, they got the finding. You are going to reject it and send it on to a B level journal. Nobody knows that you did it except the author and they wish that they had snuck it by you and you had published it in the journal that they wanted to put in, and then you can decide whether empirically you want to take a look at the citation rates of the things that you rejected versus the things that you published and being empirically oriented I do that from time to time.

The other things that you don't see, you know that the public doesn't see is the article where it came in and several people have talked about it, and it had an interesting something in it but boy, was it rotten and you went through an incredible and often two and three and four revisions slowly nurturing the person along, getting him to talk about the literature in the right way and maybe even reframe the question that they are asking because their data actually answers a question they didn't realize that it answered and you take a lot of pride from that and then there are areas that just aren't being addressed.

You know there has been a bias in this particular journal where either the editor wouldn't accept something on a particular topic or nobody was submitting it because they thought the editor had a bias against it and you say, "This is important, this is relevant, let me keep my eye open." You find one. You nurture it and another question is opening something because you know there are people who are asking the question and can't find a researcher who is trying to answer the question. You know, when you find it that is one of the things that you work on and if you find doing that stuff quietly and alone rewarding then being an editor is great. I happen to find that rewarding. So, I enjoy editing.

DR. FLODEN: Ernest?

DR. HENLEY: It was more of a statement than a question. I don't see how one can do without the ability to replicate data. I hate to tell you how often people publish articles where they think they have found a new effect and other people look at the same data and come to a totally different conclusion. So, the ability to replicate so you don't get false information out is to my mind crucial.

DR. JUNKER: Brian Junker. I have a lot of questions but I will only ask two, and I will ask them in the same breath. Part of this session is about knowledge accumulation and I guess this is basically about knowledge accumulation through journals and one aspect of that I think is the shelf life of articles in journals, how long one might expect an article to be relevant or to be cited or to be important and there were some comments earlier. Catherine mentioned the idea that ideas come around again every 30 or 10 or whatever number of years and so it is not always easy to predict the shelf life of an article. There is maybe a short term and a long term shelf life, but I am wondering if those of you who care to might like to share your perception of what you think the shelf life of a typical accepted article in the journals that you are responsible for ought to be, whatever shelf life might actually mean and also I have noticed in my own interaction with journals that referees particularly and in some cases associate editors have a very different sense of what the shelf life and impact of an article ought to be, all the way from having too low a sense of what is sufficient for publication to having the sense that something should only be published if it is sort of a timeless contribution to the field and so, that is wrapped up in perceptions about shelf life also. So, I would be interested in any reactions you might have about your own perceptions of shelf life and what you have seen from your editorial boards and referees.

DR. MC KEOWN: Maybe I can start by saying a couple of things about shelf life. First of all I think high-quality articles should have shelf life, should be still relevant a decade from now, both for maybe the major topic in them and then there may be other things about the way the author discussed some underlying issues or something that should be important but there is also another kind of scary part to shelf life. I know that there are articles like I can think of Humian's Lessons Learned article which is cited in about every paper I have written for about the last decade and in millions of papers and the reason it is cited is because the phenomenon that was described which is this kind of retrieval of information interaction between teachers and students has not changed in the schools. So, it wasn't the fault of the research that it had a long shelf life for that reason but it is because of that phenomenon that was noticed hasn't changed as much as people have been working on saying that is not really what should be going on, and I can think of a couple of articles in my field that are very old, even older than that one that get cited a lot exactly because of that reason, a phenomenon that is not good has been going on for a long time and hasn't changed.

DR. VANDEN BOS: In terms of the recycling of ideas and research initiatives, etc., I think that that is something that we will get less in the future.

One of the liabilities of print products is the fact that you know how many people are going to go to a given journal and then just keep reading backwards in time? Very few of them whereas with electronic databases our abstract database goes back to 1887, covers the periodic literature for the entire history of the existence of psychology as a separate field. If a professor should ask a student what are the historical origins of this question, you know how were they looking at it in the twenties, in the forties, in the sixties and today, and that is accessible through an abstract database that goes back to that period of time and as I said we are going with the full text. We are moving in that direction as well. That should get less in the future because that is not a limitation of electronic access.

In terms of shelf life of articles I have fascinated with that question. I actually have data on it and I haven't published it. I think that many of us at least many of the people I know in graduate school you know we heard our professors say things like I think that the average article is read, when they are really cynical they say by two people. More realistically they usually say by 10 people, the editor and a couple of reviewers and a few close colleagues and in point of fact that is not true and that has never been true in terms of the print literature and the librarians are the people who have done the research that tells us how many times a print article gets read in its lifetime, defined as whatever you want the lifetime to be and their general consensus is that a print article will get read about 500 times in the course of its life and that should make people feel better.

I started asserting well over a decade ago that the average shelf life of a psychology article was 20 years. The truth is I made that up. Fortunately I now have data that says that I am right and actually I was not as right as I could have been. It is even longer than 20 years.

With electronic access to full text and I am the only person who has the information on the access to the APA full text article database I now know a variety of things about the access, and it is really exciting.

The first thing is that despite my own previous fears there is no article that is an orphan. In terms of looking at 2002 usage data by a sample of only 10,000 people every article for the 10 years prior to that was read by someone in 2002. I have data going back further. After 10 years there is a drop off but even back to 1988, over 90 percent of all articles published in 1988, were read by people in 2002, and I don't mean by one person.

In point of fact looking at 15-year-old literature the average number of people out of this small sample of 10,000 who read any one article that was 15 years old was 11 people. It ranged from one and two for some articles to as high as 40 people reading it and again this is only out of 10,000 people.

PARTICIPANT: Is that on hits?

DR. VANDEN BOS: That is on downloads of the full text article. The power and the accessibility of electronic full text is amazing because what I now know and through the electronic stuff is that in the first year of publication one-half of all articles will be read by more than 500 people. In the electronic environment I am currently estimating and I need to go back farther and have us move forward in time to be able to speak to it, in the electronic environment at least for the articles that we publish in the APA journals it looks to me like the lifetime electronic reading rate is going to be some number that will be in excess of 5000. How much in excess of 5000 I do not have a clue at this time, but the SAR(?) articles will be way above it, but I am talking about the average.

DR. FIREBAUGH: Brian, I think you noticed that no one has given you an answer yet to your question of what should the shelf life be, and I am not sure there really is an answer. Maybe that is the reason you are not getting one, but I think maybe more important than the shelf life or the half life which I think is a little more revealing than shelf life could be infinitely far back in time going by shelf life somebody still reading it 200 years later. Half life I think maybe gives you a little more idea about how current the article is and maybe more important than a measure of the central tendency or the range of a distribution would be a measure of the shape of the distribution and if you really studied articles and saw that they were bimodal that would seem to indicate support for some of these ideas that people have been throwing out about the recycling of articles, that articles are in fashion for a while; then they go out, and then they come back in fashion.

So, I would be interested not only in what the proper half life should be which I think varies enormously across disciplines but also an interesting question might be should it peak after 4 years? I mean what should the peak be?

PARTICIPANT: At least in psychology it is a very slow declining curve.

DR. EMILHOVICH: I think it is a very interesting question when you compare it against the humanities, when you think about the social sciences sort of standing at the midpoint between the humanities and the hard sciences. The hard sciences are about discovery and the shelf life or the half life is very short because somebody is continually building and finding new things.

In the humanities I mean people don't talk about the shelf life of Aristotle and Plato, for example, and as far as I know they are still reading them in college classrooms. So, we are kind of in a netherworld because if you are talking about interventions or ideas or ways of discovery that allow you to use that knowledge for some purpose of the social world I find that a lot of interventions don't hold up well, but if you take the standard of someone who is raising sort of big questions about some aspect of culture, about some aspect of learning, about some aspect of cognition some of those pieces can hold up for a very long time. I think of an article I read by Fred Erikson in 1982, on cognitive learning that is still one of the best articles that talks about the way to think about learning, the intersection of cognition and learning and from an anthropological standpoint it is a wonderful article, and it hardly ever gets cited now but it has a wonderful shelf life or somebody like Laura Nader wrote a paper in 1975, about the problem of studying up in anthropology that is now more relevant today than it was when she wrote it almost which is an interesting commentary about the politics of how we think about doing research.

So, I think the social sciences have a trickier issue to navigate. The humanities it can be timeless and the sciences it is the here and now and you know the hot discovery in genomes or whatever you are talking about and the social sciences are in this kind of netherworld of do we produce knowledge that falls into that discovery mode that the sciences have or are we in the timeless mode of the questions of how do humans live in this world in the social world.

DR. FLODEN: Jack?

DR. FLETCHER: I want to change the topic a little bit if I may and that is to ask about something that really hasn't been mentioned just yet and that is the relationship between publication lag, the amount of time it takes to g et a review and the amount of time it takes to see the article actually the readership and I am wondering if there is any evidence of a relationship between publication lag and either quality or impact. I know a lot of people think that better journals have shorter lag times and that shorter lag times are associated with more impact. I am sort of curious about what you think about that.

DR. VANDEN BOS: It is all over the map. I look at it across journals and I have looked at it within specific journals and I can't see a great deal of relationship between the publication lag and whether it is accepted, what kind of citations it gets in the long term, etc.

The question is how brief if brief or how short is short. I think that journals that get something and have it in print in less than 2 months and there are some journals that strive to do that, particularly in the electronic environment tend to have a somewhat lowered quality in part because there is some fact checking that doesn't happen, in part because there isn't the correction of the quality of the communication and there isn't the integration of various people providing input in terms of thinking about so what does this mean.

There are some journals that have publication lags of 2 years. Why anybody publishes there I have no idea but they do. They are not necessarily any higher or lower. I think that most good quality journals at least in psychology strive to once they make a decision that they have accepted it, try to have it in print in 6 to 8 months, and longer than 8 months is a slow journal. Shorter than that you can achieve if it is a monthly and a variety of things like that. One of the things that APA is doing to move towards faster publication once an acceptance has occurred is really two things. Through an electronic manuscript tracking system we have called the JBO or the journal's back office the moment that an editor accepts a manuscript they will be able to submit it for the start of the production process. They don't have to wait until they have the next issue put together. So, when they put together an issue they will be given an inventory of what they have submitted as accepted, what has been edited, what is ready to go and they will just tick off what they want to go into the next issue. We, also, are putting some of the journals up on a pre-press server so that once it is accepted it is immediately accessible to anybody who subscribes to the journal, not in the regular database because that is disseminated by a variety of people and they don't want to put stuff up and take it down when it is replaced but by an in-press server that we have available and again that is only achievable because of the electronic environment and the opportunities that that provides.

DR. FIREBAUGH: I think the biggest factor involved in the length of time if you are talking about from first submission to the publication of the paper, if you were to analyze a lot of cases it is how long the author takes to revise a paper and resubmit. So, you know, we have got very precise data for ASR.

Every year we do a report on how long it takes the initial review process and I am looking at the report from last year and these are two editors who were my successors. The mean editorial lag meaning from the time you receive the paper until you wrote the decision letters, 13.3 weeks and the median is 13.28 weeks. So, that gives you about 3 months and then typically at ASR we had a publication lag of 6 to 9 months from acceptance to actual appearance of the article and as a journal editor that was one of the big things you worried about the most. You need to have some backlog. You don't want to run out of articles but you couldn't have a backlog that was too long.

So, it varied. When I was editor sometimes it went up to 12 months, to about a year, but still the biggest thing I noticed was how long it takes the authors to get the paper back to you in a revise and resubmit.

PARTICIPANT: Does that have anything to do with quality?

DR. FIREBAUGH: I don't know. I mean you could argue maybe it does. Maybe if it takes a little bit longer that indicates there are more problems with the article to begin with or you can also argue if it takes longer maybe the authors are being more careful. I don't really have any sense of that.

DR. HENLEY: In the day of electronics why isn't it possible once an article is accepted to have it available at least electronically within a few weeks?

DR. COUGHLIN: Some biological journals do that as well. PNAS doesn't post the PDFs that were submitted because we think that there is a large value added in the copy editing that our professional printing press does. I am sure it will come as no surprise that many, many scientists are not strong writers and the copy editing is heavy. I think also within the biological medical culture it absolutely translates as you said that faster means better and there is a one-to-one correlation between speed from submission to upon the web and impact factor and prestige. It is hard sometimes not to have that drive the editorial process, not to rush reviews, cut corners editorially to maintain that but it is something that we wear very heavily on our sleeves. It is usually part of the information for authors. It is very public how fast you can turn around a paper and it usually translates also as a one-to-one correlation with the type of publisher.

Are you a society publisher? Are you a non-profit? Are you for profit? Are you an Elsevier or a large Macmillan press or are you a society publisher and how much money can you throw out at how many copy editors? How many editorial staff do you have? So, it is also a huge financial issue.

DR. FLETCHER; What is the lag time for PNAS?

DR. COUGHLIN: When you said that you were 6 to 9, I will say that we are at 6 to 9 weeks.

DR. FLETCHER: And isn't it correct to say that it is widely perceived by the research community that might be interested in PNAS that the speed with which you can actually produce an article and get it out in print reflects the quality and is seen as having a lot of impact, so that people actually select their best research to send to a journal that has a fast turnaround?

DR. COUGHLIN: Absolutely, and as we are getting more and more published that is endorsed by pharmaceutical companies rather than the National Science Foundation or the National Institutes of Health it is also proprietary. Once it is public domain it goes into the patent record and so if you don't want to get scooped for financial reasons you will look very closely at what journal you are submitting to as well as if you don't want to get scooped by a competing NSF grant. So, it doesn't have to be big pharma.

DR. FIREBAUGH: I would like to know what the lag time is for something like the American Historical Review. I think my guess is that it would be longer than 12 months. I think it has a lot to do with their fear of being scooped that there are probably articles that are accepted at ASR probably that are not going to get scooped. So, there is not quite the worry that you would have in biology and I think that there are other fields, maybe some humanities fields where it is more a timeless thing that they are looking at any my guess is there the lag time would be even more. You said, "Slow." I think somebody said, "Six months is slow." I think in some fields 6 months would be very fast.

DR.COUGHLIN: I think a lot of fields are going very cross disciplinary. So, it is no longer this codified, we publish social sciences, biological sciences, physical science, but they are starting to meld and something that I find that authors are frustrated in these intradisciplinary collaborations is that culture of speed. In the biological sciences you only have one chance to revise in most journals. It is not an iterative process. So, from our classical psychology section collaborating with social psychologists, they don't understand sort of the rules that we are trying to operate on and we are trying to have one rule for everyone, and it doesn't fit.

DR. FIREBAUGH: That might be one of the reasons why you say that scientists aren't good writers. I mean I am thinking about English journals where the way you express it often is as important as what you say and so in those cases maybe there would be a longer lag.

DR. TOWNE: I wanted to follow up on what Bridget was saying about the staffing capabilities of the editorial staff that you have behind the scenes and connect that with one of Margaret's observations earlier about the incredible logistical and management tasks that go into actually putting out some of these journals and pose the question in a comparative sense to the panel and that is it seems to me that some of the differences in the kinds of things that these journals are able to do might be a function of the resources that they have both in terms of dollars, in terms of staff, in terms of even the underlying infrastructure of the field.

Bridget, you mentioned having sort of depositories that exist on their own that you rely on at PNAS and that you only house data internally when that doesn't work. So, there is an external infrastructure that you are relying on as well and so I am just wondering across the panel how much of this is just Glenn and Lynn and Margaret are just incredible people who give up their time in ways that we shouldn't expect and how much of an infrastructure and resources should we be considering in thinking about education journals in particular and as they relate to other fields?

DR. EMILHOVICH: That is a huge issue for AEQ. I mean it is just huge because there are virtually no resources available to support it. So, it is really on the backs of the editors and whoever else you can get to help you and in more recent years because it is published out of the Council on Anthropology and Education it comes out of our budget which is then in turn given to us from AAA because the publisher is AAA because they hold the copyright and there is just no money and so it comes out of the backs of the people who do it.

If you are at a place at a research university where you can get support through your university there is a huge amount of in kind that the university puts forward for you to do that service. If you are at a place where they don't have that kind of support as one of our editors recently had then we had to put it forward as far as the council which then created a deficit problem.

We, also, had a problem in terms of you talk about a time lag of publication, one year our publication was delayed for a year and it had nothing to do with us. It had to do with AAA in terms of they had some turnover in the production staff at their end and all kinds of problems. So, for one year the journal was late almost a year and it had absolutely nothing to do with anything that the editors were doing and that is one of the reasons why we are going to have a big debate on going digital and whether AAA can even afford as an association to maintain because they have got about 10 or 12 journals. The big one of course for them is the American Anthropologist, whether they can afford to maintain all of those journals in the current format that they have because of the expense which has become so huge and we don't have the resources like having a major publishing company behind you or maybe a major association that has a much better financial picture.

DR. VANDEN BOS: One of the things that has been said here that does relate to this question is I think it is important to keep in mind that there is tremendous variation across disciplines in terms of expectation around speed in the price of the journals and whether they are published by commercial houses or associations and all of those will play a role.

The average price of an APA journal is one-half of what Elsevier or Blackwell's charge is. They have more money that they can potentially spend although they like a very big bottom line. So, they are not necessarily willing to spend all that money that potentially would be there to do, but it is not all related to money.

I started asking the question about why does it take APA because the APA model and to the best of my knowledge it has been the model for something like 40 years is that from the time of acceptance to publication it is built on a model that you cannot get something in print in less than 4-1/2 months which is why the target is 6 months, and it doesn't make any difference if it is, well, if it is quarterly it is going to take a little bit longer because it depends on when it gets accepted, and it took me asking that question of various people involved in the editorial processing and the publishing and whatnot before I heard the question enough times and part of what I realized was it goes right back to what I was saying about journals versus articles. How do you organize your editing and production department? APA has always organized its editorial department so that if I am the technical editor, I am the technical and production editor for a journal.

So, when the editor sends me 15 manuscripts I have 15 manuscripts to edit. Then I have 15 manuscripts to mail to the authors, to get their feedback. I have 15 manuscripts to change, etc. I need a fair length of time to do that. What I am doing right now with the production office is saying, "No, you don't edit journals. You edit articles," so that if this editor submits his next quarterly thing and it has got 15 articles in it 15 editors do it. All 15 articles are edited tomorrow whereas on today's model all 15 articles would be edited in about 15 days, 15 working days which means at least 3 weeks and more if there is a holiday or somebody takes a vacation day or something like that.

So, that one simple step changes a piece of the production process from 3 weeks to 1 physical day and that is the kind of thing that we are going through right now.

I believe that we will get the from acceptance and transmittal to in print down to about 2-1/2 months. Whether or not we are going to get it shorter than that I don't know.

DR. SCHNEIDER: This issue about infrastructure and some of the comments that Catherine was making I think have really been echoed in the AERA publication process. I don't know, Margaret if you were going to speak to that but the variation across the five journals in terms of the resources that people have that are available to them to do this editing process from their institutions is absolutely incredible.

Some people still are without help with electronic, they don't have the electronic units in their offices. They don't have the technical support to be able to accept manuscripts electronically. So, the question that I have, I mean given that is that in some way going to impinge on the selection of the editors for these journals and then what does that say with respect to quality and enhancing the profession, particularly within education.

DR. MC KEOWN: That was exactly the issue I was going to bring up, Barbara. Barbara and I have served on the Publications Committee at the same time because we were editors at the same time. So, we have seen a lot of these issues played out in AERA and I think it is a serious issue and getting more serious the more that electronic processes are important because AERA does not offer any help at all and if you are lucky enough to be at an institution where there are lots of resources in that domain to even get an electronic review process set up and that can improve things greatly as I have seen with a couple of editors; if you don't then you are out of luck at this point and there is also just basic other kinds of resources that AERA offers a limited amount of money for editorial assistance and so forth and so it really is incumbent upon the institution and as Barbara said one of the big things that it brings up is who can accept an editorship.

You know if you don't have a lot of support at your institution you just best not be an editor or you are just going to be sunk with the amount of work that you have to do on your own with no support or assistance.

DR. FLODEN: Karen?

DR. FALKENBERG: This is Karen Falkenberg. I wanted to extend the conversation from this morning about translation of research into practice. I thought you all had a lot of really interesting things to say and I thought I heard some synergy building when one person would speak with another.

So, I wanted to push that a little bit farther because we know that we have a big gap there that the purpose of journals is not just to inform ourselves and is also not just to advance careers but really is to make changes in policy and practice and I wanted to kind of get some more thoughts from you all about how we could do that and who could do that.

DR. LIBEN: I am less familiar with what AERA might have with respect to this issue but SRCD for example has the social policy reports so that in essence what it has done is to create another kind of small publication that is specifically intended to get the research into a digestible kind of review form but in a review form that is meant to go to policy makers, legislatures, etc.

The public summary idea that we have been using I think has been very helpful but again one of the reasons it is I think being successful is that we do now have a DC office with someone who is charge of communications, public policy and communications and she has got a full-time job. This is not the only thing she does but it is part of that and just listening to the comments about support for the journals and editorship that you were just talking about the AERA, I mean AERA is a pretty big organization and I pay my dues and it seems as though that essentially there should be support for some of that journal stuff as well as this kind of public policy translation making people aware whether it is by special initiatives and publicity about specific things or whether it is another kind of structure for organization.

DR. FLODEN: I know that AERA has just started a new publication which is notes intended for policy or research notes, is that the name of it, Jerry? The first two issues are out, one on class size, one on curriculum alignment.

PARTICIPANT: Research Points.

DR. FLODEN: Research Points.

DR. EMILHOVICH: This is a really big issue for the Council on Anthropology and Education, not just with their journal although it is framing; it is coming out in that way.

There was a very famous debate that got started and I can't remember the dates but it was around the point where somebody in the Alaska Department of Education put up a sign that says, "We don't need any more anthropologically based explanations of school failure," and what she was really saying was you know, you are publishing all these articles about why kids aren't doing well in schools and here are all the cultural reasons. Meanwhile you just keep publishing articles about kids not doing well in schools and here is all the cultural reasons but meanwhile the conditions for those kids don't change and it intersects with diversity in a very interesting way because we have a group of scholars of color that are pushing this issue very hard now. It is what is the purpose of this research if not to begin to address some of the deep-seated questions of why schools are not changing and what does it mean that all we are doing is producing a field of people that for 20 years and if you take education as a whole you know that is what I said about my comment about going back and looking at stuff earlier. You can go back to the sixties and read articles about the disadvantaged. You know, just put on the new label now. It is called at risk but it is the same stuff.

So, now we have got 50 years of research on the disadvantaged, at risk, children not doing well, you know the undeserved, the under performing, you name it, you know, come up with the label and nothing has changed. You an go into schools and they look just the same and it is raising a very, very serious question I think for education research in particular as to what becomes the point of the research if the schools and the very field that we are presumably hoping to influence is not changing or research means so little and I don't mean to say that, it doesn't mean little, but I think that is a hugely serious question that has surfaced in CAE.

I see it, also, surfacing in AERA and I think it is a question that raises the larger question for the role of the social sciences in informing policy and practice because so much of what we produce as knowledge is not intended in the discovery mode in the way to get at the sciences do but we can't claim we are the humanities raising these big timeless questions of philosophy or classics or whatever. So, what is it? And I think that that is a real dilemma for those of us most sharply in education because we are also getting messages from Washington about the importance of developing evidence-based practices that can help change the way things go on in schools ultimately so we do do things like close the achievement gap which is the real issue behind no child left behind.

DR. FLODEN: I would like to ask whether other members of the audience have questions or comments they wold like to pose.

Please say your name?

DR. COOPER: This is Harris Cooper, again. I want to bring us back to this core question for CORE I guess which is what the Committee might recommend to improve the quality and potential accumulation of educational research that is published in education journals, and I think we have gotten pretty close to the way I would answer that question and I will ask the members of the panel to see if they can answer that question specifically before we close up, but we have danced right up to the issue of reporting standards and we did it in two ways, the first one being give us some examples and the second one being give us your data and obviously give us your data is the ultimate reporting standard but for anyone who has tried to judge the quality or evaluate the quality of research or accumulate research you know that the most frustrating aspect of what you are trying to do is stuff you can't find out and it can be the most simple things like whether when they say, "At random," they mean random assignment or random sampling when we can't figure out how many subjects were in a controlled condition or a treatment condition, when somebody reports a T test statistic but doesn't tell you which mean was higher than the other mean. It is an incredibly frustrating thing and just asking for the report, Gary mentioned the APA Style Manual. I would argue with him that just the Style Manual is also a reporting standards manual and anytime you ask for information about how a study was conducted you raise the standards because people will know. They know what the standards are and when you ask how were students assigned to conditions people will recognize that the way to answer that question will be part of how, has an implicit evaluation. So, I would encourage the Committee to consider as they consider their report reporting standards and potentially I know Hannah is going to talk a little bit about consort and trend which are two sets of reporting standards that are coming out of medicine.

They are equally applicable to education and so again just to wrap up both from the point of view of improving quality and accumulation efforts reporting standards would be critical.

Now, having said my piece I would be curious about asking how the panel members would answer that question for the Committee. What could they do, what could they recommend to improve the quality of education research.

DR. FIREBAUGH: The examples you gave of having a variable where you can't tell what is high and what is low or maybe a dichotomy where you don't know how it was coded or not knowing what kind of sampling or what they meant by random, that just sounds like bad editing to me and bad reviewing.

I know that one of the ideas that was kicked around for ASR, and we didn't have the money to do it would be to actually for the quantitative papers in ASR to before they were actually published, I mean as editor I would go through them pretty carefully and then the reviewers certainly should have gone through them very carefully but to have to pay a statistical consultant to go through and see if they could replicate the results and then to me that is step past what you are asking for but what you are asking for seems to me to be the bare minimum. Hopefully you wouldn't find any papers in ASR where you wouldn't know which direction the variable was going.

DR. LIBEN: I do think that anything that could be done to make it easier, in the APA manual you can always at least point authors to it and say,"Please see Page 63," or what have you.

We have been gradually instituting more and more web-based instructions check-off instructions for final preparations of manuscripts but I know that one thing that I have found as editor and with a dozen associate editors it is really hard in fact to keep track of every last thing even some important ones sometimes that slip by. So, the idea of having a consultant or somebody go through it much as a copy editor goes through it would be great, but I think it would be really nice if collectively there were some decisions about what ought to be reported. I mean we have discussions about mean square errors or things that, or affect sizes that are often not included and we all say should be included and should we legislate them and then every editor has to make sure to remember that they have checked for X, Y, Z or rather A, B, C, D, E, F, G, H, etc., and it is just you know you are dealing with over 500 manuscripts a year and it is already more than a full-time job, and so I think if there were some ways to succinctly have kind of the equivalent of the APA Style Manual with certain kinds of expectations for everybody about how statistics should be reported and so on it would be, and someplace that is very accessible and very routine and that works for a huge proportion of papers. Obviously it is not going to work for everything but it would work for a lot. I think it would relieve a big burden and maybe increase the quality along these lines.

DR. VANDEN BOS: I think the reporting standards is evolutionary. You have got to get the field to believe that it is important to report a piece of data before you can really mandate and it is at what point do you have 80 percent agreement that a piece of data is relevant before you can make it a standard or what, I think the key thing is you facilitate discussions over and over and over again. It has been in the career of all the people in this room when the gender of the subjects was mandated for the first time. We are now dealing with the same question in terms of when is it or is it not important to indicate the ethnicity of the subjects. In developmental research, in educational research, in clinical research there is much more willingness to accept that that is probably a relevant variable and they ought to at least have some percentages down.

When somebody is doing neuroimaging with 14 people I have sat in rooms with researchers and editors who do neuroimaging and they have told me that I am an evil person for suggesting that they might use two lines of type in their article to tell people the ethnicity of their subjects. Their answer is generally there is no theory that says that because of ethnicity there should be some difference in this, that or another aspect of what they are imaging or a biochemical process. Well, I don't know that that is true but what I do know is that if you don't tell us what it is that 15 years from now somebody who is looking for a sample that is either all one ethnic group or has a high predominance, etc., isn't going to be able to access and reuse that data to address a question related to a theory that we might have in the future, so that I don't think that we can come up with an awful lot of things that we can really apply as a universal for a whole field.

I mean psychology is not a field. It is at a minimum 15 fields. They are closely related and they overlap with other fields and it is getting a reasonable number of people presenting a reasonable range of variables that they are routinely reporting on that helps you to advance things and some of those things will become universal as they start to be important and work and some of them will drop away.

DR. FLODEN: We have time for one more question or two more, you and then Jerry.

DR. ROTHSTEIN: Thanks, Hannah Rothstein. I remember when I was a, I think I was still a graduate student although all of that is kind of getting hazy to me that David Weiss was the editor of I think Ed and Psych Measurement and he came up with a very radical policy which was that he was willing to accept articles for publication before the results were in based entirely on the theoretical importance of the question and the adequacy of the method section.

I happened to think that was a brilliant idea. Of course it avoids publishing things based on results and statistical significance levels. It didn't last very long and as far as I know no one has ever adopted that practice again but as a challenge to the editors and publisher assembled here I would just like to throw that out as something that you might want to consider to avoid essentially retrospective bias about how important a particular question or how adequate a particular set of methods was.

DR. FLODEN: Was there a response?

DR. MC KEOWN: Sure, I will respond. I will just say that I think that some aspect of that might be an incredibly great idea because I know in a lot of the articles we get at AERJ it seems like all the authors feel they need to send you are the results and we say that you need a conceptual framework. You know, you need a theoretical orientation and maybe I don't know that that would fly, you know, submit papers without results. Maybe if we could get the word around that we are going to send these to the reviewers without the results section and that would be the first round of review or something like that, I think you do have an important point there.

DR. SCHROUF(?): Good afternoon. I am Jerry Schrouf. I work for the American Educational Research Association and I guess I am standing in for Richard who is not with us today. I had two observations and one point I wanted to make or try to make.

The first observation is that also put all of our journals on CD-Rom which is word searchable and I hadn't thought of this use of it, but one could look up Robert Lynn if they were thinking of him as a reviewer and find out he had written 36 articles in JEM and you would then have that database for your looking or reviewers but I had thought that the idea of putting these things online meant that their shelf life would extended indefinitely and I think that when we talk about the very first article, Volume 1, Number 1 of ER is about class size. So, it is cyclical but it still is interesting to know what that article was about.

The second thing I know that if Richard were here he would want to hasten to say that there is work under way right now to select an electronic journal submission and tracking system for AERA journals and at the last time he spoke about this he was enthusiastic about a near decision and then finally the observation is a few years ago, maybe 10, AERA spent a great deal of money with Bob Glazier in Pittsburgh to come up with a more accessible socially politically relevant journal, PR and they did a whole number of things to make it more accessible and the way you could tell that the journals were, one was the experimental model and one was, it says, "Research Association." So, we had the experimental model and we had the regular model and the experimental model had a tag across it that said, "Experimental issue," and so I sent these up to Congress and congressional staff and asked if they would read half a dozen of them and then talk with me about what they found. So, when I called back after a reasonable period of time they said, "Which one was the experimental one?" even though it had the sticker on it and the reason was they said, "Even our most accessible effort was still jargon filled, theoretical and not relevant to their needs," and so my conclusion was that something like, and this I think answers the question that you had or addressees it, something like Research Points is going to be much more accessible and it has to be really out of the academic journal field if it is going to have an impact on policy makers.

Thank you.

DR. FLODEN: Thanks. Jerry points out that two of the people on the program didn't make it today. Richard Duran had a death in the family and Ed Silver had a medical problem with one of his associates.