DR. TOBIN: We now go into something called wrap-up. I have been asked to start it off, but that doesn't mean I am going to speak for the next half hour, but I will make a few comments that will be intended to get us to have a discussion across the papers. Is that right?

I have been typing away trying to both summarize and plan what I would say on the fly. So, we will see how this works, but I have five tensions or themes which I saw as cutting across the talks today. So, let me throw those out and we will see if it works.

I think one of the questions that has come up throughout the day is the question of whom do journals or we should perhaps say articles speak to, and we heard a range of audiences.

Barbara Schneider's talk at the beginning by focusing on citation as a critical measure of quality also implied I think a readership. We could say that if we evaluate journal articles by seeing who cites them, that is another way of saying who their primary audience is.

So, I don't mean to say that she made that argument explicitly, but I think it is an interesting case. It implies that perhaps most scholarly writing in journals is really aimed at the kind of people who are going to write other articles and so that is why citation becomes so important to us.

The panel of journal editors had a similar but I think a somewhat broader notion of the audience. They also I think talked about the audience basically being scholars in these disciplines but they also mentioned some concern about expanding the reach through things like special issues or something that the journal as an entity puts out but not under the heading of the refereed journal.

The panel where Judy and Hannah spoke was making this a more explicit problem and adding a sort of politics to it by saying that, what I heard in it is a kind of accusation to the field that if academics keep publishing papers in journals which they are doing for themselves and for their compatriots who also write journal articles they are excluding some other audiences who have a real right to this and perhaps even should be a primary audience which includes policy makers, and it is partly the problem that these authors aren't thinking of these other audiences which requires a kind of correction through going back and either telling people that you really need to write differently because the way you are writing is inaccessible, if not to fellow academics to a lot of other audiences. You are frowning.

DR. ROTHSTEIN: I didn't really hear that as my message or Judy's perhaps. I am trying to think about what part of -- my understanding of what we were saying was that if we are going to use the current literature as a kind of database that information in abstracts needed to be presented in a way that was faithful to what was in the article and that the article needed to be reported in a way that was faithful to the way the research was actually conducted.

It wasn't meant to be political. I think Judy and I probably each have our own feelings about who the audience is and stakeholder should be but I think that the primary message was meant to be purely scientific.

DR. TOBIN: Okay, actually maybe I should finish and then let people respond but I don't mean to say that is all you said and I didn't mean it as a critique. I was saying that I think you were, I was talking specifically about this notion of whom journals address and suggesting that I thought you had a somewhat broader notion that these kinds of abstracting and keynotes weren't only so that other scholars could also find them more quickly or efficiently although that is part of it but that other kinds of audiences might start being able to use this more effectively, and again I would say that although I think some of the impulse behind this isn't as Judy made clear coming primarily from academics who are frustrated with themselves and each other. It is coming, I think she started out by saying that there is a great deal of frustration that started from the bit that was voiced by the government.

So, if that is not the case, I mean we could argue where this is really coming from in the US.

Then John and Gary had a broader notion of audience still and a lot of what they were talking about again wasn't just that academics have access to scholarly writing but that the new technologies and the notion of open access also makes people including practitioners and parents have a sort of equal standing and even people in other countries who can't usually afford these journals much less want to read them.

So, that is one kind of question. I don't mean to say that these were comprehensive reviews of all the papers but just to suggest that there was a continuum of kinds of notions of audience put forward across the presentations and that is something I think we need to think about when we think about journal articles, that maybe we are not explicit about.

We talk about circulation but that is a different question than like addressivity(?). The second issue i am calling something like if the first is addressivity the second I will call dialogism(?).

By the way I am really into Bactine(?) these days and these are terms that are kind of key Bactinian concepts. He is sort of my main man.

So, you could look him up on one of the sites that was shared today but dialogism, what I am thinking about here is we had also I think a theme having to do with the presence or absence of a kind of dialogue between authors and reviewers and editors and readers and authors and other authors and the larger public, and I don't think this is something we have talked about explicitly enough, and sometimes a journal may seem to be sort of inert or not to be communicating with anybody.

It is a thing but really this is all about communication that is just happening to be through the form of a printed journal or an online journal.

So, then I think we have seen today or heard different kinds of notions of what kind of dialogue is going on. On one hand we have the review process as being a sort of gold standard for dialogue, perhaps ironically or not because it is blind, sometimes doubly so, as if you don't know who wrote it and you don't know who is critiquing you we can have a kind of higher level of engagement with the argument. It is a sort of platonic notion transcending not only our bodies but our identities.

I think of some very interesting comments especially from the journal editors about a notion of a kind of special dialogue, something like what goes on between the editor and the would-be author that was maybe held up as even more ideal than the reader's report and interestingly it is not disembodied or anonymous and it was presented today as being very gratifying for the editor who gets to have this kind of role in helping to create what will be an important article or an article that maybe never is important but wasn't even good enough to be published until it was given this editorial help and so I think that is an interesting sort of ideal that was pointed out to us is unappreciated, unacknowledged and to some extent invisible unless we have had the privilege to participate in it.

So, it gets held up as another kind of gold standard, almost like a psychoanalyst and analysand, you know something that takes place that is kind of invisible to us but was presented as being very special, and I think it was interesting to hear about it.

We haven't heard editors talk about their role very much partly because it is awkward to do so because in theory I think sometimes when we think about review a journal that follows a peer review process we don't really like to think of the editor making decisions because it seems like too much power for one person to have, but we also saw the flip side of the kind of benevolent help.

Citations in Barbara's talk become another kind of dialogue but one that gets reduced to sort of numbers, not reduced in a negative sense but an interesting one and I know a lot of people who run to the library to see each year how often they have been cited and by whom, and it is not just a narcissism thing.

It is also a really interesting way to find out when you write you hope somebody will read it, other than your mom and you don't know who has really read it, and it is very interesting and not just gratifying but sometimes enlightening to find out who read it, an it is not enough just to know who cited you but then you run and get those articles and then you read them and say, "Oh, that is what they thought I said," but I am saying that is a really important kind of dialogue that is kind of rare, that when you write you often don't know what anybody thinks of what you have written and without citations I don't know, you didn't talk about the particular value of them, but I saw that in what you were saying.

As we move to the online versions we get this sort of exploding of dialogism. Everybody can write in and sound off and we have that kind of trade-off of how wonderful this is because you have a community that is in conversation versus a kind of noise problem.

We, also, with the journals mentioned a couple of models of building in and I wish we had heard more about this, some kind of exchange between readers and authors. I think there are one or two examples of where you could write a reply or there would be a theme.

There was a theme issue with one article and then maybe six commentaries and this is done in some of the journals we heard about and not in others.

You know I think it was the New Yorker that didn't used to take letters and then some others, Spy magazine had letters to the New Yorker and then the New Yorker had to start doing their own letters, if I have that straight, but I think it is interesting this question of how open should a journal be to having its authors publicly confronted by the readership and then how long do you let this go back and forth, and then in the electronic journals we see that this gets to be much easier to do and there is a kind of lag time problem with printed journals but I think this is something to keep thinking about.

The third issue I would call the kind of diversity or heterogeneity of academic cultures behind the journals. So, we saw a lot of diversity on some things like turnaround time, what is possible and what is desirable and what is considered unconscionable and how that varies and we had this notion of the timeliness of information somehow being part of that.

We see a change with new technologies in the turnaround time issue, but back to the diversity issue, we see very different notions of timeliness, of half life, shelf life.

We see different notions and we didn't hear enough about this of what constitutes good writing. Some of the desire to have kind of standards for good writing I think hasn't dealt, didn't say enough about genre and disciplinary styles and I would think that even within psychology some of the subspecializations would not all be comfortable with the same kind of notions of what minimally has to be in and how standard it should be.

When we start looking at education as a field I think Ellen's question brought this up but there are very different notions of what constitutes good writing and good reasoning.

So, I think that is sort of a brake on the impulse to standardization. Even the question of whether journals are the most important currency varies within the more humanities oriented versus social scientific versus more purely scientific specializations within the school of education or educational research.

A fourth theme I am saying has to do with the notion of diversifying or the diversity of whom produces and whom consumes journals. I don't think we heard enough about this.

We heard a few good comments but John's talk at the end mentioned about reducing costs having a real diversifying potential but on the other hand the technology needed to access the kinds of computers and the speed of the computers has some implications for whom can use them, but I think there was also this notion that was mentioned this morning about needing to diversify the editing process and people who have control over what gets produced and consumed in journals and so when we talk about the quality controls I think we need to add that sort of politics and power issue.

Who are the people who review, who write, who edit and there was a call for more diversity and more self-conscious diversity and it is not just racial diversity or cultural diversity. It has to do also with internationalizing the kinds of people who participate and it also has to do perhaps with different kinds of stakeholders.

Then the last theme is this last issue that came up of sort of standards versus access and I think this is a really important issue and I could sense people really disagreeing about this and I think we should argue about it a little bit with some of the time left.

The kind of standards access issue or I am not sure, I think people were uncomfortable calling that a mediation issue because everything is mediated in some sense, but we had journal editors as gatekeepers as a kind of version of this.

We had peer reviewers as gatekeepers. We have notions of like a what works clearinghouse that would say like, "Let us reduce the amount of stuff that is crappy and only have good stuff," that kind of notion and then we have this what is called less and less mediated forms where the architecture of the site claims to impose little or no meaning or value judgments and then finally we had the sort of intriguing possibility that these kinds of open access sites can actually combine.

Within them they can incorporate very strict kinds of judgments. So, you could take criteria including like a what works clearinghouse kind of criteria and use that as your filter that you could put over an open access site.

So, those are my five themes and let me just turn it over first to the Committee. Is that how we do it or just to everybody?

PARTICIPANT: It is late, whatever you want.

DR. TOBIN: Whatever anyone wants to do.

Again if Hannah or any of the other speakers want to respond to what may have been my unfair characterizations that is welcome also.

Barbara?

DR. SCHNEIDER: I thought your characterizations are fine but there is a theme to what we are doing that I think and it is one I think that we have been somewhat a little bit uncomfortable with and that has to do with this whole issue of science and science in education and the accumulation of knowledge and it seemed to me that there were a few themes that were raised around this that we might want to incorporate in our summary.

One certainly had to do with the question about replication. Another had to do with building capacity in the field and how do we go about building capacity in the field particularly as it relates to this whole question about journals in publication.

What responsibility do we have with respect to our younger colleagues in terms of offering them guidance on where they should publish? What will be the consequences of that in terms of their careers, the issue about the discipline journals versus the more education-based journals.

Someone came up to me in the break and then wanted to raise the question that in the sciences now, the sciences are developing their own education journals that aren't even on any of these lists and yet these are the only journals that someone like a physicist might read, that he would never pick up an AERA journal but he would look up physics in education. So, are we in fact building our own barriers and not getting access to other groups that potentially are very important in terms of what we do?

So,those are just a few things that were on my mind. Perhaps maybe other panel members could weigh in on their own thoughts.

DR. FLODEN: Bob Floden. I was just looking at the title of the workshop which is The Role of Journals in Developing the Education Research Knowledge Base, and thinking that, I guess it is sort of a puzzle for me. You can think of different strategies for trying to develop the knowledge base.

One is to make everything that is ever written accessible so that people can find it all and then figure out what to do with it on their own perhaps and the other is to have, and it is probably a continuum but the other strategy is to have institutional structures where you have to put it most bluntly a hierarchy of journals where the places where you would go if you wanted to find the things that are the most authoritative because you have the sense that they have gotten the most stringent review and I can see something in each of those strategies and one of the things that has always puzzled me, not puzzled me, has interested me in meta analysis is it is sort of like people joke about economists, some indifference to the quality of the work beyond the fact that the particulars are reported, that you want to get everything and it is an empirical question whether or not including everything that has ever been written gives you a different result than including things that are published in peer-reviewed journals there there has been some other sort of screening process and one of the things about meta analysis that happens in that way is some inattention to the sometimes subtle but perhaps differences in the ways in which people cast the questions that they are trying to answer.

So, if you want to find things about a particular topic because we don't have uniform language and because we don't, everybody thinks slightly differently about what the questions are you end up inevitably combining some things that people wouldn't say are about exactly the same thing into a single study to see what effect they have, and that is a limitation or it is something to pay attention to and in other sorts of systematic reviews you may want to be aware of everything that is out there but people try to figure out both what it all adds up to but also to look for the different ways in which people have conceived what the question is in order to think how that affects the way in which the question ought to be asked.

So, I mean through the conversation today I have heard a lot of things that are related to this building the educational research knowledge base and some things seem like sort of pretty clear things you would like to see more of like greater clarity, making it easier for people to figure out what an article is about so that you could find the things that you want to find and providing more information in an article about what you did. I have had this same frustration that other people spoke about of going to find things only to find that half of the things you don't exactly know whether to trust them or not because you don't know what they did, not even to replicate but to know whether the conclusions are based on the evidence they collected or these are just the conclusions that they came in with and then they collected some evidence and made the same conclusions.

So, it seems like journals would be well advised to encourage people to be clear about what they did, to put some more material into the abstracts about that so it would be easier for people to find things, to do the sort of things that editors describe around trying to help authors be clear about how they can make their work better and there are other things where I am not quite clear where to go in terms of the thinking that you should get everything anybody has ever written as an important thing to do versus drawing on a system that tries to build a panel of reviewers where you are collecting systematic information on how much you trust the reviewers and using that as a way of screening information.

DR. TOBIN: Benetta?

DR. JONES: Thank you. Benetta Jones. In the presentation on the role of technology and contributing to progress and the knowledge base of journals I think that the whole issue of making more of a connection addressing the gap between researchers and practitioners I think was an issue that perhaps could be an important one and one that there could be a lot more room for exploration.

I think that is something that has evolved as an important issue to address and one that certainly might play a role in moving education research in a way that is seen as more relevant and building on and addressing issues that are of great concern.

DR. TOBIN: I guess we didn't really talk about any practitioner-oriented journals, did we, for instance?

DR. JONES: No, but the technology talked about ways in which those who are practitioners might be drawn into the discussion, give feedback and you could think of next steps in that with online journals.

DR. TOBIN: Yes, but I am thinking we had a couple of models. One is how the regular academic or scholarly journal can make itself more accessible to a broader audience including practitioners, but then I am thinking in addition to that there are also many of the content areas, for instance like reading and math will have journals that make that their explicit, make this transfer of knowledge from academic knowledge to implications for practitioners their sort of mandate. I think that is a good thing to add.

David?

DR. KLAHR: I think there might be a dualism in the term "practitioner." In some areas meta analysis has grown to such a developed area that the people who do meta analysis have quite a bit different training and set of expertise than the people whose research they are meta analyzing. So, occasionally I have found my work in a meta analysis, and I look at it and I think well, that is an interesting category. I mean I wouldn't have my research there and then I wonder what did I do to give them that, a bad abstract, or was it bad reviewing and so in the sense that researchers often don't understand what teachers, the real conventional use of practitioner doing I think sometimes the people doing the meta analysis may make mistakes about what the researcher had in mind and what the enterprise was all about and I just wonder if anybody has ever tried to you know in a meta analysis that covers say 500 papers if they have tried to sample say 50 of those authors and ask them what they think of the way their work has been categorized because I think in the sense the same reason we need feedback from the classroom to the bench researcher I think we need feedback from the researcher as practitioner to the people doing the meta analysis and I wonder if that has been done in the literature yet.

DR. TOBIN: Judy?

DR. SEBBA: As I said before there is not much meta analysis within the epi reviewing system but certainly as far as the systematic reviews go the original authors of the studies that make up any particular review are particularly targeted for comments following the review and we have had some feedback from that and I think the point that you raised about how a study is presented in a review or in a meta analysis partly reflects some of the problems we have identified today about the lack of detail in either the abstract or the lack of representation in the title or whatever it happens to be.

So, I think there needs to be, you know, I am not accusing you personally but I think there needs to be some self-reflection in that process as well of did I present my work in such a way as was reasonable to expect these people to categorize it in the way that I would have categorized it or have they misunderstood the way in which it was conceived, and I think that is a genuine and open question at this point and very difficult for us to get through it. Could I just also add that one of the things that was on one of my lists but I didn't talk about it because of the time and I think I would like to know what other people's reactions are to this was the network of journal editors which I put up which is actually a Canadian-initiated scheme for medical journal editors but I just wondered whether you thought there was some milage in getting editors together more often because it seems to me I mean not necessarily maybe virtually together rather than necessary physically together but it seemed to me that some of the problems that we discussed today come about through isolation and that there just needs to be more debate about the kinds of standards and the kinds of implicit quality standards that people are applying in selecting peer reviewers in dealing with problems of editing and so on, and editors seem to me to be quite an isolated group relatively speaking.

I just wonder if there is something that we can do to push that forward and that in itself would then help with some of the other issues that you raised in your wrap-up.

PARTICIPANT: Barbara, didn't AERA convene a group of editors at their annual meeting this year?

DR. SEBBA: Yes, and they were telling people how to write.

DR. SCHNEIDER: The policy of AERA with respect to the AERA journals is that the publications committee and the, Maudie are you still here? I don't know if you still are and the journal editors meet together and the idea of them meeting together is to talk about issues of quality related to the journals and they do that at least twice a year,once in the fall and once in the spring.

In addition to that on the AERA program they have a series of meetings and I think that what Judy was talking about is one of the sessions that is held at the annual meeting where in fact they present to people that are in the audience how to write and what the standards are for different journals and what the review processes are for the different journals. It seems to me that the discussion here today raised a different issue which was communication among the different journal editors with respect to standards, with respect to questions about assessing quality. These are the kinds of things as I have said in my original remarks that I wish that there was a lot more of this.

What tends to get discussed at the publications committee meetings at least for the few that I was at, and, Bob, I would appreciate it if you would please say something that we would have your opinion as well, it seems to me that there is a lot of talk about personal concerns. I don't have Internet access. I don't have the right kind of support. I am getting these kinds of problems. These are my issues. So, it really has to do with more the infrastructure at each particular institution and how the editor is in fact dealing with the management process of managing their own journal rather than these larger questions about quality.

DR. SCHROUF(?): Barbara has got that exactly right with regard to the meeting that she is referring to. There is also a meeting of editors that are not AERA editors but all editors and that would be, I am Jerry Schrouf from AERA and that would be a much more useful meeting than the kind we are talking about and I think the idea of doing that virtually as well as on site would be good and I am just told that Judy will be addressing that group next year.

So, that will be a good start. While I have the microphone and now am not speaking for AERA but for myself I was a little puzzled about the mediation aspect and maybe I just don't have a good handle on it but I thought the business that many of us are in which is providing doctoral training to education researchers was in part to make people have the capacity to mediate and it seems to me that if we are not going to then use that skill and perhaps I am overreacting to the general notion, then I am wondering why we bother to do this and it seems to me I guess that is the topic for tomorrow perhaps.

Thank you.

DR. TOBIN: We are up to 5 o'clock. So, presenters thank you very much for your presentations and everyone for coming.