DR.TOBIN: Thank you. Okay, we will open it up first to the Committee members.

Brian?

DR. JUNKER: The TC Record looks very cool, but it also looks like an incredible resource to keep going and whatever you can say about that I would very much like to know about it.

DR. NATRIELLO: It does require a lot of time and it requires a lot of time on a pretty consistent basis. We thought long and hard before we decided to go online weekly because weekly means 52 times a year plus.

I think ultimately we have been motivated to do it because it is new, and it is experimental and we are learning a lot from it. I think ultimately to sustain it it probably needs to be connected to some larger institutional mission.

Now, whether that is in this case a college's mission, whether that is a mission of a particular instructional program, for example, at Teachers College we now have a set of new journals that are in development that are anchored to degree programs and the way they are generating the resources or the commitment to that is they are seen as part of this sort of blending of publishing and teaching and involving students and faculty in that process but you are right it does require that commitment.

DR. JUNKER: We have been publishing weekly for a while. What are the resources that you put into it now?

DR. NATRIELLO: Right now we have a full time person who handles the technology. We have a half-time person who handles book reviews. We have me and we have three other editors who are working on it although the other editors work primarily on the print journal products and I end up doing most of the sort of online transposition of that material, but it is not impossible to do and the tools have made it pretty flexible, and we have gotten some tremendous, just to give you a couple of examples, some tremendous input from readers. For example, one of the major things that we did and this is probably the best example from an editorial perspective because there are fewer problems here but we made a major commitment to increase the number of book reviews that we published in the field and i think we were publishing maybe 20 or 30 a year and I think next year we will publish close to 300. Ordinarily that would have been I think something that we wouldn't have thought about taking on but at one point we put out a call for book reviewers because the hard part about getting the book reviews ramped up is how do you find that many reviewers and we had about 1400 people who wrote back to us and offered to be book reviewers and in addition to that they wrote biographical statements and/or included links to their online CVs that allow us to determine whether in fact they have the expertise to be book reviewers and I won't say we haven't made mistakes. Most of them I think we haven't published, but one or two we may have published, but for the most part we have been enormously gratified with the response of the book reviewers.

We have, also, had tremendous luck in using our readership as reviewers. I know a lot of folks who have talked about, you know how do you select reviewers and how difficult it is and the key I think for us seems to be just to have a very large pool of people so that as people do not honor the commitments that they honestly intend when they make them, in fact, you have five or six or seven people who you can ask and if two or three or four come through you are in reasonably good shape, and again what we have developed is a system whereby we are able to link biographical information into database with the reviewer, with the history of how he or she has reviewed for us, with some qualitative judgment as to whether those reviews have been helpful or not in the past and over a period of 3 or 4 years we have now gotten to the point where we have a really well-developed pool of quite good reviewers that we can count on. That turns out to take an enormous amount of strain out of the editorial process.

We can always do better but you know, we couldn't have made that progress without sort of a reading audience and I should say, you know, along the lines that John was talking about and others in terms of expanding, we will also deliberately go outside the small group of people who are well-known scholars to pull a reviewer. So, for example, if there is a paper that is about secondary school mathematics in addition to national scholars in that field we will try to pull in a secondary school mathematics teacher or curriculum specialist to also give us a perspective on that piece and oftentimes they spot things that the other reviewers just never really thought of or offer some really helpful comment about how the author could put this in a set of terms that would make much more sense to folks.

We, also, did this with parents, with newspaper reporters, people who are in a different kind of audience. So, for example, we had a paper on the sleep patterns of adolescents and it hasn't appeared online yet. It has been out in the print journal and you know someone pointed out to us this is really an article about bedtime and from a real practical parental perspective if you could cast it that way you would have a different kind of audience come to it. It would be no less technical, now less revealing about what we should do in terms of school schedule and things like that but from the public's perception it was just a different way to look at the article. So, we have gotten lots of help just from those kinds of things.

DR. EISENHART: Margaret Eisenhart. I just wondered, you seem to have a lot of data about your journal, and I am curious whether you know much about the readership in terms of characteristics like if you compare the print version and the online version are teachers more likely to be represented in the readership of one versus the other? Are young scholars more likely to be represented in the readership of one versus the other? Can you tell us something about that?

DR. NATRIELLO: Sure. We actually do know a fair amount about the readers. We don't know as much as we now think we should know, that is when we first, and this was in 1999 when we first went to this system of having people register we didn't think we could ask them too much about themselves. So, we asked very little but in fact they did volunteer all kinds of things. So, they typically have their institution. They have their title and things of that sort and so we have got a sense of that.

What we don't know much about oddly enough are our print subscribers other than where their address might be, and like many journals most of our print subscriptions are now going to institutions. Our print subscribers for the most part I think had been people in higher education. So, they were educational researchers. Most of those folks can now get campus access or desktop access from their library digital subscription. So, there is not a compelling reason particularly to get the print journal anymore. So, they are a dwindling number.

The readership that we have from what we can tell is a much younger readership online and indeed one of the things we discovered, we did some studies back in the mid-nineties when our print subscribers were declining and as best we could tell at that point the journal had a very committed group of subscribers, many of whom were approaching retirement and so we were going through this period where the journal subscribers were retiring and we were not replacing them with enough print subscribers from among the younger ranks. We tried lots of different ways to up that kind of uptake of the print subscription,most of which turned out not to be particularly successful, and I don't know whether it was the wrong time, the wrong product or what but again it was interesting.

We did a study at that time of other journals in education. We looked at about 15 of them of ones that are relatively well known and what we discovered was other than the association journals where you get the journal as a part of membership most of them were experiencing declining subscription rates. So, I think there is sort of a general trend. It is very different online. Of course, we are not charging online at this point. So, the readership continues to rise. We have got lots of graduate students. We have got lots of undergraduate students which is really very interesting.

The other data we do have which really comes to us from Blackwell because they do our print journal is they now have better data on downloads of our institutional subscription, and although those are rising and again those are primarily on college campuses, those are rising, they are not rising nearly as fast as our online usage is rising. So, an article that might come out and in the first week have 4000 readers online at the end of 2 years in the institutional subscription if there are 100 or 150 downloads that would be a lot.

So, it is really a very different kind of scale almost to even look at these.

DR. KLAHR: I am a new subscriber to your journal within the last hour.

DR. NATRIELLO: Did you elect to become a reviewer as well?

DR. KLAHR: No, and I didn't ask for e-mail either but what I didn't get a sense of since I was so fascinated with your interface, I might have missed this in your talk but I typed in something like elementary math education and I got 11 hits ranging from a recent one to something in 1930. So, my question is that is a pretty selective set of 11, and I don't have a sense of what criteria do you use when you decide what to put on this. This is an overwhelming amount of material that you could have put on, and you only put on a very small amount. So, what is that process?

DR. NATRIELLO: It could be that we only had 1 articles that actually fit that particular search. In terms of what we are putting online our plan is to put everything the journal has ever published online and everything should be in the catalog but just because it is in the catalog doesn't mean for example that we have an abstract that also could have been searched or the full text that could have been searched at this point.

So, there are probably a lot of other things that once we get the full text versions of those online would also come up in the search and fill out the collection but at this point we are not keeping anything off in a deliberate way. It is just that we haven't gotten everything digitized yet.

DR. DICKERSIN: Are any journals using what you have developed yet or is it in the demo stage?

DR. WILLINSKY: There are a number of journals using it. In fact I was telling Gary, we heard from the Mongolian Academy of Sciences, and we only actually hear when they have a problem. Isn't that so typical? Never a thanks. So, they are currently installing it but the two best journals that have been using it are probability; in fact we are working with a whole group on probability journals, electronic journal probability and electronic communications of probability. We also have a, I mean there are a number of them that are implementing it, but one I would also point to is post-colonial text which has editors in Sri Lanka, India, South Africa and the West Indies and Canada and that has been a real experiment and also we have an editor in Cameroon which didn't work out because the access to technology is an issue, but otherwise we have been successful in bringing new editors and new communities into the editing process and so our system, I will just go back to Brian's question, our system has been designed on two principles. One is it is free and the other is it reduces the amount of work that is normally involved in editing, by automating many of the processes, by having e-mail come up to remind you that you need to ask in a polite way. We have a thank you button. Did you thank the reviewer? So, we try to reduce as it were some of the barriers for moving from a print journal to the electronic media.

So, if you would like a list of journals that we are participating with I would be happy to provide it.

DR. LAGEMANN: John, you began your comments with kind of a plea for less mediation if I got it right.

DR. WILLINSKY: Yes.

DR. LAGEMANN: And what strikes me is that what both of you have described is just getting more and more information out there in a whole bunch of different ways.

DR. WILLINSKY: But an ability to hone in on it more accurately with better indexing.

DR. LAGEMANN: But that is totally in the hands of the person doing it. It is not the field accumulating information.

DR. WILLINSKY: It is the field accumulating information but the key to overcoming the sense of info glut or overwhelming is the accuracy of the searches and that is just as true for Judy and Hannah in what they were talking about, that what we are going to see is an increase in information. The question is whether we are going to increase access or not, but the key to increasing access for me is more accurate searching so that people are not overwhelmed. I mean I thought 11 was fine, hits in terms of the mathematics one. If you get 100 it is not meaningful. I think we have seen the limits with Google.

DR. LAGEMANN: But the problem with the 11 is how do you know whether you are getting an apple, an orange or an old shoe. I mean one is what was it 1930, you said?

DR. WILLINSKY: Because they are all TCR and what we think we need is more specialized databases. There are probably further terms you could have put in. You put in elementary school, did you?

DR. KLAHR: Elementary science education.

DR. WILLINSKY: So, this idea then that part of it is this honing in and what I think it will lead to in terms of quality is a more exhaustive literature search, that is what we see in our literature is often a large number of articles but not always the most relevant ones and the exhaustiveness and one of my concerns with ISI that wasn't addressed this morning is its narrowness. It was dealing with about 15 percent of the education journals and if we told students that should be the index they use as a guide it is a very misleading aspect and particularly with that overlap or gap that we saw between ERIC and ISI.

So, we are in a state right now where we have overlapping indices with large gaps around them, and we need more sophisticated systems driven by concerns about quality and the accuracy of indexing I think is the biggest insurance against a sense of just being overwhelmed.

DR. TOBIN: I would just like to push you to follow up a little bit more on that. What about what Judy described in terms of quality indicators for instance? How do you feel about that and do you have such a thing?

DR. WILLINSKY: I know I am being so democratic that you think I will reject that, and I am concerned about a democratic approach to knowledge or democratization of knowledge and I think both our systems speak to that, but the idea of beginning to index articles by peer review is the first step which we don't have any kind of system right now that does that.

The idea that we can then do background in terms of TCR versus, I mean ultimately you could imagine asking a system show me the top five cited articles in this field; show me articles that have been done by faculty members at this institution because the granularity of that indexing allows for that.

Now, will the public know? Well, the public does have a sense of those kind of criteria. We are just doing a comparison. One of the studies we are doing is a comparison of hits versus ISI. The problem with ISI is very few people have access to it. It cost 150 odd thousand dollars a year to subscribe to ISI. We take it for granted. Many institutions don't have it.

So, how does it compare, the kind of work that Gary has been doing, for example, around hits and access and downloads; how does that compare to an ISI ranking? And so we need to provide guides to the public that do rough measures of quality but at least some kind of assurance.

DR. SEBBA: I think there is some confusion, John between whether you can see systematic review as a form of mediation. I agree that the other system I described, the research summaries on the web I think are a form of mediation for which I don't apologize because I think that we would argue, I don't think it is necessarily undemocratic in that it is completely transparent, but I think it needs to be said that we responded to requests from people like teachers who would argue that they simply don't have the time to read that original work although obviously if they are pursuing a higher degree of something they would be reading that full work, but I just want to come back to the point you made about mediation.

I see mediation as people or processes between the original research findings and the practice or policy which enable that to be somehow simplified and in simplification I agree with you there are all sorts of dangers. I don't think systematic reviewing does that. Systematic reviewing is a form of research in itself, whether or not it is medical systematic reviewing or educational systematic reviewing. It is actually taking a field of research and reviewing it in a way which is transparent which applies quality indicators but which are transparent. So, somebody else can come in and reapply them and say, "No, you have got it wrong," and in fact we encourage through the system of systematic reviewing, we encourage other people to challenge the protocol on that review and the inclusion criteria and all the rest of it. So, it is completely transparent. That was one of the purposes of that particular system is to invite challenge through that transparency, but I do think it is a form of research. It is not a form of mediation.

DR. WILLINSKY: And you amy well provide access to the original articles as well.

DR. SEBBA: Oh, absolutely.

DR. WILLINSKY: But I think if you look at the United States' example in terms of the approach of No Child Left Behind and the emphasis on scientifically based research and the concern with evidence-based practice is that it will replace and not only reduce funding for all other kinds of research but make all other kinds of research invisible as it is seen as the only reliable guide.

If we imagine a universe in which there is the Cochran-Campbell kinds of meta analysis as well as access to a wide range of research that does not fall within the parameters that you have established for accepting them then I am totally happy. You are absolutely right. I was being glib in the sense that the work that you are doing is not simply a simplification or a mediation; it is in fact a form of research, but the policy implications --

DR. SEBBA: There are no parameters.

DR. WILLINSKY: Sorry?

DR. SEBBA: There are no parameters.

DR. WILLINSKY: On what studies are used in your analysis?

DR. SEBBA: There are no parameters.

DR. WILLINSKY: I thought I saw parameters. All right, okay, then I misspoke, but at any rate the concern that I am hearing from researchers in the United States is that the evidence basis, what counts is evidence. I mean the notion of evidence-based research, did anyone ever do any research that even the work on Newton, I checked the letters, I mean that evidence base is there and that definition of science is possible because there is no access and what I am arguing for that if we increase access to the entire body of research then as Frederick Erickson has argued there are parents who would be just as concerned about the experience of children in classrooms as they would be, their child in a classroom as they would be about the test scores at the end of the year and so this idea of what is at stake in terms of the evidence-based debate is a serious one in terms of the larger research question.

DR. DE HAAN: This is Bob DeHaan. I would like to ask both of you to think about or talk about whether the amazon.com model for reviews might work, that is just allowing any reader to provide reviews.

DR. NATRIELLO: As a substitute?

DR. DE HAAN: Not necessarily as a substitute but as an addition or as a --

DR. NATRIELLO: We, in fact, let people comment on articles and so, you know, any registered member of the site can write at length about an article.

In fact, if you go look at my book review of the Committee's book, Reba Page has a comment on that book review that is almost as long as the book review itself. Now, if you know who she is and you value her opinion you know how to place some value on that. Amazon does a little bit better than we do because they actually let people say a little bit about who they are and why you should trust them and maybe they have put together a list of their favorites and things like that.

So, I think there are lots of folks who are struggling with alternatives to peer review or complements to peer review simply because that is a process that is so dependent upon voluntary labor that as you try to ramp up the publishing effort and as you try to decrease the time between submission and publication you bump up against the fact that you don't have the resources to really do it the way you want to do it.

So, you can could imagine journals that would take very different tacks that would have review follow publication. Now, that is not where we are heading because we think there are some problems there. There are, also, folks who are experimenting with systems to try to automate the quality control process and that has got another whole series of problems but I think one of the things that sits in the middle of a lot of the discussion today particularly about quality is the peer review process and those of us who have edited know that it is not without problems and that those problems are very substantial and that you can try to cross one bias with another bias but in some fields you can't find that other bias. All you have got is the one bias. In fact a lot of the crisis in educational research publishing has to do with the political cast of educational research journals and the fact that an entire segment of the policy-making class has decided that our political cast does not admit a wide range of materials on a fair and balanced platform.

So, it is a huge set of problems and the kind of system you are talking about which would be more open and more transparent might solve that problem and probably create a lot of others.

DR. DE HAAN: It would undoubtedly create a lot of others but since, I mean you can do sort of in numbers what you were trying to do by having a large pool of potential known, that is known to you reviewers. In this case if you have 30 reviews instead of three or four the reader can get a pretty good idea.

DR. WILLINSKY: Our system also has the comment feature. We don't have the stars, and we don't say that this person also read the following five articles because of civil liberties issues. I thought I would make a small pitch.

DR. TOBIN: I think we have time for another question maybe and then we will move on.

Anyone else?

Okay, let us thank the panelists.

(Applause.)