DR. FLODEN: So, this is part of the larger question of how journals can help to make connections between the individual studies that people are publishing and a range of other things, practice, other studies in the same area, theoretical pieces and so now we are shifting into the question of coherence and what sort of things the journals are doing to make it easier for people to connect the individual pieces that get published to one another in some larger body of work and let us start with the question of access to data and reanalysis. What are the journals, what are your journals doing or thinking about or your organizations doing or thinking about in terms of making it easier for people to get access to the data on which one piece is published for possible reanalysis or possible integration into a look at some larger question?

DR. FIREBAUGH: I guess I would like to start off because I was unsuccessful as editor. This actually relates to quality I think as much as coherence but I was struck by a question which someone raised in the previous session talking about impact scores and citation rates and so on that in some cases anyway that the bad articles get cited quite a bit and perhaps it is because they are bad articles and maybe the citations are positive but maybe they are negative. You can't tell from the statistics whether they are negative citations or positive citations, and I must confess that as editor of ASR probably the thing that gave me the biggest worry was that I was publishing articles with significant problems, errors. I mean there are not too many things that keep me awake at night but a few cases did and you see these wild claims sometimes where somebody takes some good journal and goes through and 50 percent of them had serious problems in them and so on but often methodological problems, and we can talk about trying to improve the process kind of by working harder, you know getting better editors, getting better reviewers but I really think that I know as editor I was putting in more than 40 hours a week as editor, and I had my own research agenda, and I was, also, teaching some.

So, it is difficult to sort of look at it individualistically. I think you have to think in terms of structural solutions and so, when I was invited to be on this panel I saw that we were going to be talking about quality and coherence. The one structural solution which I tried to push unsuccessfully through the publication committee of the American Sociological Association was that ASR have a data sharing policy similar to the American Economic Review which is basically that something which is published in the American Economic Review I can actually xerox it, if I can find it. Here is their policy. It is the policy of the American Economic Review to publish papers only if the data used in the analysis are clearly and precisely documented and, and this is the controversial part for the ASA anyway, are readily available to any researcher for purposes of replication.

Details of the computation sufficient to permit replication must be provided. The editor should be notified at the time of submission if the data used in the paper are proprietary or if for some other reason the above requirements cannot be met. I really think that if more journals had policies like that, you know, first of all I think authors would be a little bit more careful, but secondly I think that again you can't catch all the mistakes but if you have something out there that hopefully then if you have sort of immediate replication or pretty quick replication the errors would be caught and I was told by someone and I think it is the American Political Science Review that actually has a replication section. There is some political science journal that has a replication section where they actually invite you to do replications of papers that they have previously published, and structurally I think that is the major thing we could do to diminish the amount of problems in our journals.

DR. FLODEN: And why weren't you able to get that through?

DR. FIREBAUGH: Interestingly enough it is the qualitative people who objected most strenuously. They said,"It is proprietary data," and even though you know you put in some weasel words I think as editor you would have the option to exempt certain types of data, some of the people with quantitative data sets that they had collected, they wanted a longer lead time to get all the articles out of them they could before somebody else would have access to a data set but even there I mean I think it was really excuses because there are ways around that. You could say if you listen carefully to the American Economic Review all you need to do is not give the other person your whole data set. You need to give the person enough of your data set to be able to replicate your results including maybe some of your codes, and most of the errors I think if you want to call them errors in sociology in American Sociological Review probably have more to do with they are not blatant errors; I mean I am sure there are some blatant errors in terms of somebody coded something wrong or they didn't understand their printout and they misinterpreted, but they probably have a lot more to do with sort of sensitivity and how sensitive the results are and some of you may be familiar with a kind of a classic paper called Let's Take the Con Out of Econometrics by I think it is a guy named Leemer, an econometrician named Leemer and he made the point that you can report sort of significance levels and so forth and maybe for large data sets like NELS(?) it is not a problem. For smaller data sets that I think you use in education and we use in sociology sometimes your results can be very sensitive to whether you put this variable in or whether you put that variable in and so on and so forth and if you have a replication you know that somebody is looking over your shoulder I think maybe you are a little more temperate in the claims you make about your results.

DR. COUGHLIN: I am shocked. In basic biology journals as well as biomedical and applied journals it is an absolute industry standard over journals A, B and C, the tiers that sufficient data be published so that people can not only reproduce, evaluate and advance the published work, if the data set is too large or in a conformation that is difficult to put in print it has to be housed in a ratified housing depository. Most of them are approaching databases housed by the government, by the National Institutes of Health. Some journals go further. We do. We have to house it in house. It is an incredibly expensive endeavor, electronically to store these things in perpetuity but I have weekly fights with pharmaceutical companies and technology transfer firms that didn't have time to file the patent prior to publication and readers want the data and they have to give it to them, and the punitive measure we inflict if they don't is we ban them from publishing but not reviewing because we still need them as referees.

DR. EMILHOVICH: I want to give the take from the qualitative side because as a social scientist, yes, I think it is very important that we find ways to ensure the fidelity, the authenticity, the integrity of the data and also ways that people can look at a phenomenon. The problem is in the kind of work that is done in the anthropological world and other areas where field-based methods are used is that you are really studying complex phenomena in its situation, in its context and it can't be decontextualized so that you could say, "Okay, I am going to just pick it up and just move everything over to another context."

The most famous case of course that came out was when Margaret Mead's work was done around the 1920s in Samoa and then another person, Derek Freeman came along some 30 years later and said, "Margaret Mead got it all wrong. This is not the way the women thought about sexuality at all, except that when she was interviewing them, they were young girls and where probably talking in a different way," and when he went back he was talking to middle-aged mothers and grandmothers, you know, who weren't anxious to talk about whether or not they had lots and lots of sex.

So, it is a different context and so the replicability issue is extraordinarily difficult in that kind of research. The second problem is when you try to make your data available it is not that you expect somebody to replicate what you found. You are dealing with sort of understanding a culture or understanding a phenomenon and the other piece is that much of the information you get is confidential.

I have never in the 20-some years I have done ethnography, I have never, ever done one where I didn't find out something that people did not want me to know or that it would have been terribly compromising to reveal it to some larger group, and the main asset you have as the qualitative researcher is being able to go in and that people will tell you things because they trust you not to reveal it, and if you were to identify your sites, for example, doing school-based research is extraordinarily difficult because you have got principals who really want to get across the message that there are things going terribly wrong in the schools, but you have got to protect the identity of the school.

If you release that identity in order to make a point about okay, somebody else go into this school and replicate or see what I saw, you have now damaged that principal's reputation or those teachers' reputation and then they are just not going to let you back in.

So, it is a much more complicated issue when you are not working with quantitative data, when your data are really based on observations, when they are based with interviews, when they are based with document records, when you look at private letters, private correspondence. It raises those kinds of issues that are extraordinarily difficult and you just can't say, "All right, I am going to turn over my data set." I am just going to erase all the names and then you go in and replicate the same thing at this setting.

DR. COUGHLIN: No, certainly there is a degree of freedom within the advancement. Obviously genetics is very quantitative but if you are doing a limited study on a small family even blinding that information is moot because everyone can figure it out, so that there is a degree of blinding and degrees of freedom within the journal mandate to relinquish all .

DR. VANDEN BOS: In terms of data sharing it has been the policy of the American Psychological Association for something like 25 years that to publish in an APA journal you have to keep your data available and give it to competent reviewers, competent researchers for up to 5 years for purposes of replication and verification of your findings. The amount of data sharing that occurs is probably 1/10 of 1 percent at best.

It is a hot topic. It is a difficult topic. I have participated in two federal conferences having to do with how do we better encourage data sharing and one international one, and I can't say the initiatives have achieved too much in my opinion.

People want to hang onto their data. They want to be able to milk it. That is fine. I think that there should be some limit after which and whether that is 3 years or 5 years or whatnot, that they should make it more generally available for other people to look at it and to ask different questions of the data.

Right now in the APA Ethics Code it says, "APA members share their data for purposes of replication, and any other things are not required. " So, if you want to use my data set to ask a different question I am not obligated to give it to you. That was hotly debated in the last revision. That went into effect just this past June.

I don't think that journal policies, our journal policy certainly has not encouraged much data sharing. I don't think that professional ethical codes are going to do a lot for data sharing, but I think that there is a variety of things that are on the horizon that are going to do a lot for data sharing, and part of what we need to do is to both make it technically possible and make it a professional standard that you do it and I am referring to Internet access to data.

One of the things that we are developing right now for the APA journals, you know our journals are all print journals and about 6 years back we digitized all of them and the database of articles for us is now around 33,000 and for the 50 journals that we cover we currently go back to 1988 and as of January 1, we will go back to 1985, and by the end of next year we will go back to Issue 1, Volume 1 of all APA journals, so that we have a large electronic database of full text articles that are available for people to use.

The thing that we are working on right now is to capture the value, the opportunity, the unique features of electronic access to make more stuff available, to make information available that is not economically feasible to do with a print product and so we are developing and I have got a few enthusiastic editors who absolutely hate me because I say things like, "We are someplace between alpha and beta testing of this and for two articles you publish next year we will let you put supplemental electronic only data in the record," and we are developing the procedures for that so that what we are trying to set up is that if yo have got video clips, if you have got a manipulable data set, if you have got a rating system that you use to rate a behavior that you have observed, that you can actually put it in an electronic training manual where you say, you know, there are seven levels of this; here is the best example of it and you describe it in words and click her to see three different examples of it and then level 6 is a little bit less and you try to use the best words that you can to describe this a little bit lower and click here for three examples of this so that people can have instant access to be able to train themselves on your material. They cannot just use the words which have always been printed in journals but they can actually see what you call a seven and a six and a five and a four, putting data up, putting various user's manuals up. We are still debating about putting up software for running various things but the Internet provides this kind of wonderful opportunity to make much more material available. We are putting it on an open web site and then in the article itself there is a statement to URL that we, not the author,not some individual researcher, not some individual university that might later decide that they are not going to support it anymore, and it is located on a URL of a machine that APA maintains that APA sort of says, "We are going to maintain forever."

I hope we maintain it for at least 50 years. After that I will count on the historians to argue for why we need to keep it available for another 500 years but that kind of opportunity where in the print product you have links to a variety of supplemental stuff that either it is technically impossible to put into a print journal or that it is just economically unfeasible because it is a huge table that would take up 20 or 30 pages, that you would have to cut it up and tape it on the wall to be able to utilize.

So, I think that the Internet is providing us with a variety of tools that we can use to enhance making data sharing a reality and that is what is going to make it, not journal policies or association ethics.

DR. MC KEOWN: A couple of comments. One, I think that those things sound great, but I also think that there is a lot more that can be done in printed journal articles. I mean the number of articles that I see both as an editor and a reviewer that don't have examples in of either the questions that were asked in some measurement tool or dialogue or discussion if it is talking about some classroom things that were going on is just amazing and I am always saying that it seems quantitative and qualitative researchers have that same problem for different reasons. The quantitative people seem to think well, you have got the numbers. So, you don't need anything else, and the qualitative researchers seem to say, "Well, I did a coding system and based on three phases," but they don't give the examples of it, and we are constantly saying, "You have got to give examples," and I think that that is something that both reviewers and editors need to monitor a lot more and we have got to have examples of what goes into your evaluation of what happened as well as what happened if you are doing things in classrooms or with human beings, you know, what were the interactions.

The other thing I want to say is I think even more important or at least as important as data sharing as far as coherence is how to connect things that are like pieces of research, like research ideas. There are several versions of that. One is similar research that is in other fields, for instance in education and in psychology or linguistics. There might be similar questions being asked and research being done that you are not aware of because you are not reading those other journals, perhaps, or even in the same field but it happened to get published in another journal that you weren't aware of and again this is because there are so many journals out there. How do we do that? How do we keep track of that so that we don't find a year later, oh my gosh, this question I was so interested in somebody else did a very similar thing, and if I had known that it would have been very helpful. I mean yes, we can all read a little more widely, and I think we know lots of examples of people who write more than they read, but I think that that is maybe a simpler issue, a more basic issue, but a very important one.

DR. VANDEN BOS: I am a publisher and one of the things I publish is journals but as a psychologist and as a researcher and as somebody who advocates around policy issues journals as journals are functionally dead and they have been for 15 years and nobody has noticed it. We still disseminate it in that way and journals are very, very useful, okay? And I value them very much and part of the reason I value them is because I get to throw them in this briefcase and then when I am someplace I get to pull them out and I can read them. It is a very efficient and actually fairly inexpensive way of giving me information.

Now, I know empirically from several different things that I can determine which journals you people will subscribe to and when you will stop subscribing and the critical number is 30 percent. If you are reading more than 30 percent of the articles in any one journal you will continue to get a print copy of it as long as it is reasonably priced.

If in that particular journal at some earlier point in your career you read a lot if now it drops below 30 percent that will pass the threshold for you and you will stop subscribing to it, but the bottom line is a journal is a terrible way to find out what is going on in the world. It is a good way to exchange information, to a core set of people who have very, very common interests so that the questions there are ones that you want to read 60 percent, 70 percent of the articles.

We have quite a long time ago moved away from doing research where we look at journals, what is published in journal A or journal B. That is really irrelevant. What we use is secondary databases such as Psych Info and different databases in different journals that searches across every journal, you know. It searches on the basis of the terms, the questions I want to answer, the topics I am looking at, etc. and it deals with the question of, I mean it ignores the question of where was it published; what is it about. You use the database to get to the references and now because of linking you know in many cases you can move from the abstract to the full text article and so the question of where it was published and in terms of finding it, you know, should become a smaller and smaller and smaller issue because we are not a journal-based science. We are an article-based science, and it is the electronic media that helps us to find all of the relevant articles that come to bear on whatever question or practical issue that we are dealing with.

DR. EMILHOVICH: I want to raise an issue that hasn't been talked about yet because it is really beginning to fascinate me and Margaret hit upon it a little bit when she talked about people could read more widely, but it is also in terms of the time frame .

I have been told and I haven't checked this with people, but some grad students have told me that certain journals will tell them you can't list any citations that are older than 5 years and this is sort of anecdotal. This is one of those qualitative pieces. So, you can just take it with a grain of salt here but what I am finding though and it is one of the ways you can tell you are getting older is that you start to see things that you did when you were in school and then you start seeing it come back.

I am seeing themes in certain education journals that were big issues in the late sixties, early seventies but all that is happening is that they are being repackaged under different labels, but it is the same stuff and you are saying to yourself, okay, what are the advances in the field here if what I am reading is stuff that I read 30 years ago and when you also point out to students that if you went back and you went way back to something that for them is the dawn of time like look at the early 1900s and some of the things that were being published in the 1920s, and it is particularly true in anthropology and ed; there is a whole body of work that was done in the twenties and thirties that was really wonderful work, and it really informs a lot of how you can think about current questions in education and you find out that students, young scholars, grad students and young scholars are totally and completely oblivious to this work. They didn't even know it existed because it is so far back in time that all they are thinking about is something is really old research if it is older than 10 years and it doesn't get cited, and if you don't believe me, go to look at a lot of journals. I have started checking lately, and I have gone to look at some journals and I start looking at the citation dates, not the names, the dates, and you start to see that there is a pattern that nobody is citing stuff much below a certain threshold even when some of that earlier work would be extraordinarily helpful in helping them understand at least the progression of ideas or lack thereof depending on the field or the topic, but they would begin to see that there is a historical continuity about the way people have studied certain issues in education and then ways that these issues keep coming back into the dialogue.

The whole issue around intelligence testing is a good example. I mean that has been up and down starting in the late 1800s all the way through to today and it is the same arguments.

So, I think that that is something that when you talk about being an editor are you sensitive to that; are you really raising those kinds of questions or when you take articles in do you just simply say, "Okay, I am not going to worry about the historical progression of ideas," and that I think is where these electronic databases and AAA is considering that as well is going to be an extraordinarily important kind of thing.

DR. FIREBAUGH: There actually is some evidence on this. Those of you who may be familiar with some work in sociology in science where people have looked at the distribution of citations for particular articles over time and have talked about the half life, you know, the point at which half the citations have been made and the basic conclusion is there is enormous variation across disciplines and I think in general maybe Bridget could say something about this and often in the quote, hard sciences, the citation half life is very short and if you are reading it for the first time in a journal you are behind the times because it has already been circulated.

In fact I had somebody from economics, a friend of mine mentioned the other day, he said, "Well, by the time it comes out in the American Economic Review if you are in that area, and you are important you have already read it because it has already been out on the web and so forth," and just one other comment I think that really is reflected in the immediacy index which people talked about this morning that actually instead of showing the relevance of articles or the timeliness of articles it could actually show the reverse that journals that are slow in publishing, and so they are way behind in the field are getting a high immediacy index.

DR. LIBEN: Actually I wanted to say something very similar because when you are talking about the immediacy index I had written down what about a lastingness index and I have made similar kinds of analyses about citations t the ones that you are mentioning. It seems to me that education people certainly know about the importance of assessment and how assessments end up driving education, and it seems to me that what people need to lobby for is something like the flip of the immediacy index, the lastingness index or whatever as part of SSCI because I do as an editor complain when someone hasn't in fact cited the older stuff.

I think we are constantly reinventing the wheel. Your comment about the cross-disciplinary wheels as well, I mean I just had a call from someone who wants to do a special issue of child development because essentially it is a group of people who got together and realized that there is this early education out there and people don't know about assessment and they want to mention that how you assess something is important and it was like a whole new idea to a couple of very important productive developmental psychologists, and I think that it was a real indication to me that that group of people who is now discovering education doesn't know anything about the education literature and so I do think that -- and your comments also about where you publish, I mean somehow, I don't know if this room full of people can somehow move on and have this new vision but one thing is really maybe getting journals together. You know it is not just what you tell your junior colleagues about publishing in your disciplinary journals and that those are more important and they count more and they are higher prestige but maybe what we need to do is to try to figure out ways to collaborate some in simultaneously publishing on the same topic in an ARA journal and an APA journal and an SRCD journal so that there is like a linkage about a topic where people are actually talking, and I don't know maybe I am just being, I had too much to drink or something but I do think that the disciplinary issue where we talk about multi-disciplinarily(?) are funding agencies are all making this big point but until people really do something to figure out a way to bring us together and not just take a little tiny bit and say, "I am a developmental psychologist, and I know it applies to education," without really working together and maybe some kind of journal-related work would be the way to do it.

I think we are just going to constantly be reinventing the wheel historically and disciplinarily or whatever the right word would be.

DR. COUGHLIN: If I could just briefly address Barbara's immediacy index, you are right, we don't pay much attention to it if at all, one because it is a bit artificial. If something is published at the last quarter of the year it doesn't have any chance of getting cited and it is in the calendar year.

One of our competitor journals which I will name names, Nature, drives its publication roll-out based a little bit on immediacy index. So, 2 years ago when they published part of the mouse genome and part of the human genome as a comparative map they held it for a few months so that it could be published in the early part of the calendar year so that it would have a very hot, we call it the hot index, the Amusi(?) index. We are also not so concerned about the sinkers that get published because while they will have a spike in immediacy they won't have any lasting and even though the half life is very short people will only beat it up so much and then they move on and stop citing it.

What I have heard over the last 15 minutes or so is that there is obviously a need for standardized station of sharing, a codified code of ethics, standardized for presentation whether it be as Gary said in the web in some formatting that can be easily used and repurposed but also that repurpose across journals because it doesn't really matter where you publish so long as it is published on the web.

Something that we currently have growing pains on is with this wide distribution and with intrajournal linking, proper attribution of ideas. I think scientists are particularly used to citing when something is in a printed reference but how do you properly cite when you get something off the web. Maybe you didn't read the whole article but you just went and got the data set or watched the movie clip. It is not exactly plagiarism, but it is not exactly ethical not to cite it and so I think that is a growing pain as we move towards this electronic world and a wider distribution of data sharing which is a positive step; it is just that the policing and attribution is not as easy and I think underdeveloped at least from my vantage point.

DR. FLODEN: So, we have been talking some about data sharing and replication. Would any of your journals publish replications? I mean if somebody took somebody else's data and ran through the data again and found that they agreed with the conclusions that were drawn would that be publishable material?

DR. LIBEN: We have not but I just will say one of my radical suggestions in another context which was a graduate training context was to suggest that master's theses and perhaps undergraduate honors theses or the equivalent be replications and that there in fact be you know like one pagers or half pagers in journals and even good journals that basically report those replications because an awful lot of honors theses and masters not at our institution of course but elsewhere are really not worth the kinds of time and that actually having some of these replications I think would be very good.

I mean in terms of replications of other, again I just got a call about someone who is about to submit in essence a different kind of analysis of, it is the NICHD data set on child care and you know wanted to sort of deal with what are the issues here about potentially using a data set that is now available and that there are alternative analyses and what about different conclusions and so on.

So, I think the answer traditionally we would not and Child Development certainly has not spent its pages on straight replications. This is like the question about bias and no results. The news in a replication is the non-replication not a replication.

DR. EMILHOVICH: I don't think you could publish replications for some of the reasons I described. It is a little bit like saying that you can't step in the same river twice but what I think would be very interesting is an idea that I actually heard from a colleague a long time ago and I participated in a conference at NRC, National Reading Conference around this idea, and this was Judith Green. It was to take a video of a classroom lesson for example, because there is a lot of interest right now around what is effective instruction and have a group of different people do their take on sort of the multiple perspectives concept of what is it that they see in that video and then write from that.

So, they are working off a shared database of a phenomenon but then seeing how different people, what are the lenses that they are using to arrive at their interpretations of what it is that they are seeing whether they are looking it from a more linguistic standpoint, whether they are looking at it from a cognitive standpoint or learning standpoint, whether they are looking at it from the literature on pedagogy, whether they are looking at it as a means of content, delivery of content.

I mean there is a number of takes you could do, and she actually did a session which I participated in at NRC a long time ago. It was another one of these going back to things you have done in the past model and it was very successful because it was fascinating to hear different researchers bring to the table a different way of seeing a very complex phenomenon that we are all trying to understand because it has so much import in terms of how we help students learn better.

DR. MC KEOWN: As far as replication I guess it depends on what you mean by replication, how close a replication as far as AERJ but it gets close to some other issues that we certainly have looked at.

For example, you know if there is some instructional approach that is looked at certainly you want more than one look at it. You don't want an exact replication but the question might be well, okay that only worked because it was in this geographical area or because it was the researchers doing it themselves or something. So, you do want other looks at that.

Then there seems to be a point of no return of how many groups exactly do we have to look at before we say, "Okay, we know that finding," and that certainly has happened where sometimes reviewers will look positively on a piece and we as editors say, "You know this really doesn't advance anything because this really has been done, and this doesn't add to the answer to that question," and then there is also the use of large databases. We have gotten quite a few articles lately from the early childhood kindergarten database. I don't know if that is the one you were talking about, and again, it is you know how many studies do you want to publish on that that are similar questions. I think those are all important issues.

DR. LIBEN: Could I just add on little thing about this videotape classroom issue that I just want to throw out? I think it is a great idea but at least at out institution it would never get through an IRB. So, you could never get approval to have for example a videotape classroom database in essence that could then be looked at by subsequent researchers.

So, when you get signed permission to use such a thing it has to be specific to the kinds of questions that are being asked. So, I really think that while we are looking at electronic changes which I think you know is another session also later and a lot of what Gary said I think is fascinating, and we are doing some of this database stuff, too, but it does bring up all these questions about all this stuff is so available; who is going to pay for in essence these subscriptions.

I think this idea is also a wonderful idea and I am all for it, but we also run into just practical issues just to throw that out.

DR. VANDEN BOS: Around the question of replication, I would say there are many more replications going on than we give credit to and the question does depend on what do you call the replication.

If you are talking about empirically-based practice it sort of mandates that you have replication. The question is how exactly like the last study is the present study and is it 5 percent different or is it 50 percent different and you know to some extent to be able to say something about what is an empirically based appropriate practice to do under particular conditions, you have to have some base of 20, 30, 40 studies all of which in some way are overlapping and triangulate on and answer different pieces of different aspects of the question. So, I mean in terms of absolutely straight, you know, I am using exactly the same methodology with a different sample replication, you know I don't think that there is an awful lot of them, but I think there are an awful lot of replications where this person uses 80 percent of the same methodology as somebody else uses on a different sample and 20 percent of what they do is something different because of a few unique questions that they are interested in and then the next person has a 70 percent overlap with both of those two studies, has their own unique questions. If we didn't have that kind of partially overlapping replication to talk about empirically informed practice would not be realistic.

So, I mean I think there is much more work being published that is a replication. The reanalyses where I just look at your data set and I do my statistical analysis of your data set, I think that is pretty rare.

I advocate data sharing. I have been very unsuccessful getting anybody to share data with me even though I have been very nice. I think I have made 17 requests for data to do analyses on and I have had zero of them given to me and I have found many more, well, you know when I moved from this place to that place I guess that trunk got lost. I can't find this but it is because that graduate student left and they took it with them. You know, the number of reasons why data is not available even as short as 2 years after publication is amazing to me, but I hope that we do get to the final question, the final thing that is on this section in terms of talking about the role of synthesis in terms of developing the cohesiveness and coherence of research in a given area. I think there needs to be more of it.

I think that putting together what we know around the range of questions is a whole approach that some of the same people who do research do it but it is a different endeavor and if we want to see it happen we have to put some financial incentives out there whether it is specific programs; you know, if I were putting together a research program around practice issues I would be putting half of my money into synthesis. I would be asking very specific questions and I would be asking people to pull together the data as you best know it and I would then be having them presenting their meta analysis or qualitative integration and whatnot to a combined, in a setting where there was a combination of researchers and practitioners and part of what I would expect to come out of that would be that the practitioners would talk about the questions that the researcher never addressed as a way of forcing the research community to know about the questions that they are not addressing that have practical significance.

So, I see synthesis and supporting them as being the core of it. I, also, think we are not going to get data sharing unless there is a 3 percent add on and a mandate that requires that in some publication and putting it on the electronic aspect of the web that data is made available and pick your time frame whether it is 3 years after first publication or 5 years. You need to make your data set available, but it costs money for the researcher to make that data set available and you have got to provide some money to support that particular piece.

DR. EMILHOVICH: There is a group that does what is called meta ethnography which does something similar to what you are describing but synthesizing across a body of qualitative research. The topic that they did was around if I remember correctly was around the desegregation studies and I think George Nablitt and a group of people at North Carolina did a whole meta ethnography of that literature. I think that is going to be very, you are going to see a lot more of that particularly as we move towards the anniversary of Brown versus the Board of Education next year is the importance of that particular question and we saw the most telling example of where social and behavioral research made a difference was in the Supreme Court case where the justices did use social and behavioral research.

Much of it was synthesized or empirically based to talk about the compelling social interest of having diverse classrooms in higher education and the lead lawyer for that, Christopher Edley who I just heard present a little while ago talked about how the next big case that is going to happen will be through K-12 and that what we don't have is a lot of research that really talks about it from that perspective of K-12. He said that you could talk about compelling social interest with a group of people at the college level to be in with diverse perspectives. He said that he doesn't know whether or not it could be supported to say is it so important when you have got 4th graders trying to learn how to read, whether they need to be in diverse classrooms, but I think that those are the cases where that kind of synthesis there is a large long body of research on desegregation and the effects of it and I think you are going to see a lot more of those syntheses and I think that if you take those big social questions because they link to policy issues particularly for those of us in education those kinds of syntheses are going to become very critical. So, I think you are right to raise that and say that that should carry more weight in the future.

DR. FIREBAUGH: The Journal of Marriage and Family every decade has something called The Decade in Review and I am not sure how they select the authors and so forth and I am wondering if there are other journals who take inventory every X number of years and what do we know now about the effects of divorce on children or whatever.

DR. LIBEN: This isn't every X years but we are doing a 2005 child development issue. The idea is going to be to look back at the founding which was 1930, and there is going to be a call for papers that says, "Okay, go back to that 1930 year and respond in some way. What do we know now that we didn't know then or are we revisiting exactly the same issues; are our methods doing any" -- I am a little scared of deciding to do this because my fear is that we will decide we really are not much further along but oh, well, that will be toward the end of my editorship. So, who cares?

DR.VANDEN BOS: I just wanted to tell you about one resource that is available. As you may know APA publishes the APA Publication Manual about various style questions. One of the big questions and I believe Bridget mentioned it was how do you cite electronic stuff and when we were doing a revision of the Pub Manual the last time we had a subcommittee that spent a lot of time looking at that. Some of what they came up with is in the Pub Manual but on the APA web site if you go under APA style there is a document. It is about 30 pages long on citing electronic sources, and in that 30 pages I think it describes how to cite something like 45 different electronic kinds of things that you might find.

So, if you are looking for a way to reference electronically accessed stuff there is a document at the APA web site on what we recommend at least to psychologists in terms of how to do it.

DR. FLODEN: So, we have reached the end of this segment.