Roundtable Participant: Kenneth Wong, Co-Editor, Educational Evaluation and Policy Analysis

DR. WONG: Well, since we have a division of labor within our knowledge industry, and our journal has the primary focus on policy issues, how it works, how they are designed and redesigned, and how they are implemented, and what is the knowledge and database in understanding the process of implementation, and as well as the consequences of those policies and practices.

So, to me, the proposal on structured abstracts seems to be a natural setting as far as communicating the research design and methodology and the findings and the consequences to a wider. So, to me, there are three advantages that I see from the EEPA perspective.

One is that it is a very efficient way of assuming that the authors are willing to contribute and think about how they compress their 35 page article into a page and a half, about a 400 word abstract as a relatively efficient way of communicating to a broad audience. And that raises some implications here as well.

And that is that we may be talking about changing the purpose of abstracts in professional journals. And that is that up to now I think the abstract has primarily served an internal function. And we articulate in a condensed way to our colleagues who are interested in contributing to the same journal, so that they can take a quick look at what are the parameters, and what are the issues, and what are the methods and biases attached to the articles that are published in that particular journal.

And one way to do it, a short, quick, efficient way to do it is to identify what the abstracts are, and then see if my research would fit into the mission of the journal. But now with the possibility of the structured abstract, I think we are talking about this interface for the broader policy and practitioner audience. So, that's one possible implication.

The second advantage that I am thinking is that structured abstracts would quickly illuminate what intervention works for whom, under what circumstances, and for which population. And that's a quick way for policymakers, as well as practitioners, including teachers and principals to quickly identify whether this particular type of intervention would be relevant to my daily work.

And as I think about my own work in a particular classroom and a particular setting, are these the kind of populations, for example, if there is an article dealing with LEP, limited English proficient students, and if I'm an LEP or bilingual teacher, I may be able to access that kind of intervention, and the findings and the limitations of those programs right away.

And the third advantage is the methodological visibility of research. We all use certain methods in natural settings, in case there are in-depth interviews from what are on the one side, and then we also have randomized field trials on the other side. And then in between there are a whole bunch of comparative case study, quasi-experimental correlational studies.

And right now I know the EEPA readership is keenly aware of the importance of methodological openness as far as the research is concerned. So, the structured abstract would enable us to get at that right away. That is, it's one of the nine components. I think the climate of accountability, as well as the climate of moving into the what works clearinghouse paradigm I think would complement this kind of structured abstract.

But it does raise some other tensions. I think Norm's question and comment raises some very important questions. I think we have to deal with this tension between diversity in terms of our values and belief systems, and why we are doing what we are doing. And I can see that there are folks who are very committed and passionate about a particular setting, or a particular set of issues.

For example, there are people, including myself, political scientists who are interested in the rules of the game in society, as opposed to particular interventional programs that work or don't work. So, there is a continuum between the institutional, societal concern of how power is structured and distributed, and how do we understand the nature of that power structure.

That may or may not fit in this component called intervention at all. It may fit in the first component, the context or background, but that I take it as a theoretical and intellectual debate. And so, somewhere there we need to juggle and strike the right balance, as my colleagues around the panel have suggested.

DR. FLODEN: Can I just jump in now? This structured abstract seems not entirely clear to me. We were talking about different things at different points for different purposes. Sometimes in the last panel they were talking about a list of keywords, so that when you did the search, if you were looking for randomized field trials, you would find them, because they all be called the same thing, and that would mean the same thing.

That's really quite independent of this. You could have an unstructured abstract, and if the words were all clear when you did the search, you would find the right things. Other times we are talking about having it be a structure, something that you could use for a search, or to help the authors make sure they address all the important points.

And another thing is, and Kenneth was sort of suggesting this I think, that a long enough structured abstract is in the sense a replacement for the paper. It's a summary. It's not really an abstract. It's something that you would give to somebody to say, well, here is what the study is, and here is what they found out. And they would read that as a stand alone piece.

I worry about the last thing the most in that this idea, like when you have journals of abstracts, and all people really look at is the abstract, never going back to the original piece. And one of the reasons I worry about it is related to what Kenneth was saying too about some of the theories that go into things.

And I think about in reading an article and reading the article itself rather than the abstract, one of the things that I'm looking for, and this betrays my background as a philosopher, is what the argument is. So, what's the connection between the research design, the sample, the findings, and then the conclusions that people actually draw?

Because if you read whole articles, and you read them carefully, you will often find that there are questions to be asked at the very least about how people went from the survey they did to the broad conclusions that they draw. And say a two page, or if you want policymakers to read it, a one page version of an article.

It's going to be very hard to evaluate the logic of the argument, and whether you really want to believe the conclusions that people are drawing based on the research that they have done.

So, I see the value of abstracts not in this sort of a substitute for the paper. I would worry about journals of abstracts that invite people to treat the abstract as the whole thing. But I think they serve a very useful purpose in helping people decide whether to read an article, or going back in history for helping people identify things that have addressed similar topics before.

And there, having a list of common terms would be a helpful thing. It might be very difficult to do, but it would be a very helpful thing. And having structure so that people, particularly in hand searches, not so much in machine searches, would be able to quickly go through things and see well, what's this about would be helpful.

And in that, I worry about the suggestion in the paper we heard about in the first session that these things be longer. You know if something is two pages long, it is going to take me a while to read it. One nice thing about a 75 word abstract is I can get through it quickly.

DR. EMIHOVICH: I want to just sort of tie together some of the things the panelists have already started saying, and then ask them to expand on it, because I think they have already touched upon something really interesting.

Ken has talked about of the four journals up here, his most explicitly deals with the issue of interventions, and is most likely hitting policymakers, first of all because policy is in the title, but also because it's a lot of the nature of that research that is recorded there focuses on that. Both Pam and Terri, and to some extent Bob have said that probably practitioners and/or policymakers are not reading their journals, or they wouldn't expect them to be doing it.

So, if we think about this in the context of structured abstracts, what is it that it would take, what is it that you would want those audiences to learn from your journals, and what would be the best way to get that information to them given that they may not be likely to have the time or understanding, or even interest in wanting to read all the articles, but you would hope that they were getting their attention caught by something in those journals that you would want them to be thinking about? How would you do that?

DR. WALTERS: That's assuming that if we were going to be able to reach policymakers and practitioners.

DR. EMIHOVICH: Right, if you wanted to. You may not want to.

DR. FLODEN: What if we don't want to?

DR. EMIHOVICH: Well, that's a good question. Do you want to?

DR. MC CARTY: There is a simple answer for me to that question, and then there is there is the long answer. Yes, in the case of AEQ one of the things that I signaled in the first issue that I edited as editor when I took this on in 2000 was I think we need to do more to bridge the sort of theory/practice divide that still characterizes education research. That we need for our research to make a difference in the world. That's the point of doing the research. So, I do think that's important.

It's one thing among others in this proposal that I think is very valuable, in that it does call our attention, it rivets our attention to that audience, and for those journals where it is appropriate to attend to those audiences.

I also think that abstracts are important, and it's valuable to kind of heighten our conscientious need about the need for a crisp, focused, clear abstract that clearly signals what the article is about. So, I do think that it's important.

But not all articles for example in Anthropology in Education Quarterly are amendable to policy analysis, are amendable to immediate applications. I think Felice referred to knowledge translation, knowledge transfer, very popular concepts right now. That's not the purpose of those, but those are still important forms of inquiry.

If you look in my paper, you will see samples of different abstracts that I just pulled out over the years. Bill may have mentioned John Ogbu. I assume you were referring to that abstract in my paper. Probably one of the most influential collections of articles on minority school achievement that still stands as one of the most important analyses of this area of research, and that one was by John Ogbu. It's a very controversial article, as well as seminal article.

And basically his whole point -- and if you know John Ogbu's work, you know that this comes through in other pieces that he has written -- is that before we can even think about what would be appropriate interventions in practice or policy, we need to understand the root cause of the problems. Ogbu of course locates those problems in the larger socioeconomic context.

I think that's a very important contribution to make. That is, sort of theorizing practice, theorizing policy. Helping us understand those root causes of the problems, the social injustices there, and that type of thing. If we don't do that, the policies and the practices that we jump to are not really touching those systemic structural kinds of problems. And those are very important issues for us to grapple with as educators.

So, there is that type of work, which is not necessarily -- they are important forms of inquiry, they are not necessarily amendable to a uniform kind of format either in the abstract or in the reporting, the article itself. That's a very important point I want to come back to. That's what Norm was talking about, how the abstract can constrain the article.

And in fact, I think it's on page 30 in the Mosteller et al article that they explicitly state that the purpose of the proposal, the structured abstract focuses on the format of the article itself. So, that's where I think we start getting into difficulties. We start down a very slippery slope, and I have real concerns about that.

But let me just go back as one more example, then I'm going to pause again and let somebody else talk for a few minutes. If you look I think on page 3 of my paper, I used an example of an abstract, two authors that were doing research in a community in Africa on healers, and the apprenticeship that occurs between a grandparent and her grandson in the context of learning to heal.

And the contribution of this article, of course it would be just totally fruitless to try to map that kind of content onto anything like this, but the contribution is in helping us understand more fully, sociocultural theories of learning, and how the social and cultural context is tightly interwoven with learning.

And there are applications of that, that would be appropriate for a teacher working in urban schools for example. But it's not an intervention, it's not a direct kind of correlation. It's something that we learn from, and we think about, and maybe it's applicable, and maybe it's not.

I would like to see us get into a little bit more of that discussion of the forms of inquiry that are appropriate for this kind of a model, and other forms of inquiry that are less suited to that model, not saying the abstract is not still very important, but that are not amendable to that kind of container to put the research in.

DR. WONG: May I comment on this? To me, the discipline-based research, and the first the articles that you publish in the anthropology and the sociology journals form the foundation for the intellectual formulation of policy issues. So, to me, I wouldn't want to overemphasize the dichotomy that ours is a policy journal, and theirs is discipline-based. And I think there is a need for us to actually foster our potential clients and users to fully engage in both types of journals.

So, because after all I think a lot of the lenses that we use as policy analysts are grounded in our own discipline and our own disciplinary orientation and biases. So, that's one point that I hope to underscore, and therefore I can see the need that in a way the proposal for the abstract is equally applicable.

That is, that it is relevant to all journals, because after all, policymakers can learn from what anthropologists are doing in a particular setting in understanding the societal norms, and how it shapes the way people behave and act and talk and communicate. So, in a way I think it's very important.

So, my point here is that I see the structured abstract as fixing or improving the supply side of our research findings. And hopefully, in organizing the way we communicate and supply our findings to the larger audience. Then it will change the dynamics of the demand for educational research.

So, in other words, right now maybe one of the reasons why they are not using the your sociological education journal and anthropology journal is because the supply of the information is not readily available. It is possible. I raise it as an empirical question for us to think about ongoing evaluation down the road, that Bill and Ed can think about.

And that is once we change the supply format and the content, then maybe the demand side would be changed as well. So, I'm looking at it from a different perspective.

DR. EMIHOVICH: Anyone want to comment?

DR. WALTERS: In a roundabout way. I want to come back to the question. Sociology as a discipline is very much committed, especially has been recently very much committed to reaching a larger audience. There is a move for public sociology, for example, and something similar going on in anthropology.

The problem is that in the way that policy questions tend to get defined, sociological research by and large is a step or two removed from that. It's more likely to be addressing the rules of the game, to use the terminology that you used before, Ken, and less likely to provide concrete guidance about how to do X, Y, or Z.

And so, when I think about the role of structured abstracts in a sociological journal in general, Sociology of Education in particular, I keep coming back to the fact that the primary readership are scholars. And that the information access and retrieval and summary and so forth needs of the audience of scholars is likely very, very different from an audience of policymakers or practitioners, even if they were reading this Sociology of Education.

So, the Journal of Sociology of Education is a publication of the American Sociological Association, which publishes seven or eight journals. And so is subject to the editorial policies established by the ASA. And the ASA as an association has been addressing this question of how to get sociological research into the hands of a wider audience.

And basically, the solution that we came up with, imperfect though it is, was we launched a new magazine called Context, that was launched in the year 2000. And it's called a magazine, it's not called a journal. And it is intended to be reaching a wider audience. It is modeled very loosely around the Journal of Economic Perspectives, or that was at least an early model in the discussions about the need for this.

And the association has been doing workshops for people on how to get your findings in newspapers, how to write op-ed pieces. How to access other kinds of publication venues beyond the journals. So, I'm not arguing that this is the only way to go. I'm reporting that this is the way the association has been going. And we have not had any kinds of discussions that would suggest that practitioners and policymakers would be regularly reading Sociology of Education.

Actually, when I step back from that and say what could we do to make the work more accessible, I'm not very optimistic that practitioners and policymakers are going to regularly peruse any of these journals. So, another piece of the question is how to get at least bits and pieces of information into their hands that would then take them to select structured abstracts perhaps, because they are not going to be regularly sitting down in front of a search engine and looking for all relevant research on school vouchers.

DR. EMIHOVICH: Before we lose this point and go on to something else, I'm curious, taking your point into consideration, which I think makes sense for you about whose audience that you are trying to reach, does the notion of structured abstracts work for you in the scholarly realm?

DR. WALTERS: I think some form of more structure, more uniformity could work and be quite useful, but it would not be this. It would be much more general. In the context of Sociology of Education this would be a straightjacket, and would not fit well with the vast majority of work that is done.

DR. FLODEN: Could you say a little bit more about that? Because I noticed your comment too about it would be fruitless to try to put the article you talked about into this. It seems like with a lot of these things, it wouldn't be a big stretch. If you wanted to talk about the setting, wouldn't be the setting be appropriate for this article, or the research population, or the purpose, or the background?

The one up here on the list that seems the most problematic is the intervention, which suggests a particular focus of research. But most of the most of rest of them are pretty general. What sort of a straightjacket is this?

DR. WALTERS: Well, actually even just the language of population participants and subjects assumes individuals as the unit of analysis, and much sociological research does not take individuals as the unit of analysis. Intervention program practice obviously privileges a form of research that sociologists very rarely do.

Data collection and analysis also is too narrowly defined. I mean it could be relatively easily reframed to be a larger issue and theoretical questions. But in sociological research in my view -- I'm going to speak for myself, not for the discipline as a whole -- the coin of the realm is speaking to big theoretical issues.

So, the best articles, the most compelling articles are going to start off with the sort of big, theoretical issue at stake, then move to a specific research question. And then tell you something about evidence, something about how evidence was evaluated, perhaps something about major findings, and maybe something about implications.

So, you could jiggle this around. You could strike certain categories, you could rename others, but that's coming up with a different kind of template.

DR. FLODEN: Not so different. It seems to me when I look at this and I think about the articles that I publish, the one thing that is missing is something about what sort of the structure of the argument is. And if you getting from something to the conclusion, what's a synopsis of the steps that get you from evidence to conclusions? And that's missing from this.

It's sort of implicit in this that you already understand that, that it's a research design where the logic is commonly understand, and that leans things in a particular way. But with some rephrasing of those, and something more about a synopsis of the argument, it seems like that would be something that would be quite applicable across a wide range of different genres of work.

DR. WALTERS: I think I said, or at least I hope I said in my short comments that I think from the perspective of sociology as a discipline and sociology of education as a subdiscipline within it, that some greater uniformity in abstracts makes a lot of sense to me. And it would benefit the current primary audience of scholars, as well as other potential audiences. But I think this particular model of a structured abstract is too limiting and implicitly privileges some forms of research over others.

Just one other thought along those lines. Another thing that is problematic for sociology, again going back to an earlier comment about how sociologists view education as a fundamentally social institution, and we're almost always looking at processes that unfold and complicate a social context, it becomes extremely difficult to summarize findings in a sentence or two. It's just really, really hard.

So, that's not to say it would be impossible to do a better job of abstracts, but that's just something that has to be reckoned with in any proposal.

DR. MC CARTY: I just want to say that I do think abstracts are important. I'm emphasizing that again. And I want to, also just to go back to a note I took when Bill made his presentation this morning, that this format may be appropriate for those kinds of areas of inquiry and disciplinary orientations that do fit that structured abstracts make retrieval easier.

In other words, as I understood what you were saying this morning, and I think both of you said this, that this may be appropriate for some forms of inquiry, and not so much for others. I think in the case of this particular format, that this structure seems particularly well suited for experimental types of designs where you have terms that are very well defined. You have variables that can be, and that are thoroughly operationalized. Hypotheses are amenable to testing or intervention.

But as I said, not all important inquiry is of that type. And to force all educational research into that mold, it does privilege that type of research, and in my view it has the potential, and this is the real danger, of censoring new knowledge and creative insights, because it also forces the article itself, hence the research, into that mold.

At the same time, as an editor, there are things that I'm concerned get communicated in an abstract. And I work very closely with authors once we accept a manuscript, to make sure that those elements are there in an abstract. I hope we are improving on this. We have a lot of room for improvement I'm sure. And again, I think that this has called certainly my attention to the need for further refinement.

But my questions to authors are what can we learn that is new about this? What's important about this work? What is anthropological about it? Why should it be in a journal that has anthropology in its title versus Harvard Education Review or something like that?

So, there are important considerations that an editor and an author do and should make in terms of what gets communicated in an abstract. But for the field that I represent, this particular format would be very limiting, and it would basically transform the kinds of problems that we are interested in investigating, limiting them to a very narrow kind of few. And I think that's not what we want to do.

DR. EMIHOVICH: One of the things that struck me listening to people here, and Bob you mentioned it, we are missing the sense of a structure of what led people to these conclusions, sort of how did they get there. And I'm wondering if we think about these abstracts, the purpose of structured abstracts is in one sense, to get information out to people that can use it for evidence-based decision-making. That's one purpose. It's not the only purpose perhaps, but that's one.

What is it that we would want to see in one of these abstracts, without getting into the question of length, or turning it into another article, that would help us understand the researcher's own decision-making process as he or she went through the process of creating this study? Because Felice had gone back and talked about the history of science, and sort of the way abstracts were use around the 1920s.

I had done some work of my own in looking at the language of how people used to report out their findings. And one of the things that disappeared, that was most interesting is that there was a lot more of scientists talking about dead ends, lost opportunities, things that didn't work out, reasons why it didn't work out.

All of that has been washed out, regardless of what the format is, of how you report out the information. We never get any insight into what dead ends that people got to, conceptual breakthroughs. And they did all this in the process of doing the research itself. It was not something they knew in advance. It is what they discovered, not just the finding of what they were trying to study, but their relevations or their understanding of the entire research act itself, and how they thought differently about research.

All of that has been washed out of any journal I have ever read. And as a result, we have this picture of social scientists or scientists, whether we are talking about the hard sciences or social science, is these omniscient knowers. That they just came to the research cold. They followed a series of steps, and then they produced these outcomes.

But we don't ever see the uncertainty, the sort of places where the research failed. That's why we don't report no results any more, when actually it probably would be an interesting thing to know why this didn't work at all. And I just wonder what your thoughts are about that, and also for the audience to consider as we open up the discussion. Could we design an abstract form that would allow a place for that sort of information to seep through?

DR. FLODEN: No. That all seems like it would be interesting stuff. To try to put that in addition to everything else in the abstract seems difficult. And you raise another issue in talking about those sort of dead ends. It seems separate from the abstracts. But one of the things that is something else that people would advocate in terms of being able to look at the literature as a whole is things that field trials.

A registry of field trials for whenever people start a study they let people know that they are starting it, so you know who is working on things, and you can keep track of the ones where people give up, because they don't find anything or something like that, rather than only looking at the ones that get published, because those are the ones that have positive results.

This way of writing that puts people in white coats, omniscient, the sort of rational reconstruction of the study, rather than the things that you actually did is a characteristic of social science research in general. Joe Gusfield(?), the sociologist, has interesting things on drunk driving research, and the ways in which people choose language to make you give a different sense of what it is that was found, depending on the terms you use to define what you are looking.

He uses social drinkers, as opposed to drunk drivers. For example, he is conjuring up two very different images that might have the same blood alcohol content as a way in which the language that we use affects us. And abstracts I think naturally sort of tend towards that, of trying to encapsulate this, make it seem like it all makes sense. But I don't know that we can get around that.

DR. WONG: I think the abstract is lying right in between the reader's level of curiosity and the full article. So, maybe one way to reconsider this question that we have been talking about is that what would be the right way of communicating to the potential users, readers, so that they would then fully engage in reading the full article? And the final audience certainties, and the limitations in design assumptions of the article.

So, the question that I have then is does the structured abstract perform that function well, the way it is proposed right there? And I would like to hear what my colleagues would have to say. From my perspective, my sense is that for the EEPA audience, it does provide that potential incentive for them to find out, because EEPA does, as I mentioned several times, we deal with interventions and policy design and redesign, things like that.

But even there I am not truly satisfied with three issues. One is the lack of emphasis on theoretical underpinnings of the research. The second is the lack of focus or lack of emphasis on the importance of competing paradigms or rival hypotheses that Campbell talked about back in the 1960s I guess.

And the third is in the findings and results parts, it might sweep all of those things that Catherine was talking about under the table, and not allow the researchers to really be open and talk about the limitations and the uncertainties and the missing values, and things like that in the results.

So, those are the three concerns that I have within the EEPA boundary. But my larger question, is this the right kind of tool to engage our readers' curiosity so that they would go to the full article?

DR. EMIHOVICH: That's a great question.

DR. WALTERS: And I just want to follow-up on that. Listening to Ken's comments reminded me that I think for a scholarly audience, the implicit purpose of abstracts anyhow is to draw one into reading the full article. As a matter of fact, that's part of the instructions that the ASA has for authors of manuscripts submitted to all the journals. There is very little guidance in terms of the actual format.

I reproduced it here. It says the abstract is a written invitation to your readers to read you entire article. And that's the way I have always implicitly thought of an abstract. I haven't thought of it as a stand alone piece that would substitute for reading the article, but yet at least implicitly that seems to be the purpose for the practitioner's access to it. So, I don't know exactly how to sift through that.

DR. FLODEN: The one problem with that is if everybody were successful in writing abstracts then, we would have to read everything, because everybody would be drawn into every article.

DR. WONG: But different people have different interests.

DR. FLODEN: That's a different thing. Rather than drawing in, it's to give the reader a sense of what they would get if they were to read the full article. And if it's of interest to them, then they would read it. And I think one of the advantages of having abstracts for the editors working with authors is it is a way to work with the author to help them make clearer in the paper itself, what it is that they have to say, what is important about it, what's new, what's anthropological.

Because it's hard to write an abstract. And as someone said in the earlier panel, you should start with writing the abstract. That doesn't seem quite right. But writing the abstract can be something to make you at least rethink, well, what is central here? And to be able to go back to the paper then and make sure that in the paper itself, you have brought out what is central in the article, as you have come to realize it through the process of writing the abstract.

DR. MC CARTY: I think Pamela expressed my sense of that too. And again, just reiterating that there are things that an editor looks for in an abstract. It's not a haphazard process. It does need to draw readers in. Not everybody is going to be interested in the Lau(?) healers in Africa.

So, it's not that it's just this amorphous kind of arbitrary thing. There are certain things that one expects to see as a reader and as an editor in an abstract. They may be different though for different foci of inquiry and complexity and all of those kinds of things.

DR. EMIHOVICH: So, you know Bob's question reminds me of a comment one of my old professors in anthropology used to say, when he said, it's an interesting question. Why doesn't everybody come to know everything? And the answer is because there are constraints that are set up through the social structures that allow people to have access to information under certain conditions and in certain venues.

And so, everybody doesn't get to know everything. You get to know what it is that you are expected or supposed to know given the constraints of where you are. And it's hard to break free of those.

So, am I hearing correctly that even if we don't want to look at the format of the structured abstract in quite as strong as saying perhaps it's a straightjacket. But that any abstract itself, regardless of the format we are choosing, is by its very nature a constraint on what information people are going to get independent of actually going and reading the full article.

So, what we are really debating is not that there is going to be a constraint, but what is the nature of that constraint?

DR. WALTERS: And I guess I want to ask us to consider how people come to identify articles or abstracts to look at, and then perhaps articles to read following them. I revert to my own experiences in the absence of other empirical data. So, I spend a lot of time sitting in front my computer doing various kinds of searches through Academic Search Premiere and JSTORE and so forth.

And I'm not sure that the abstracts play a particularly important role in what gets called up in my list of retrieved articles when I'm searching for something. Embedded words turn out to be useful. Then the abstract becomes a quick way for me to do a little check to see whether this is at all relevant to what I'm interested or not. And so, it's an intermediate step, it's not the first step.

DR. WONG: I think there are economies of scale and the efficiency question here as well. That is, if we do have the structured abstracts in place, it is possible that one could spend on hour and maximize the 60 minutes to identify maybe 100 structured abstracts. And that would allow the potential users to maximize the limited amount of time that they may have to get the greatest number of possible findings that would fit their interests.

As opposed to right now where they have 60 minutes, but they might have to rely on the other search engines, and do a more limited yield, so to speak. And that raises a larger question in my mind, and that is the way I look at it is that here is a tool, and it is not going to be able to resolve a lot of the fundamental tensions that we have within the social sciences, because it's such a diverse field.

So, I think what I see the structured abstract as is, here is a middle ground. From what I'm hearing is that my colleagues are not completely opposed to the notion of some kind of an organized abstract. The second is then within the nine components, there may be four that people seem to feel comfortable with, four or five. And then there are four or five that are less applicable to their particular journals. But then we can either broaden the definition, or completely delete or eliminate them from the proposal.

And then the third step, it seems to me is that right now we just lack any evidence, even in the medical field to tell us exactly how structured abstracts are going to broaden access or change the behavior of the users. And that is a fundamental question that I have. Without knowing exactly how and to what extent structured abstracts have actually bridged that gap as Bill and Ed were just talking about.

The aim here is to bridge the gap, whatever the practitioners and researchers or non-researchers or non-journal editors, to bridge the gap. And we don't quite have the data to inform us exactly how large the gap. We know that there is a gap, and even how large the gap is, what's the baseline on that, how do we know that that gap has been reduced in those journals that have used the structured abstract.

DR. MC CARTY: May I just say a couple of things? As another little part of my research for this workshop I did look at some medical and experimental type journals in biometrics, human biology, human reproduction. These journals and other medical journals, since you mentioned that, and I don't of any data on that question you are asking either, but they offer different formats of abstracts within the journal itself. And they have different sections within the journal that accommodate different kinds of research.

In other words, it's not a one size fits all kind of structured format. There is structure within those different sections, but recognizing that there are different kinds of physical science type of research. That is not one fixed structure. That's an important point for us to think about here.

The other thing though that I don't want to lose, I think we are drifting into really technical issues here. That's important. In fact, in the Mosteller et al article you say that's what you want to focus on, are the technical aspects, having prefaced the article with the larger sort of disciplinary kinds of issues, and even slightly epistimological issues.

But I don't want us to lose sight of really grappling with those issues. And I think there are two things here. One, we just barely touched on -- Nancy Cartwright's comment this morning dealt with that more directly -- but the first one is the extent to which the abstract constrains or contains the content of the article itself, hence, the organization of research and the content of that research. That's a critical issue I think that we need to not lose sight of.

And the second is the purpose of the research itself. And I would like to just frame that in terms of a comment that one of our readers made in response to my list serve query. He said that a structured format might make educational research "more palatable to a broader audience." But he described the format as in his words, "window dressing that leaves the harder conversations of the inequities in schooling, differential school achievement, the social justice issues unexamined."

That is, what is the research for? So, in terms of an abstract then, who benefits from this kind of a structured format? Who benefits from this type of research? And I would like to see us deal with those deeper issues, and the abstract then to me seems an important, but a technical issue, and maybe not as central to what the point of the research, the interests that are being served by the research, including how that research is going to be interpreted by policymakers and practitioners, what all of that means.

DR. EMIHOVICH: That is a really important point, Terri, and it just sort of raises the question that I wanted to get to at the very end, because I wanted it to lead into the open discussion, and sort of really get people stirred up and thinking about it, because I think it is a very critical issue as we start to think about how we are using research to serve social purposes, and what changes.

Because the critique that has been leveled against education research is that there has been all these years of research, and schools looked just the same as they looked ever. Whereas, you can say in medicine at least you know people don't do the same treatments any more. There have been some advances.

And leaving aside that whole analogy and comparison, which is problematic in its own right, and I don't want to get into it, but there is some validity to that in the sense that we've got 50 years of research on marginalized students or underachieving students, et cetera, et cetera, and you know what? The research today looks just the way it looked when I was in graduate school not 50 years ago, thank God, but I could go back and find research that old, and it would still look the same.

And in fact, I could even do an experiment, and I've tried it, I could change the dates on some papers that were written in the early sixties and seventies out of the anthropological literature, and I could send it out for publication, and people wouldn't even know the difference. And that's a really telling statement about educational research if that could happen.

So, when we play with the categories for research, because had talked about saying well, maybe not all these categories fit. There are cases where they do fit, and there are some basic categories we might want to see, what would be the interest in adding another category that said that the research at the end, whether it raised deeper social questions about equity and social justice, would there be some value in marking articles to say do they do that? Does the author even get into those questions?

And whether that would, as Ken said, does that become the invitation for people to want to go back and look at the research itself? And then look at the quality issues, because there still has to be an issue around quality. We are not going to be talking about people's ideology being framed. We do want to have some reporting quality.

But I think that those are deeper questions that push us in a different direction about how we think about the utilization of research. Whether it is for policymakers, whether it is for practitioners, or whether it is even for ourselves in thinking about pushing the field in new directions, both theoretically and methodologically.

And I think that's a bigger question that just sort of the actual format, but I think it's something that we would want to go into. So, I'm going ask each my panelists what they think of that idea, and then we can start to move into perhaps the open discussion.

DR. WONG: When I about your important question, it looks like we might be dealing with this question of who is in charge. That is, right now I think the structured abstract, from my perspective, doesn't quite take away the control of the content from the contributors of the article.

So, if the contributors would like to underscore the importance of social justice, or for that matter the other continuum is emphasize market-based reform or something like that, then in the conclusions and recommendations they could shape those specific details and put it in the implications of the concluding section.

Or when they talk about the intellectual underpinnings, they can talk about do the treatments work, or whose work right there. And then talk about how the build on that line of investigation to do their studies. So, I think there is a lot of room, from my perspective, for the contributors to continue to exercise their authority and control over the context.

Now, that also gets to Teresa's earlier question, which I don't quite know how to handle, and that we talk about censoring and privileging particular strands of research and methodologies, because of the nature of the structure. And I think that's a very important question, because that may be potentially one of the unintended consequences, that may have privileged a particular paradigm.

But my middle ground approach is that maybe there are ways to fix some of the concepts and definitions here. For example, intervention, we don't want that. And we can put in societal dynamics, or something like that. So, to change it in a way that opens the box up a little bit.

DR. WALTERS: I want to go back to a comment I made earlier about how in my view the best sociological research starts out with sort of a fairly large theoretical context, and then derives a testable research question from that. And so, at the end of -- I talked about the front end of a paper implicitly. That same paper will at the end, come back to that larger theoretical question.

And so, that could be reflections on the larger social and institutional arrangements, the rules of the game, the power dynamics, the constraints to improving instruction given existing arrangements. And so, there would be a way to incorporate that within the context of a structured abstract, but it's not the same thing as conclusions. It's theoretical implications in a much bigger sense.

DR. MC CARTY: I'm sensing that we summing up on a very huge issue, so I'll look forward to hearing comments from people in just a few minutes.

I think I want to just say a couple of things here. When we look at this particular format, we have to ask what kinds of research is accommodatable by that particular kind of format? What does that say about epistimological issues? That is, what does count as knowledge? What kinds of things do we think are important?

If you draw that out to the concerns in the Mosteller et al article, then what does that say about the kinds of policies and practices that we might be concerned about? Whose interests are served by casting research in a particular way versus casting it in other kinds of ways? Those are questions that need to be asked.

Having said that, I also want to say that I think in terms of educational research journals, that editors do play a very critical role. Editors are in effect a kind of gatekeeper, an individual who is really sort of helping to build knowledge in the field just by virtue of what we do as editors.

And so, I think that beyond the structure of abstracts, there are messages that editors and do and should communicate that speak to these larger issues of what we're using the pages on this journal, the online resources for. What do we see as the purposes of the kind of educational research that we? What does that mean for the types of manuscripts that we want to encourage? The formats for that type of research? The individuals who conduct that research, their own backgrounds and subject positions?

So, I think there are other things beyond the abstract that fall within the realm of the editor that perhaps will be more effective in addressing these kinds of issues, and helping the particular journal address those types of issues than simply focusing on an abstract format.

DR. FLODEN: So, I think education is an inherently value-laden area of inquiry when we are talking about the things that we think of as desirable or undesirable, not sort of merely descriptive. So, if the abstract failed to give the author a chance to make clear in some way what the sort of things they see as valuable in education, and what sort of theoretical frameworks they are drawing on, so that people could use that as part of the basis for thinking about whether or not they want to read the article, that would be a problem.

And sort of consistent with what some other people have said, there are some things in the particular list that might already lean things in a particular direction, but I think I'm with Ken in thinking that in the background and context and conclusions or recommendations, perhaps as written, perhaps as slightly modified, there would be an opportunity for people to make evident what they are thinking about as desirable in education, and what sort of theoretical framework they are drawing on.

And I would say the journal I'm editing right now is a journal of reviews rather than individual studies, but that's also the case there when you are thinking about how to structure the review, what you are going to look at. You are making decisions about how -- say it's a review on a professional development as a policy tool, something that came out in the last issue.

You would have to think some about so, well policy tool for what? What are you going to use as a way of thinking about how you are going to look at the research? What counts as professional development, what doesn't count as professional development? And the abstract, whether structured or unstructured, an intermediary way for the author to communicate something about that to the reader, to help them decide whether not they want to look at it.

And in the course of this conversation, I think I have become less and less convinced that the structured abstract per se is going to help much with machine searches, since the machines can look across all the fields anyway, whether they are in a paragraph or so on. What the structured abstract I think helps do is helps the editor and the author -- reminds the editor and the author what the particular journal sees as the sorts of things you want to make sure are being talked about in the article itself, as well as in the abstract.

And as we heard in the last session, in my own experience in looking for things, often the problem isn't that something was missing in the abstract, but that something was missing in the article. You want to know something about if it's something about mathematics education and its effect on students, something about what you used to decided whether or not students learned anything. And in some articles it's just not there. We did this and students learned more.

You need to know something about that, and whether it's in the abstract or not I think is not going to make that much difference in terms of machine searching, because machines will be able to search the whole text. But the abstract is a way of reminding people this is something that ought to be a part of the article in some way.