Q&A

DR. BRADBURN: I see that time is running on. We should open it up for questions and comments and reactions and new information that might have come from these discussions.

DR. TOWNE: Lisa Towne.

I had a question for Susan. I'm curious about the process of indexing terms. You talked about the fact that there needs to be a critical mass of articles that reference something before it makes it, so to speak.

Is there a process by which you go back to those terms and revisit them in the case that I'm thinking -- I can't think of an example off the top of my head, but I could imagine a scenario in which the meaning of that term might change over the course of a body of knowledge accumulating over time, where concepts are tweaked, validated, changed. And so, I'm wondering how in the process of indexing, you think about language changing? And how that mechanically might work, and the complexities around that.

MS. HARRIS: I don't know the entire answer to your question, but I do know that we do keep an inventory, if you will, of terms that have been used over time. And that is how we establish the critical mass of course.

In terms of the meanings of the words shift, I'm not at all certain how we have quantified that. One of the things that we have just begun to entertain is in fact forgive me, keywords as part of the journal structure for exactly the point that you are talking about. Language meaning does shift. And so, what we are trying to match is when the keyword appears, and the context of the keyword. And as it shift over time, we'll have a better handle on what the ultimate meaning of it was.

But we're just at the very beginnings of that kind of an analysis. It's far more complex than it sounds on the surface, absolutely. So, all we have really is just a word, number of times it's been cited, and then we go back and look at it and assess it.

MR. JOUBERT: MESH, which is the Medical Subject Headings used by the National Library of Medicine has a very rich tradition of using controlled vocabulary. I think it's somewhat easier now with the age of computers, the way that they are. And if you pull up what's called a scope note in PubMed, it will actually give you a previous indexing term, when that term was introduced, which are very good for longitudinal studies.

So, that is one way that NLM actually addresses that. And from anything past that indexing term, you have a see reference, which is you might be looking for this, but you might want to look at this other thing as well. And in the age of computers those are all linked by unique identifiers. So, that makes updating all those previous records a lot easier.

DR. BRADBURN: It sounds like analogous to what was mentioned in the previous session about see related articles. It's see related terms.

PARTICIPANT: As a person who is not a researchers, but tries to use research, do you think it would be useful to include information about who was funding or sponsoring the research?

MS. HARRIS: Our journals do reveal that. That's done in the author information. It's revealed regularly, but not in the abstract.

PARTICIPANT: That's one of the questions I was asking in terms of the abstract, because a huge problem for kind of lay people who are trying to do things it to try to get indicia of quality I would say, as you are trying to sort through. And that's why I was asking that.

DR. BRADBURN: I guess I would hate to see the sponsorship as a criteria for whether you read the article or not. Again, it's all the kind of problems of what you get into the abstract, and whether you put so much in the abstract you discourage people from reading the article, which is what I think some of us are afraid of with the structured abstracts. That in fact, people will depend on the abstract and not actually go read the article.

So, I see the abstract as more a decision tool of whether you want to go read the article or not, not as a substitute for reading the article. So, it's important that sponsorship be -- someplace in the article you can find out who sponsored it.

MS. LEVINSON: In meta-data that kind of record information is often included. It's a government report, grant, contract.

MR. JOUBERT: In most medical research, if not in the abstract, near the top, you want to know who is funding this research, definitely.

PARTICIPANT: The National Criminal Justice Reference Service puts that in their outline before the abstract. It's an entry. And as a big Psych Info user, it would be a nice entry to have, and I think it's certainly something to consider for ERIC. It just gives a sense of -- I was looking at an article by Barbara Ruskin(?) and saw that it received funding from the Campbell Collaborative.

Then it gives you some information about the links to a set of sort of collaborative networks that I wouldn't have actually know just from looking at it. I thought, oh, that's kind of interesting, so I think it's worth including. It wasn't in the abstract though, it was in the tag lines, just as you have stuff listed.

DR. WALLER: Alicia Waller again.

I'm just sitting here thinking about how I do searches, and how I would explain that to a student. And I think I do more of an almost semi-random walk on a network or a map. I don't go through decision-making trees. And one of the reasons is because if you go on one branch of a decision-making tree, you have cut off the rest of the tree and you are stuck in that branch.

And so, in thinking about new ways to structure key words, or to think about how we set up search engines and those kinds of things, I would throw out the idea of a mapping as one that is different, just like concept maps help us understand more deeply what students understand than do categorizations or definition lists, because it allows things to trade off of one another. And you get a better chance of covering a field.

The whole idea of it being a linear tree I think is somewhat artificial to the way that I do research, and that a lot of my colleagues do. Because instead, we tend to wander a map as we do our searchers, and add terms in, and then take terms back out, and that kind of thing.

DR. BRADBURN: Citation indexes are another way of moving around in a kind of non-linear way among articles too.

MR. JOUBERT: Yes, I do think people have talked about your user group. I think that's very crucial. As I mentioned earlier, when I was reading the Haines article, I first envisioned this strictly as just an editorial decision. But I'm glad we've had a lot of this discussion, because I have been able to think out of the box about what this actually means to users.

But you really do have to consider your user group. Is this for the benefit of the larger research community, or to people that are outside of our discipline, because I think it does make a difference.

DR. VERMALY(?): Judith Vermaly. Throughout the day there has been a little theme that I want to pick up on. And in fact, the last comment or one before, or part of it. And that is the world is not made of individual particles. That is, how you get from a particular article to a deeper understanding of a larger issue.

And one of the ways, which reminds me of how I decide what movies to watch, is you get to figure out which movie reviewer has your same taste, because you can compare how you reacted to how the reviewer does.

But several years ago when I was at the University of Vermont, I was walked through a medical informatics model which in fact applied indicia of quality which would allow a busy clinician or any other user to figure out whether the marks of quality of not only the work itself, but the way it was presented in the article were present, and therefore you could decide how much weight you wanted to attach to that particular piece of work.

There was also the beginnings of a mapping that would actually allow you, in a different way than simply other, to look for not exactly a semantic web, which didn't exist then particularly, but to look for related things.

I'm curious if that's advanced? If there is a working model either in the medical literature or elsewhere that would let you get from an individual piece of something, to some sense of a larger picture?

MR. JOUBERT: Well, they can adapt somewhat. I feel at least in Canada and in England there is a larger movement in the medical community for evidence-based practice. It's controversial. Brian Haines, who wrote the ad hoc, he was on that committee, is a person who is at the forefront in that area. His background actually is clinical epidemiology.

And people in the medical professional, as shared here, feel that part of what they do is an art, as well as a science. And being in this kind of linear, as they describe it, method of actually saying person has X, Y, and Z, therefore the clinical manifestation is C.

So, the jury is still out. We are implementing something called CRIS at the National Institutes of Health, which is Clinical Research Information Systems, which is actually tied to the electronic medical records. There is hope in this field, but you definitely have people that feel that clinical medicine doesn't reside in a small box, if I'm understanding your question correctly.

There are a lot of schema that exist, especially in AMIA, the American Medical Informatics Association. Nothing is perfect at this point, but I'm hopeful that as computing power increases, that some of the hiccups that they are experiencing now will probably be addressed.

But you do run into the problem where that art crosses a science too. And people feel that I have 12 years of experience as an ER doctor. And there are things that exist, very expensive products that exist, that are there to assist physicians in making clinical decisions.

DR. VERMALY: To pick up on that, I do remember having in a former part of my life, been an associate dean in a college of medicine long enough ago that I think I'm past the statute of limitations. But the question is the concern about tacit knowledge or clinical judgment, which some people think is as much as 80-85 percent of anyone's practice.

And the extent to which it's picked up in medicine, there have been real efforts in expert systems to model and capture some of that, leaving the real clinical judgment I think you are referring to, to the more mature and practiced experience of people who have had a number of years at work.

We have spent a lot of time today comparing health care, medical research, clinical research to the challenges of the application, interpretation of social sciences to a variety of large public, as well as private issues. I'm curious from your point of view, but also the other panelists, how good is the analog? Are we likely to get stuck if we continue to think of the medical model, so to speak?

MR. JOUBERT: Well, I think for me, just on a personal level, one of the things I've got out of this, I came to this session thinking reading these three objectives from the ad hoc working group, I'm like, well, we're all on the same page. We're not.

And I don't know if it's because of the translational nature of research in medicine, where not that one type of research is better than another, but the motivation was there to handle these issues quicker in 1987, for whatever reason.

But I think as any discipline, I love the comment the person said that she was into cross-disciplinary collaboration until she became the director of the center. I think that it's good to have discussions like this, because I think there is stuff that we can teach each other.

I have learned one thing, that the information-seeking patterns of social scientists are very different than they are for clinicians. And that is just different. It's not good or bad, it's just different.

MS. LEVINSON: I think in IES particularly, that the exploration of this model comes at a particularly good time, when over the last couple of recent years there has been a lot of change and discussion about research, evidence-based research, and interdisciplinary uses around school issues and school problems.

So, I think that the timing, as I said earlier, is parallel to the situation that I think some clinical/medical researchers faced. Beyond that, I think everything has to take its own momentum, and that the researchers that are thinking about this, that are here today, that are inside this working group, as the new ERIC takes form, will find its right mix.

And I think that's good. I think it will have a life of its own, and then if it expands beyond one particular kind of literature, then it will take its course.

DR. BRADBURN: Is part of the redesign effort for ERIC looking at how users search? Or looking at the user community and sort of saying what sort of strategies do --

MS. LEVINSON: At the moment there are none. But what we do have in place that we didn't have in the other ERIC model is the capacity to do research on an annual basis, topics primarily suggested by the steering committee of what needs to be studied in order to in short, keep ERIC modern and up-to-date. And that could well be on the table.

Again, we haven't come to the end of our first year, but to date we haven't done any user studies. Our contractors have done usability studies on the interface that we now have that was put in September, but not user studies yet.

PARTICIPANT: I was thinking in a way when I said in terms of a medical model. And if I were looking at a whole slew of articles, and I saw that something had been funded by NIH, I would know that it had gone through a pretty rigorous process in terms of making decisions about what was being funded, and how it was going to be studied, and things like that.

And so, that wouldn't mean that other research that was done wasn't also of a high quality. But for a person who might not be as familiar with the field, that would be something that would be -- as you are trying to decide how many of these articles am I going to read? And what am I going to read first?

DR. BRADBURN: Other comments, questions?

DR. BORUCH: When you mentioned what are some alternatives or different ways you were once thinking about this, related questions. The fact that the Institute for Scientific Information, the ISI, produces current content, does lots of the stuff is now selling a product in which for $30,000 I can get every issue of eight journals, a full text search for the appearance of my drug, or the appearance of a particular word.

It sounds like part of another wave of the future in this search business. They seem to be doing well. Basically, their population of journals includes I think about 8,000 of them, the full text searches. And the buyers of these things are big commercial drug companies and the like.

In trying to understand whether they could do the same thing for the social sciences, or why other people aren't doing the same thing, they basically asked me how much money do you have to give us to develop a system and all the rest of that.

So, the back drop question is one, what do you think of the financial viability in the future of full text searchable stuff? And two, where does this kind of conversation fit into that larger much more massively complex capacity for analysis and retrieval?

MR. JOUBERT: Well, there are two sides. I will try to address this the best I can. Before I came to NIH I was in an academic health science center, where we were not funded, and we're very well funded, and consider ourselves blessed at NIH. And we have products that you are talking about.

What I hope will happen is that the open source movement will actually take ground, and stuff like the Public Library of Science and Biomed Central, where there is a fee involved in that, but access is actually much more substantial, because you are actually bypassing vendors like Elsevier and the more expensive science vendors.

That's what I would like to see in a more perfect world. Because I think there is an open source movement in my community to actually promote equitable access. SPARK is another in the academic --

DR. BORUCH: It sounds like something we ought to be aware of on the committee.

MR. JOUBERT: I'm sure SPARK -- and it depends what discipline you are looking in. Biomed Central is very popular, and it's out of England. There is also the Public Library of Science, which is also open source. I don't want to comment on the model, because I'm not exactly sure how the copyright is handled.

But there is an initiative at NIH. Zerhouni has the Road Map initiative as far as creating open access. That's what I would like to see everything moving, rather than paying $30,000.

DR. BORUCH: Right. Are there people about whom we should know?

MR. JOUBERT: SPARK. I can't remember what the acronym stands for, but that's Scholarly Publishing --

PARTICIPANT: DBASSE took it up on a day event in November, and it's not quite as black and white as might have just put it. Obviously, open access is very complex issue, and one that DBASSE has put a full day into already, with several presentations, including from Gary Vanderhis(?).

DR. BRADBURN: I don't know, what I read recently, Google proposes to put all of Harvard's, Michigan, Stanford library online, electronic versions of all the holdings. It didn't mention whether journals -- it said all the holdings, so I presume that meant journals, but I suspect there may be copyright problems on the journals, or at least more recent editions. That's a 10 year project.

It does I think suggest where the future lies, so to speak in terms of sort of basically getting text into electronic form that can be used. Then it becomes a kind of economics of how you fund it, and how you make it available. If it gets captured by ISI or places that want to make a profit out of it, the price can be very high, but there are also forces for making it open, or at least at a more manageable price.

MS. HARRIS: I think one of the things that you are alluding to is a natural outcome of the bolus of information we have available to us across all sorts of different platforms. And that is you can actually customize your information needs, not just because you are in Psych Info and you choose key terms.

But if you have an organization that is interested in a very specific drug program, or if you have an organization that has very specific treatment needs or whatever, for a fee, if that's the model or as free access, if that's the model, if there is a way for us to slice that out so it becomes a customized product, I think that's a very logical outcome of helping to resolve the enormity of the content we are trying to search against, across all those different platforms.

That's a collaboration of a lot of people, and a lot of resources to make that so. But from an end user's perspective, I think that would be an ideal outcome of the fact that we digitized just everything in the universe.

DR. BRADBURN: Any other comments? Well, it looks like we have about reached the time for a wrap up. So, Hannah, where are you? She has the unenviable task of telling us what we said.

Agenda Item: Themes & Future Prospects - Hannah Rothstein, Baruch College, CUNY

DR. ROTHSTEIN: For those of you who are still awake, you might be wondering why is it that somebody from a management department is here in this role, since I don't have a background in education, except for all the years I spent becoming educated, and then all the years I'm still spending trying to educate my children. Due to the age range of my children, I am now in the 22nd year of parenthood, dealing with educational institutions. I think maybe that doesn't make me an expert, but it makes me a good critic.

My PhD is industrial/organizational psychology, and at this point, 24 or 25 years later, I really think of myself as a social scientist, and not wedded to a particular discipline. That being said, most of my searching experience is with APA products. And I probably wouldn't be sitting here today if it wasn't for some of the vulnerabilities that I encountered in the APA system. So, I hope I get to just chat with Susan a little bit too.

But before I proceed, I want to thank today's sponsors, because they are very important. The whole day wouldn't exist without them, and if I don't mention them now, I will probably forget, and then have to kick myself later. So, I would like to thank the National Science Foundation, I would like to thank the Spencer Foundation, and I'd like to thank the Hewlett Foundation, as well as Lisa and the rest of her incomparably talented, indefatigable staff for doing all of this.

And I would also like to thank all of the other speakers to allow me to listen to them, because it's been a real treat to hear people with so many good ideas, and so much passion for their work. And it's going to be something that I'm going to remember for a long time.

My particular interest at this point is publication bias. And what publication bias is concerned with is the worry that what appears in the literature, or what research is available to the public in a particular research area is somehow systematically unrepresentative of all the research that has ever been done on that topic.

And I wanted to lay that out, so you understand my background and interest in abstracts. So, I'm concerned that first of all, everything that is out there may not be a systematic representation of what has been done, conversation in part to have with authors, in part to have with editors.

Then I'm concerned that what actually is reported, and is out there in published articles, that is accessible to us is not an accurate representation of what's been done in those studies. And I think that I don't know if it was Catherine or maybe Teresa who alluded to it when they said well, we know we've run 50 multiple regressions and then we up with that one table that we pick.

Well, one of my worries is that table is really selectively chosen, and maybe if we saw the other 49, things would look different. So, I think as many of the speakers suggested, even when you get to look at whole articles, they are not really accurate or faithful representations of the research that has been done, and that concerns me.

And so, obviously, the next step here is well, what is out there isn't everything that has been done. What we see isn't everything that has been done about what is out there. And then I'm concerned that the abstract is not an accurate representation of even what's being said in the article. So, if you follow that path.

The reason that I'm particularly concerned with abstracts, and why I would like to see some changes in abstracts is because I view them as the front door to a lot of research. An analogy I have been using lately is that if the abstract were the singles ad, and you had to have a blind date with the article, would you really trust the abstract as your guide to accept that date or not?

And I think in most cases the answers are you know, I don't really think I'd trust that. So, we should have a mechanism that we can trust to at least be a faithful representation of what is being said in the article.

Now, one of the truest things that I have learned from my jogging partner, a clinical psychologist, is that the world is divided into lumpers and splitters. And I think we have heard a lot of lumpers, and a lot of splitters today, and you can kind of know already who's who.

Since I'm a lumper, first I'm going to state what I think we all agree on. I think we all agree on the fact that current abstracts have some serious deficiencies that need to remedied. And I think we all agree on the fact that one size fits all structured abstracts are not going to work. Beyond that, I don't think we agree on much else. But okay, two things are good.

A little bit of my concern is that I feel bad at this point, for the abstract, because at the same time we have both given it too much and too little credit. So, I think that it does, for better or worse, cue people as to what the article is saying. And I think you're right, a lot of times they are not accurate.

I'll just my own APA journal background. I don't search just APA journals. I review for three of them. I have two friends on the council of editors, both of whom I lobby a lot. And I do a lot of reviewing, and I think that abstracts -- nobody really checks to see whether they conform with those wonderful list of things they are supposed to.

And often they really have a barely passing resemblance, even in the best APA journals. And I think APA produces the best journals. I haven't read yours as deeply, although I have looked at them to prepare for this a little bit. So, I think that something needs to be fixed. Whether it's this particular form or not, I guess we should discuss that. But I think if the comparator of interest is the current abstract structure, that's just not working.

So, I think that people give abstracts too little credit in terms of what they signal about the article, but I also think that we give them too much credit if we think that somehow they are going to have this huge political effect on the kind of knowledge that is allowed, and the knowledge that is created. I don't see that happening.

I think that the editors have so many other places where they can control things, from who the pick on the editorial board, to who they get to review the articles, to the decisions they make. So, I don't see just the abstract itself as being so constraining that it's going to fundamentally change the nature of the way research is done, but maybe there is something that I'm missing there.

I'm more concerned about how popular topics or policymakers' priorities affect what gets done. Sort of the political hot thing of the day, and whether that influences research, and what we think we know. So, I think maybe we're dumping a little too much on the head of the abstract, at the same time as we are devaluing it.

I also wanted to mention I work closely with both social scientists and medical researchers. And hold onto your fantasies about medicine, because the reality is a lot messier. First of all, the number one complaint I hear from people in the Cochrane Collaboration, which is the organization that does meta-analyses of health care interventions is that practitioners don't listen. They ignore them.

They either do what they were trained to do, what the guy in the office next door does, or God forbid, and this is probably the worst, what the drug salesman tells them is the best thing to do. So, even though they do have maybe better abstracts, it doesn't seem to affect practice.

The other thing is that in our own country we talked about autonomy of physicians. Well, what about managed care? So, there are a lot of things going on that affect our fantasy about how research and evidence-based practice happens in medicine that I really think is a little less than true. That doesn't mean we should get discouraged though. I think we should plug on to make sure that we have the best and most accurate depiction of the research for whatever audience.

Now, I wanted to first thank the first panel for making my job a lot easier. I think that it is helpful that Bill and Ed admit that this is not a one size fits all. It's not going to work for everybody, but I do think it got the discussion going. If it hadn't been this, I would have been something else that comes out of the specific research area that reflects these idiosyncracies of that kind of research. It just happened to be this one, but it didn't have to be this one.

As was pointed out, it really should start us thinking about the general process of how we can improve the quality of what abstracts look like.

And I think Felice did a great job in pointing out the ways in which we can expand from this model, to a model that would fit behavior and social sciences more generally. It's interesting to hear all of this referred to as an intellectual movement and a call to arms. I rather like that.

I think that it's good that we realize that these things can be written in basic science. My own favorite journal, which is Journal of Applied Psychology has "applied" in the title, but it does publish a lot of theoretical research. And when I sort of tried to play around in my head with what a structured abstract would look like say on an article comparing theories of leadership, it seemed to work.

Now, I wonder, are we interested only, just looking from the perspective of a researcher, are we interested only in our own narrow field, or do we want to stay up-to-date in a broad area as well? Again, I here is the lumper versus splitter thing.

I want to be an expert in my particular field, but I don't want to become so narrow that I don't know anything else that is going on sort of in the broader field of industrial psychology, or even more broadly in social science research. I think structured abstracts really would help us be broader and expand beyond what Bob calls the historical accident of discipline.

So, I think one of the constituencies that we haven't talked about was all the people who live in the space between the people who are interested in that particular research question. Then we talked about the brokers. Then we talked about the users.

But what about us in the middle who just want to be informed social scientists beyond the specific questions we ask? Do we really want to know more and more about less and less? Or do we want to be able to know something about more and more? Which means we need to get back to things like commonalities of language, and ability to cross boundaries and still have some understanding about what's going on.

I think people are concerned if we have some kind of structured abstract that we'll have to reduce the length of the actual articles. Well, they're too short anyhow to contain all the information that we want. We have heard that, right? And I really think that there is a limited life left to paper journals.

So, once we have everything online, and with I think the cost and availability of memory, we should have infinite archival capabilities, and there should be something in articles that enables us to click on what's in that part of the article or that part of the text, and drill down for more information.

That puts a big burden on authors, but I think more people write too much about too little, and are probably better off writing more in-depth about what they did. Someone mentioned tacit knowledge. I think there is a lot of tacit knowledge that doesn't get communicated in the journal article format. And we should be thinking about ways that we can include that.

So, I'm definitely not thinking that people are going to use the structured abstract to replace articles in areas where they really want to go in-depth. If anything, I'm proposing that this has to be one of the set of changes to make scientific reporting a more accurate reflection of the science that has been done. And I only see that as one piece of it.

Another thing, let's try to think about this Medical Library Association, where it does have models for lots of different types of articles, ethnographies, case studies. But rather than trying to react just to this, I would suggest that people in the audience take a look at the breadth of structured abstract types that are available.

Someone mentioned that we should have study type. And then study type would then branch us into the specific headers or categories for what's in the articles. I think that's a very good idea. I think this is a lot more expansive than most people were imagining.

To me, as a research synthesis, it was only when I started reading abstracts and articles really closely that I realized how much was missing. And I think that those of you who don't like the idea of imposing more structure and more guidelines are not going to like this at all, but I would really like to see more reporting standards for the articles as well, not to constrain people into having to follow specific instructions, but more as a checklist to make sure that there are not glaring omissions.

And I think again, Susan very honestly alluded to this. But I think that editors and reviewers are really so overburdened, because of the incentive system for editors and reviewers, that even though they do a very careful job, and spend a lot of time, there is still a lot of things that slip through the cracks. And it would be nice if somehow the incentive system were changed so they could actually do a better job, and there would be less missing stuff.

But I think that one of the things about having a structured abstract that works for me is that it really helps you notice things that might have slipped by, that are missing from the article. And even if it just became sort of a checklist against which you can measure or look for these glaring omissions, that itself would be I think a worthwhile thing.

I think that Norman in this last session asked about what we might have as alternatives to structured abstracts. I like to think of them as supplements to structured abstracts instead. I think that there really do need to be indexing changes. I think again, as Lisa mentioned with Susan, that things aren't retagged when something changes. We don't have the sore eye.

And I think this happens in education. The same theory or the same intervention's name gets changed over a decade or two, so when you search you really only find part of what is going on unless you are very creative, and have a great historical sense. It would be very helpful if there were thesauri, if we knew how these related terms were derived.

People are concerned about control issues. I'm a little concerned about who decides for me what related terms are, who decides for me what related articles are. I would want to at least know how that was mapped out before going down one of those paths, because I feel that it's much more dangerous in terms of someone else making decisions for me than having a structured abstract.

I really don't understand the idea that no matter what format it takes, or no matter how much control within a discipline have over these things, it invariably has to be a straightjacket. I think if anything, it would open things up to more public inspection by more people, because people could find these things more easily. And if you can't find it, then it could be beautiful, brilliant research, but people are not going to know that it's there.

I think for those of us who are concerned with equity issues, I think we don't understand how important abstracts are to people in developing countries, who have bad libraries, who have limited resources, who really rely on these abstracts, because due to resource scarcity, that's how they decide where to put their limited resources in terms of hunting down studies. If we want to really open things up to people with limited resources, the abstracts are I think a doubly important purpose, because they really do rely on databases a lot more than we do.

To me, cumulative knowledge across social science implies certain commonalities. If the commonalities are there, then I think if we work hard, we can discover them and build these into the abstracts so they can be used to some degree across disciplines. I don't see that common abstract structure does any better job in reflecting the complexity or the problem or the article than the alternative that is being proposed. And I like to compare things to the available alternatives, not to some ideal abstract or some ideal system that doesn't exist.

So, I think that whoever talked about the healer and the grandchild, to me that was absolutely fascinating. And if I had a student who was doing research on something like occupational apprenticeships or transmission of knowledge in informal settings, even though I'm in management, I would want a student to somehow be able to find that literature, because I think that literature would enrich what they are doing.

And I don't think that under the current either way abstracts are written, or the way search engines are built, they would ever find that. And if they ever find it, I would never see them again, because they would be in the library forever. If you want to do such an exhaustive search that you are really going to track down everything under the current system, you are never going to get past that library search thing.

I'm sorry that I have never heard of Prof. Ogbu's research before. And so, I guess my question is you said that he was the most influential person in his discipline, but apparently it doesn't have a much greater audience beyond that discipline. Maybe it's just my lack of knowledge, but if he's that influential, he should have made it into the popular press. He should have made it into lay science articles. Steve Olson should have been writing about him, and he has.

And I think actually it would be really great if things like sociocultural context, background, theoretical perspective could be built in this. If I got to add to my wish list of what was going to go in an abstract, I would pick something about variables. Like I would like to know if there are predictors, what the predictors were.

I would like to know especially if they are outcome variables, what the outcomes were. That doesn't appear here, and I think that's a failing. But again, I think it's a failing of one particular suggestion. I don't think it's a failing of the whole model.

If you are concerned about again, the complexity issue and how to communicate complexity, telling people that what they should do is write op-ed pieces or press releases about their research is definitely going to make things way simpler, and there should be something intermediate. And I think that we want to try to get more complexity into the hands of people who are going to make those decisions, without making it so complex that they can no longer understand what's going on.

I don't know whether it's structured abstracts that could do that, but if we write structured abstracts that help the brokers, and they said that it did, then maybe we can actually communicate some of that eventually to the people who do policies, and policy and practitioner people.

I think Pamela mentioned that she thought that structured abstracts might privilege certain types of research, but just listening to her talk, she mentioned five kinds of things that she looks for as an editor, and I don't see why those can't serve as the five headings for the structured abstracts in her field. They were: theory, the research question, the evidence and how it was evaluated, the findings, and then the conclusions and the implications.

Just asking every author to lay things out straight like that, I think would be really helpful to people. I wish you were an editor in a journal that I submitted to, because it sounds like you put in more work to abstracts.

In any case, I think once you take the things that you expect the researcher to tell the audience, you can put those into something more of a structured abstract, and we just have to be a little more imaginative about it. I think that as someone said, evaluation has to be built into this. You mentioned that somebody in Rehabilitation Psychology did away with structured abstracts. Was that on the basis of evidence? Was that on the basis of any kind of experiment?

MS. HARRIS: I don't know the answer to that.

DR. ROTHSTEIN: Okay, because I was curious.

Professional judgment. Well, evidence shouldn't have to compete with that.

I think editors should work with authors on their abstracts. I did some informal surveys, talking to all the editors that I knew, and that's certainly not typical.

Teresa mentioned focus, methodological approach, sociocultural context, and significance. You didn't mention them as possible headings, but I'm suggesting that they could serve that way. I think that having to pay more attention to the abstract would make authors make their assumptions, context, and sort of implicit belief systems more explicit, which I think would be good for everybody.

I think the panel with the brokers were pretty positive about their use of abstracts. I'm certainly concerned about misleading abstracts, but again that's probably not a function of the abstract, as much as the effort that is put into the abstract. And I think that whatever else we decide, one of the conclusions of this workshop should be that abstracts really deserve a lot more attention and effort and finetuning than they have received presently.

They really do serve mostly as an afterthought, which is kind of sad, since they are the only thing that most people see of any article.

Steve mentioned that he kind of was excited by the fact that abstracts are often laudatory or exhortatory. I'm horrified when that happens, because I don't really think that's what should be in a scientific article.

People think that an RCT is so easy to summarize into a structured abstract. Try it. It isn't. But again, I don't think that theoretical articles, articles in my field for example, leadership, motivation, productivity could all be related.

I see abstracting as a part of a larger picture that would involve as I said, just more effort, indexing keywords, thesauri, and more historical perspective. I think that's something that editors and editorial boards could clearly have a lot of impact on, again given the fact that the incentive system would help them do that.

I think although I am part of the ERIC steering committee, so this is not a wholly neutral statement, I think that Luna's group shows a lot of courage in at least tackling this problem, even though it may not turn out the way they expected or it may not work on its first shot. I think that hopefully that as in biomedical science, once people see its advantages, they are going to all jump on the bandwagon, and you can say that you were the first one.

I think that if we moved any of these systems, there would have to be retagging, there would have to be revisiting the old stuff, or there is going to be a privileged position given to new research that is in the new forms, and that would be a tremendous historical loss.

I think that rather than looking at the medical model, which may or may not, for reasons I have mentioned, be relevant to interventions and people in sociology or anthropology might want to look at what epidemiologists have been doing, because epidemiologists on the health care side do deal with social contexts, do deal with larger issues, don't deal with interventions at the same level of physicians and surgeons do, and yet they have been able to make a lot of I think progress in this area of improving the nature of both reporting standards and abstract standards.

People who are interested in improving reporting standards and see abstracts as just a piece of the problem, as I do, might want to look at two documents. One is the Consort Statement, and the other one is the Trend Statement. The Trend Statement was developed in part by people at CDC together with some social science editors, mostly who have some health care interests, because it was developed for interventions with people with HIV. But based on the fact that when they went back to look at articles, a lot of information that everyone was sure would just be there, wasn't.

And again, my final point. I've been rambling, and it's after five o'clock and you all want to go home, but I really see the problem with abstracts is just a piece of the problem with journal reporting standards in general. But I think that if we tackle the smaller piece, which is the abstract piece, it will lead us to be able to have the tools and the strength to tackle the larger ones, so that we can accomplish the objective I think that we would all agree on, which is to enhance the accuracy and the validity of research reports that appear in the published literature.

Thank you.

DR. BRADBURN: Bob or Lisa, a final blessing? Thank you all for coming.