Response to Proposal: Questions & Issues to Consider - Felice Levine, American Educational Research Association

DR. LEVINE: I feel like I should start by saying that in a totally unstructured way I'm going to take you also on a trip, as it were. I was planning on showing a transparency that would have been the structured abstract of what I'm going to say. So, if Bill and Ed would prepare it as I say it, then we'll be able to put it up when we're done.

And actually I have entitled this, "A Personal Journey Leads to Openness, Optimism, and Further Inquiry." So, it is a journey of sorts, and I meant that seriously. And I'm very pleased actually to have the opportunity at this opening session to share some thoughts and observations at this important workshop on structured abstracts in education research.

The recommendation to -- and many of you may not have seen yet the October report, but I'm quoting from the report, "develop and implement policies requiring structured abstracts," is one of a baker's dozen in the report released this past October on advancing scientific research in education, prepared by the National Research Council's Committee on Research in Education.

Many of the recommendations in that report are part of a mosaic of considerations that together signal important directions for advancing education research and the infrastructures that support it. The report offers the research community and national research associations like AERA much to chew on.

AERA's own research committee will take the first crack at systematically examining all 13 recommendations in terms of their implications for steps appropriate for AERA and for educational research.

While I am Executive Director, as Bob said, of the American Educational Research Association, I am not speaking today as an ED of AERA in any formal representational role, but instead as an executive director, and in that capacity also, as publisher of now two social science research societies. Previously, I was executive director for manuscripts(?) at the American Sociological Association.

My own journey to this issue began quite slowly. It is accurate to say that of all the recommendations in that NRC report, the one on structured abstracts almost passed me by as a priority issue for our collective consideration until about eight weeks ago when an email invitation to attend this meeting arrived.

As a perfect test of recency and primacy effects, it now seems almost second nature to be wondering and worrying about structured abstracts, what this intellectual movement that I have now come to know may mean, how we should be thinking about it in scholarly publishing and dissemination, and why it was not so centrally on my screen before.

Indeed, I took it as a sign of my own preoccupation with issues of training and education in the social and behavioral sciences, including in education research, with questions of open access and electronic publishing, with commitments to data sharing and data archiving, other issues also covered in the other report, and with human research protection challenges in the social and behavioral sciences, that I was totally unaware of the structured abstracts movement and its potential for education research.

While I read the Mosteller et al piece in AERA's Educational Researcher, as I said my colleagues last year as executive director, of course from cover to cover I read it, it did not then strike me as a call to arms, of course reading an article in the context of a journal. With the transformation of abstracts a we have come to know them in the social and behavioral sciences until I re-read it in this context.

Like Lily Tomlin in the play, "The Search for Signs of Incredible Life in the Universe," I determined to search for truth and for distinguishing what truly is the essence of this innovation for education research. Also, like all good researchers, I invoked a combination of networks and knowledge interrogation to help in this journey, and to bring me to learn more about the realities of structured abstracts.

First on the networking side, many of you may know, but I know quite well, and I contacted Eleanor Singer, a senior sociologist and methodologist at the Institute for Social Research at the University of Michigan, who has been a good friend and colleague over the years. Especially because of her early research on productivity in science, looking at citations, I felt sure she would have some definitive reference or view about structured abstracts.

In a brief email back to me, I learned three things. First, Eleanor knew nothing about structured abstracts, immediately relieving my anxiety that I was not scientifically unhip. Second, and reflecting her strong expertise in survey research, she concluded based on what I had said in my email, that it was probably a good idea, because more structure tends to reduce intentional and unintentional bias. And third, invoking her own network, she noted that Bob Boruch was a very smart guy, so if was drawn to structured abstracts, this is almost literally as a quote, there was undoubtedly something there.

I moved beyond this in my search for truth, as Lily Tomlin has defined it, through rigorous research, first Google Scholar and then JSTAR. Both searches complemented each other, because the JSTAR archive has many more US journals in the sciences, including in the social sciences, with of course the rich database of full text access over much of the 20th century.

And Google provides an effective sweep, Google Scholar in particular, of information to the Internet and the world wide pools of material, many more abstracts than full text, but an immense number of online searches, including the Public Med Central at NIH.

As a caveat, I want to emphasize that my journey is far from complete. Today, I am here more as someone intrigued by the ideas and the information I recently encountered, and what for all of us is yet to come. Nonetheless, I am ready to share some thoughts about structured abstracts, abstracting, and the capacity of databases of abstracts for building the base of knowledge primarily, but also with other potential applications and use.

In my remarks I want to broaden our view on our agenda today by first focusing on structured abstracts in education research as a case in point for examining its potential to all fields in the social and behavioral sciences. And second, questioning the assumption that as a generic matter, structured abstracts are more appropriate for particular designs or methods or for work with applied focus, intervention, or external customers or users.

Indeed, its primary value may be in facilitating not the policy or practitioner community, but in facilitating researchers and building an incremental body of research. The Mosteller et al article in ER in attention to structured abstracts by the What Works Clearinghouse and the Campbell International Collaboration have led us to an emphasis on it potential for education research, and especially for policymakers and practitioners who are users and consumers of applied work.

The issue of the links between research and policy in any field, including in education, are much more complex than the mere accessible linking of knowledge that this line of reasoning might imply. But in any event, and without dismissing that potential, we may sorely misunderstand and misspecify the potential of structured abstracts if we conceptualize and confine their use in that way.

So, to this end, I want to do three things. First, to step back with some observations on the emergence of structured abstracts and historical attention to quality abstracting more generally. Second, in light of the Mosteller et al article, to raise some questions and issues that merit consideration, and to engender a culture of experimentation as we examine alternative structures or strategies.

And third, to conclude with some observations on next steps in the consideration of structured abstracts as they relate more broadly to making effective translation of research, and to the improvement of research itself.

First, the history of systematic abstracts. The emergence and use -- and it's covered in brief in the article -- of the term "structured abstracts" in the mid-1980s, I think 1987 was a landmark date in the context of clinical medical trials has led us to think of such abstracting as having its primary use in reporting on interventions and evaluation research.

Since that time, the use of structured abstracts has fanned out to other domains. It has been adopted in basic medical science journals, and especially in Great Britain in social and behavioral science journals as well. The value of well prepared, and precise abstracts was recognized quite early in the 20th century when both analytic abstracting and journals of abstracts were coming to the fore.

In 1921, Warden Fulcher(?), writing on scientific abstracting in Science addressed the question of, "Is it worthwhile for scientific journals to provide abstracts at the beginning of their article?" His answer was, "Of course it depends on the nature of the abstract."

But then he went on to say, "Suppose it summarizes the methods, conclusion, and theories that let's you give all the information any reader not a specialist in the narrow field involved needs? That is, suppose each is the result of a careful analysis of the article by a competent abstractor?"

He emphasized, "Such abstracts and the possibility of creating abstract journals," -- for us electronic databases of journal abstracts would be the equivalent -- "would facilitate researchers quickly learning about the research they need, allow for integration across domains of research, and permit more efficient and effective use of time," as they were putting it in the twenties, given the overwhelming amount of materials that was much than even researchers either needed or could read.

Having promoted the use of analytic abstracts in the Astrophysical Journal and the Journal of Physical Review, Fulcher again reported in Science on his study of some 800 readers undertaken by the Division of Physical Sciences of the National Research Council two years after the introduction of these abstracts.

The report indicates that better than 88 percent read the abstract before reading the article, and that 45 percent report that they read many abstracts instead of articles, emphasizing that quality abstracts allow selective attention to that which you want to find. He commended the utility of abstracts across scientific fields.

Just a few years later in 1924, Albert Jenks made the same point in Science in an article entitled, "A Suggestion for Abstracts for Anthropological Literature." Jenks at the time was chairman of the Division of Anthropology and Psychology at the National Research Council. Praising the value of the complete abstract that he noticed was used in the biological sciences, he criticized by encouraged anthropological journals to strive for abstracts that provide more coverage of the data or entities in the article.

He estimated that perhaps occupying 4-5 percent of the journal article with the abstract was about in the zone he sort of envisioned, to have a complete abstract, as was done in the biological sciences. He emphasized the advantage of complete abstracts, giving researchers much more scientific knowledge than they could possibly read in article form, saving their time in not reading works of little value or relevance, and improving their habits in writing and presentation through exposure to more logical abstracting techniques.

This brief history is instructive, because it reveals how much emphasis was placed on the value of systematic, complete abstracts to science across fields and methods of inquiry. These early articles also emphasized that abstract writing is a skill where editorial staff or professionals may need to assist and play a role. Thus, there is no assumption that the review process should focus on the quality of abstract in assessing the strength of the article, but emphasis is on insuring that a systematic abstract is available as a predicate to publication.

Second, what we need and need to know about structured abstracts. Mosteller, Nave, and Miech point out many of the advantages of structured abstracts. While their article recommends a particular form and format for doing this work, there are devils in the details that require collaborative work by journal editors, authors, and journal publishers, including research associations.

While the ER article provides a useful example of a structured abstract, and commends a framework of nine elements, the authors themselves point out that different specifications may work best in different fields or subfields, that there needs to be evaluation of different approaches, including the unintended consequences, both positive and negative, on the substance of research in an article or oversimplification of its use, and even how the length of a structured abstract may affect use by different audiences, and its ready accessibility through different electronic mediums.

There is also some suggestion in the article that the presence of structured abstracts in different journals might encourage databases like the ERIC system to aggregate abstracts and make them accessible online. The strength of an archive of articles and abstracts as in ERIC or another archive like the National Criminal Justice Research Service, which is also an archive of articles and abstracts, is that they seek to arrive at some standard system of classification with benefit of field experts in professional abstracts to guide and assist.

Where such a database to be built on disparate sources of even structured abstracts, it could reduce the value of these informative systems and degrade their use. It is these kinds of issues that merit attention by working groups of editors, authors, information systems scientists within and across social and behavioral science fields.

Pilot tests examining intended and unintended consequences could be pursued in the finest Donald Campbell tradition of reforms as experiments. As part of this process attention needs to be given to where and how to focus effort in building an effective infrastructure for structured abstracts.

The new ERIC sets forth some criteria for determining whether a journal should be included in its database. The JSTAR archive that now includes education research journals included AERA's journals also has criteria it uses for inclusion in that archive. The ER article notes a very high number of some 1,100 education journals that would benefit from structured abstracts. As part of this process of experimental reform, it would see wise to focus on a subset of peer reviewed research journals in education and what high quality abstracts should include.

Some further reflections from the executive director cum publisher, the final section. This NRC workshop on structured abstracts is likely is ripe with issues important to building and advancing the social and behavioral sciences, facilitating enhanced interaction and integration across fields and disciplines, and widening the opportunities in particular for effective scholarly use.

The Mosteller et al ER article makes reference to efforts in the social sciences and other fields to use structured abstracts. Hartley's research for example cited therein, points to the feasibility of structured abstracts for different types of articles, and for those using different methods. From his research Hartley also conveys a general positive picture of the value of structured abstracts, although noting some of the methodological flaws and the limited number of studies.

Equally important are the issues that need to be addressed as an integral part of considering structured abstracts and the quality of published work. Part of the strength of the NRC October report on advancing scientific research in education, which is applicable really in most respects to all of the social and behavioral science fields, is that it connects issues of data quality and data infrastructure, publishing and information access, and research training and education as part of advancing scientific knowledge, and building the knowledge base in a field.

As researchers, journal editors, publishers, and various public consumers we need to assess the relative value of investments in certain forms of reform. Structured abstracts seem promising and worth exploring and evaluating options. I would hope that we at AERA, with others from the American Sociological Association, AAA, anthropology, political science, American Economics Association, American Statistical Association, and APA and APS would be conscientious and collaborative about rethinking abstracts and how best to proceed.

And I want to kind of give the rationale underlying that alphabet soup. And that is that it really made a difference, quite surprisingly to me, in examining the early history in the social sciences to identify across fields comparable classifications when the social science abstracts evolved in 1929 under the imprimatur of the Social Science Research Council.

And a lot of the writing of the subgroups from different disciplines, they themselves were rather staggered that they could take different theoretical and empirical and methodological modes of proceeding, and come up with a classification system that could inform that abstract system. So, I think some collaboration would be a help and a plus.

With advances in search engines, information technology, and information sciences, and growing synergism across social and behavioral science fields, at least in the context of some big themes and issues, it could certainly make a difference again. Questions will still remain for the social science community about how best to undertake both research, and make translations that can have useful purposes in policy and practice.

Structured abstracts would seem to be most advantageous for those researchers and other mediators involved in knowledge translation, in which I take it you are referring to as third parties, and those are third party individuals across third party institutions or data sets or databases.

There is no doubt that improvements in abstracts could potentially help all user communities, so it is surely worth the work. Nevertheless, considerable other work remains for journal editors and publishers about how best to insure rigorous review, how to articulate reporting standards without narrowing or constraining either what or how information is reported, and even what gets published in what outlets.

In a recent 2004 article on randomized clinical trials by Timber and Dickerson in the Journal of Internal Medicine, considerable concerns were expressed in the medical sciences about the absence of information necessary for critical assessment of a study's internal validity, concerns about the disproportionate publication of positive versus negative or null effects, the need to archive not just data, but information on procedures and protocols, problems in preserving access to electronically only publications to name just a few.

In the social and behavior sciences, including in education research, these and other equivalent challenges exist. So, as we continue our conversation and our journey today on how and how best to analytically and conceptually improve article abstracts, we would be wise to examine how improvements in this tool fits with other ambitions that we share with scholarly publishing and communication.

And I look forward to the opening question session, and the other sessions to follow, where I hope we can unravel some of those issues.

Thank you.