Welcome & Workshop Objectives - Robert Boruch, University of Pennsylvania

DR. BORUCH: I'm Bob Boruch. I'm chairing this workshop.

Welcome everybody. It was arranged partly through the auspices of the Committee on Standards of Evidence and the Quality of Behavioral and Social Sciences at the National Research Council. Norm Bradburn, who is someplace in the room is chair.

That committee's agenda, broadly speaking, is to assist the Academy in thinking about the next 5 or 10 years and its role in enhancing both the quality and the integrity of evidence in the social and behavioral sciences. This particular workshop is a piece of a much larger effort. And I believe that's part of the National Science Foundation, as well as Hewlett and other outfits.

There is strong interest in so-called structured abstracts. There are a number of reasons why some of us have become interested in this stuff, although it may seem pedestrian to some. Certainly, the difficulty of assuring full and complete access to information in journals, whether they are electronic or otherwise is a major one.

A couple of years back we did a search for randomized trials in answer to major educational research journal from 1963 to about 1999 to 2000, looking for randomized trials in the math and science arena. We did a hand search backwards and forwards, that is to say looking at every article, not only on shorts, keywords, and the like for papers. We discovered only 35 out of 1,200 articles on that topic.

In doing a parallel search using ERIC we found only 1 of those we located through the hand search, 1 out of 35. And in using Psych Info we uncovered only 10. Now, the hit rate there is pretty low. It turns out to be the case that Megaline's(?) experience at the National Library of Medicine is similar. They, five, six, seven years ago missed at least 30 percent of the articles on clinical trials in the medical arena that were published by relying solely on machine engine-based search strategies, and the sort of evidential base for those strategies.

Generally, speaking a lot of the machines may search the abstracts and the keywords. The abstracts, as many of you know, are exercises in lexical promiscuity for many scholars. They are not peer reviewed. They are often opaque, often merely a table of contents. The keywords are similarly not taken seriously by many authors.

And partly as a consequence of the fact that there is an explosion of information out there, not to be too trite about it, understanding how to more readily access scientific information and digest it quickly is part of the aim.

So, that brings us to structured abstracts, and a proposal developed by Fred Mosteller, Bill Nave, and Ed Miech a couple of years back and published in Educational Researcher. The immediate questions before us are why should we take this thing seriously? How, when, under what conditions should structured abstracts be considered seriously, and for what kinds of journals or journal articles? Who would do these things? To whom would they be addressed, what audiences? Entirely scholarly audiences, as opposed to lay audiences? What are virtues and vulnerabilities?

Beyond that, these structured abstracts are merely a platform for thinking in larger ways about how to make scientific information more accessible to scientists certainly, as well as to the public, and to enhance its usefulness.

With that, the current session includes, Bill, who will make a presentation; Felice Levine, Executive Officer for the American Educational Research Association as the commentator. We've got time for questions and answers, and then that's the only thing that will stand between us and a coffee break.

Beyond that we've got a couple of roundtables organized around "key influentials," people who have a vested interest one way or another in the business of producing information, getting it used, and so on. The first set of key influentials included editors and former editors, defrocked editors of journals in the social and behavioral sciences. Their interest in this should be obvious.

The second set of folks with informed opinions, or who could develop informed opinions about this include those who could be classified as intermediaries or information brokers. This includes folks like Debbie Viadero at Education Week, journalists more generally and education writers, science writers, crime and justice writers, and the like. It includes folks from the Institute of Education What Works Clearinghouse. Terri Wilson will be here I think from AIR.

And the last session that Norm Bradburn will chair includes folks major responsibilities for professional associations and the like, including government responsibility for production of evidence. This includes Luna Levinson, who is the project monitor from the federal side for the development of the new ERIC clearinghouse, Doug Joubert from the National Institutes of Health Library, and Susan Harris from the American Psychological Association.

With that, let me introduce Bill Nave, who is a consultant to the State Department of Education Maine. He will make a presentation along with I guess Ed.