Why We Need Structured Abstracts in Education Research: Presentation of a Recent Proposal - Bill Nave, Consultant, Maine Department of Education

DR. NAVE: Good morning.

I'm a little nervous, because I left my wife at home with my brand new Toyota Prius, which when I left, had exactly 100 miles on it. My wife is a furniture maker. She drives a four wheel drive pick-up, stick shift. That's what she has always driven. She doesn't like automatics.

How many of you drive a Prius? All right. I'm joining your club.

I'm nervous because when I handed my wife the key, it's not a key. It's a little -- you know those automatic things that lock and unlock your car? That's all it is. There is no key. I told her to put it in her pocket, go out to the barn. She reached for the door, and I had unlocked it as she reached for it.

She got in and sat down. I got in on the passenger side. And she said, "Now what do I do?" I said, "Well, start it." There is a button on the dash that looks like the on button on your computer, that same little symbol. I said, "Put your foot on the brake, push the button." The lights came on. This computer monitor came on, and she said, "I don't hear anything."

I said, "Well, take the parking brake off, put it in reverse." "Where's the gear shift?" "See that little lever there? Just push it to the R and let it go." And then she felt it start to back up, put her foot back on the brake and said, "I don't like this."

I'm nervous about leaving her with that while I drove her truck to the airport. I didn't want to take my brand new Prius to the airport, for goodness sakes. So, I hope when I get back home she is okay and my car is okay.

The rest of the talk will be -- and by the way, don't forget this Prius story. I'm going to return to it at the end. The rest of the talk is in four sections. First, what's behind this story. Prof. Mosteller and Edward and I didn't just sit and down and say, let's write an article about structured abstracts. There is a story behind the story. That's what we'll do first.

And then once we decided to do this article, what was our thinking behind it? What was our thought process? We want to open that up to you, because that informs the structure we ended up with. Then we will describe that proposal, which you all have in your packets. It was published a year ago in Educational Researcher. And then our conclusion and our suggested next steps.

So, what's behind this? Several years ago when I was an undergraduate student at the Howard School of Education, we had the great fortunate to be conscripted -- I'm not sure that's the right term, by Fred Mosteller to work with him on some education issues. And that collaboration continues. We still talk once a week via telephone on current articles we are working on.

But at the time we were working on a search for, just as Prof. Boruch suggested, how many field trials in education have been done. And we were using ERIC for that search. So, when this faculty seminar at Harvard on evidence-based education was being formed, we got the invitation to be the kick off paper for that. And so, what we presented later became the article, "Rare Design: Where are the Field Trials in Education Research?"

Well, we had tried to find how many field trials there were. We asked a lot of folks on campus. They could name one or two, Perry Preschool, Pygmalion in the Classroom, but that was about it. So, we set about our own search first using ERIC.

And what we did with ERIC was keyword searches as Prof. Boruch suggested, randomization, field trial, experimental, several combinations of keyword searches, and we could never find more than somewhat less than 1 percent of published education research articles abstracted in ERIC that seemed to describe field trials.

But based on the feedback we got from that seminar, when folks expressed some doubt, one, that the number was so low, and two, that our keyword search was the effective way to do it, we proceeded as Prof. Boruch suggested he has done, of doing a more systematic approach.

We started with dissertation abstracts, because we thought that well, if those are dissertations, the advisors of those students writing those dissertations would probably make sure they had an abstract that contained the right information on it. So, we did a random selection of 200 dissertation abstracts each from 1992 to 1994, looking for randomized field trials that focused on interventions inside classrooms K-12 that might inform and might provide evidence for better classroom instruction, 2/10ths of a percent.

Well, we tried a random sample of ERIC abstracts from the same three years, and we didn't look at abstracts. We hand found in the library, all of these 500 articles. We read the articles. The simple focus was on is this a randomized field trial or not? Again, the same screen, K-12 classroom-focused, intervention in the classroom, 2/10ths of 1 percent.

Well, then we decided to take another approach. Let's do a census. Let's read every single article over this five year period, 1994-1998 in AERJ and EEPA and see what percentage of those articles described field trials in education. We assumed that because of the nature of those two journals, we would find lots of field trials. Well, by comparison to 2/10ths of a percent to 4 percent, we did, but still, not a very percentage.

Well, one problem we kept coming across throughout our search was keywords were misleading. Abstracts were not helpful. In some cases definitions across articles even in EEPA and AERJ were inconsistent. For example, some authors said a randomized field trial includes random assignment of classrooms, or random assignment of schools, where other authors said no, no, it can't be a randomized field trial unless it is random assignment by student. So, there wasn't agreement we saw in the field for us to be comfortable with what percentage are we finding.

So, these I guess you would say frustrations in trying to do a search for a specific idea, and finding it to be very time consuming and a bit of a challenge. It got us to thinking, even a simple yes/no question we found not easy to answer, not by the abstracts, not by the keyword search, in some cases not even by reading the articles.

Some of the articles we found called the study that they described an experiment, but then did not say if there was random assignment, didn't say how the units were allocated to treatment/non-treatment. So, was that a random trial? So, the articles themselves didn't provide enough information for us to make that determination.

And then motivation. Well, we had the motivation to keep going, because we wanted to know this. Time for a little bias confession. I'm a practitioner. I taught for 25 years. I taught drop outs. My goal was to make sure my students learn and are ready for life. So, a little side story here.

Beth, a dedicated fourth grade teacher. We have a follow-up article to our structured abstracts coming out in this month's CPN(?). And that article opens with the story of Beth, a fourth grade teacher in a medium-sized city in the Midwest.

She gets a call from her sister who is a fifth grade teacher in a medium-sized city in the Southeast, who describes a conference that she just went to where one of the speakers, a local teacher's college professor describes peer tutoring in reading in elementary classrooms, and the tremendous benefits it has both for the peer tutors, and for the students in the classroom who are being tutored, and suggests that Beth ought to try this.

Well, Beth just finished a master's degree a couple of years ago. She thinks that's a great idea. She drives downtown on Saturday to the local university teacher's college library, gets on the ERIC system, and starts trying to find articles on peer tutoring, because she had asked her sister, well, how does this professor know that this works?

Well, he didn't provide references, and Beth's sister didn't know. So, she said, go to the library. You can find it. And what Beth found when she typed in the keywords peer tutoring and reading was thousands of hits; not very helpful. She tried various combinations of keywords, and finally at the end of a long day somewhat frustrated, and she was able to find two or three articles that might be of help, and she went back to try this out in her classroom.

Now, as a practitioner with the Maine Department of Education trying to support practitioners in improving their practice, that's a roadblock to getting information to them. We'll come back to Beth. Don't forget Beth either.

So, one of the issues that is creating a roadblock to access research information is time. Your book spent this whole day, Edward and I spent months combing through articles, looking for information. And then once we found an article, I probably would suggest that Edward's and my experience in scanning articles to find information, the articles may have given us an edge in terms of taking less time to get information on an article than it took Beth to get information from the articles that she found.

And by the way, she also found that a lot of the journals that had articles that she wanted to access, were not in that particular education school's library, nor were they available online at the time Beth did her little search.

And the organization of the article, I mentioned this a little while ago. Some of the articles that Edward and I read over this period of the last decade were better organized than others. Some were more complete than others. Some completely left out any methodology section, so we couldn't infer how they did their study.

And then at some point in our conversations, Edward and Prof. Mosteller and I had this aha, medical journals have structured abstracts. They seem to be helpful. Why can't we do that in education? Well, at first Prof. Mosteller cautioned us against making this medical-education comparison. Educators don't like to be compared. We are not scientific, and so on.

Yet, on the other hand, Edward's wife is on the faculty at the Yale Medical School, so it's likely that Edward -- in fact, I know when those journals come in, Edward grabs them first before Dawn can get a hold of them. So, Edward knows what these abstracts look like.

Prof. Mosteller, as you know, has done a lot of work in medicine and public health, so this was part of his vocabulary. One of my part-time jobs to feed myself as a graduate student was working with Professor Emeritus Walter Egerman(?) at Harvard Medical School, MIT. He was one of the co-founders of the MIT-Harvard PhD-MD program.

His project was to find out where all of his graduates were. Did they indeed become physician-scientists as the program hoped to do? One of the places we published those findings was as an a freestanding abstract, and to me that was -- a what? The publication is an abstract? So, that idea was in my mind as well.

So, then again as a practitioner I thought well, is the lack of structured abstracts in education research journals one of those impediments to getting findings in research into the field of policy and practice? And my immediate thought was well, maybe it is. So, that became part of our discussion.

Another part of our discussion was the realization that research findings don't self-disseminate, nor do they self-actualize. Somebody has got to tell the story, and the story has to be heard by someone. So, the story is told at a conference in a presentation or in a publication. Well, if the right people aren't there and it goes no further, what good has it done for me and my drop outs in my classroom in Maine? Obviously, research results don't self-actualize.

So, given this sort of background, we think there are at least three limitations -- there are a probably a lot more we haven't thought of -- on getting research into the hands of practitioners. Hard copies of articles, Beth found when she was in that library that it didn't subscribe to some of the journals listed in ERIC that had articles that she wanted.

Online copies may not have been available in that library, because that library didn't subscribe to the hard, so they wouldn't have access to the online copy. And Beth would only have access if she subscribed. She didn't, clearly.

Fortunately for Beth, her sister, a trusted third party at least had told her about this idea. Edward and I frequently email articles back and forth to each other, or URLs back and forth alerting each other to interesting things to follow-up with. So, not all educators have a trusted third party.

One of my jobs with the Maine Department of Education is professional development for educators, so I bring articles, I bring URLs, I make copies. So, in some ways I'm that third party to disseminate information to them.

So, given this background, here is our proposal, structured abstract -- and you have this in your packets -- with several components, not all of which fit all kinds of articles, all kinds of journals, not all of which fit all kinds of studies. But those that do fit for a particular journal or for a particular kind of study would certainly make it easier for practitioners to access information if a structured abstract was online, on a journal's Web site, if ERIC had structured abstracts.

It would have made my job and Edward's job in doing this search much easier. It would have made Prof. Boruch's search easier. And I suspect -- Bob can correct me if I'm wrong -- that he didn't personally do this. He probably, like good professors, has a cadre of graduate students like Edward and me, actually doing the work. You don't have to answer that.

As I read the comments of the first panelists who will follow this, I thought of the comment about Prof. Ogbu, whose thought was we need to fix society before we can figure out how to teach kids in classrooms. And when I started out teaching in Bedford Sirus(?) and Oceanheld(?) Brownsville in New York City, I'm glad I didn't have the thought of golly, we need to fix society before I can teach my kids in this classroom.

My assumption was if I do things right, engage the kids, personalize what I'm doing with them, they will learn sixth grade science, seventh grade science. And it was my experience that they did. So, my bias again is as a practitioner. Well, we'll return to this.

So, this model of a structured abstract, does it address some of the concerns that led Edward and Prof. Mosteller and I to this proposal? Let's have a look. Our initial search, a simple yes/no question, does this article describe a study that was a field trial, yes or no? A structured abstract would say that in the methodology section. It would certainly have saved us a lot of time.

Beth spent a whole day trying to find to support her teaching reading to her fourth graders. If the ERIC system had structured abstracts, she could have quickly narrowed down her search to a much smaller set of articles to look at among the thousands that she got from the keyword search. And that would have saved her enough time to spend with the articles she did find, to get the information she wanted.

Well, it doesn't do anything with the self-actualization or self-dissemination of research findings, but it does lower the threshold to allow those findings to get out to practitioners more easily.

Accessibility issues. If ERIC and online journal Web pages had structured abstracts, it would lower the threshold for this kind of problem as well. So, we think that structured abstracts improve access, and make it easier to assess which articles to follow-up with, to get hard copies of, to track down. And it gets past the limitations of the current system.

Potential outcomes. These are the sort of dream outcomes that we thought might be possible when we were discussing and writing and rewriting this article. ERIC, if it uses structured abstracts, becomes a lot more useful. Now, a slight aside here. We are not suggesting that somebody needs to go back through all of ERIC and write structured abstracts about all the articles. We are suggesting that from here on out maybe this would be useful. I don't think there is enough money to hire enough graduate students to go back and do all that.

If journals had structured abstracts on their Web pages, we think that would increase the number of requests for articles in their journals, which would give those journals a broader representation in the field. It certainly would be much easier for me to send emails to Edward with a structured abstract, saying Ed, take a look at this article. And for those of us, and those graduate students who do reviews of research or someone doing a meta-analysis, this would save a bit of time for folks like that.

Quality control issues. One of the issues we thought about was if the format of structured abstract was in front of an author before he or she submitted to a journal, at least would assure that all of those pieces would be there, methodology sections wouldn't be missing, full detail about whether this experiment was randomized, and at what unit of analysis was it randomized?

Busy editors. Edward and I had the pleasure maybe, the challenge of working on the Harvard Education Review as student editors for two years during our stay at the Education School. The editorial process we found was very time consuming, not an easy task at all. If HER had a structured abstract format, then we wouldn't have had to try to come up with those little paragraph introductions to convince folks to read this particular article. And it also would have made sure that we knew what things to make sure authors had in the article.

So, it would have provided guidance for us. And we think if a journal adopts a structured abstract, has that guidance for authors, it's the author who writes the abstract. It becomes part of the whole editorial process. Provide some guidelines so things don't get left out. And reviewers of the articles would also have a framework.

So, where do we go from here? One way I think to think of a structured abstract is that it's a bridge, a bridge from a title to the full text. And an intermediate, a hybrid if you will, like the Prius. It stands between both worlds. It can stand alone as an email attachment to send to a friend. It provides enough information for a reader to know whether that article, one is of interest, and two, has the details the searcher is looking for.

It's also a bridge from research to practice. Again, I'm a practitioner. And this would help me find the right articles to get into the hands of the teachers I'm working with, and the principals I'm working with.

Benefits. Potential benefits for journals, quality control, I discussed a little while ago. Increased dissemination of the articles, because of the online presence of the structured abstracts.

For authors, potential wider dissemination of the author's work. So, there is an audience question there. If the author's target audience is other scholars in that field, and this is of theoretical interest, does a structured abstract fit? Maybe not, but from a practitioner's point of view, if I am writing for practitioners to use my research findings to improve teaching and learning, I want my work disseminated more broadly.

Consumer of the research like Beth, like us, improved ability to get a hold of the articles that will help us. And again, here is my bias. For all of us, all of us citizens who pay our taxes to support education, I definitely want the best research in the hands of educators, K-12, so that they can do the best job possible for my children and my children's children. I want kids in Maine to graduate educated, ready to do high intellect jobs, because the paper mills are disappearing. You see the picture.

So, here is our summary. Oh, the rest of the story on my Prius. Joyce was very nervous getting into this car, so when she backed out of the garage, I took her on a five mile trip. So, after she backed out, turned to leave the driveway, I said, "Okay, put it in drive."

Well, she had just clicked that little lever to the R. Now, she clicked it to the D, and let go of it, and it popped back into place. And she, "I just drive?" I said, "Yes, just drive." So, her first comment was, "It's so quiet." In fact, she didn't feel the gas engine kick in until we climbed the first hill. It was running on the battery.

We got up to the stop sign. I said, "Okay, hang a right. We'll go down the highway here." "On 202? On the highway?" "Yes, and you're going to have to floor it, so we can get up to speed here. This is 55 miles an hour on this section." Well, she did, and the gas engine kicked in, kicked off depending on whether we were level or going up a hill.

Three or four miles down the road I said, "Okay, take the next right, we'll go back home." "Where's the turn signal?" "Same as on your truck." "Oh, okay." Turn the turn signal on. So, we got back home. She pulled it into the barn. I said, "Okay, turn it off." "But the key is in my pocket." "Push the little P button for park. Put the emergency brake on. And then, how did you turn it on?" "That button." "Push the button."

She pushed the button and everything went off, the lights went out, the computer screen went off. She turned to me and said, "That wasn't so hard. The only thing different was how it starts."

So, we hope someone will implement structured abstracts, and then evaluate how they work. This is what the journals in medicine, which only started this in 1987, I believe, they didn't do an evaluation with baseline data as to how the structured abstracts worked. Did they follow this theory of action and actually help with the access and so forth?

We in education know better. We'll evaluate this from the start, and see how it works, how to improve it. So, take it for a test drive. In fact, today maybe as test drive of the idea, and maybe we'll find at the end of the day the only difference is in how it starts. The structured abstract is the bridge, is a hybrid, online hard copy. And there is your model. There is our suggestion for a model.

DR. BORUCH: Thank you, Bill.

Felice.