Workshop on Understanding and Promoting Knowledge Accumulation in Education:
Tools and Strategies for Education Research

Day 2 – June 30, 2003

Agenda Item: Ways of Taking Stock: Replication, Scaling up, Meta-Analysis,
Professional Consensus-Building - Dr. Laurie Wise

DR. LAURIE WISE: I’m going to talk to you for a few minutes about professional consensus building. In some ways this is sort of the qualitative flip side to the more quantitative approaches to meta-analysis that you’ve just heard about and this is a little painful for me because I’m primarily a quantitative researcher, measurement guy and so on. However, I have been privileged to experience the National Research Council’s process of forming committees and as a means of building a consensus around a topic, and so I’m delighted to talk about this.

It’s not, at least my intention is not necessarily it’s an ad that you should have more, therefore fund more committees, although not that there’s anything wrong with that if you’re a sponsor out there. But what I’d like to do is offer this discussion as a stimulus for thinking about how we can scale up or find more wider applications of the idea of building consensus around findings, issues and solutions, rather than continuing to perseverate in the perception that the field is widely divided and can never come to any useful conclusions.

I started this sort of like a recipe and then I decided that Martha Stuart is no longer my role model so, take that with it. But the first point I wanted to make is that the NRC committees are formed around an important problem, somebody asks for this study to be done. Quite often it’s written into Congressional legislation. I was involved with a committee evaluating the voluntary national test that Congress wanted in evaluation as a way of trying to settle some tensions and some disparate opinions about how to proceed with this task. Other times agencies will identify the topics or occasionally the topics and the committees are launched based on a dialogue between staff at the NRC and people at different funding agencies, including foundations and so on. But the important point is that the committee is there because somebody wants an answer to a question, not because it was internally felt that this would be a good thing to do.

The key element of success is the process that the NRC uses to put together a good committee. The first aspect of the committees is that they invariably have very diverse perspectives, people are brought in from different fields who have different ways of looking at problems and that allows committees members to learn from each other but it also prevents people from taking a narrow view of the problem. So on this committee here, for example, we have people from a wide variety of fields, ranging from physics down to sociology and some of the softer sciences, don’t go there.

The committee selection process has a review before the committee even is officially named that checks to see that there’s a balanced representation of views, so if there’s an issue on which people are divided the committee will not have all the people pile up on one side and no representation from the other. There also is several steps to ensure that the committee is free from bias, there’s an extensive process where you fill out forms, you discuss with each other, not just financial biases, financial biases are obvious if you’ve got some stake in a company that’s doing something that’s being discussed or evaluated, but also research or methodological biases, so if you have published a position that seems relatively intractable that gets aired and that may be a basis for not including people on a committee.

And while those of us who serve on the committees like to talk about the committees as doing all the work, the heavy lifting, of course, is really done by the staff and in particular by the study director who devotes pretty much full time to bringing together information, presenting it to the committee, challenging the committee to come to consensus about the information that they have been privileged to review.

The third step is that the committee process is not a one shot deal, especially because of the different perspectives that people come with it’s necessary to take some time and dialogue over several meetings in order to fully understand the ideas and to think generally about how to apply some of the different perspectives to the problem at hand. I would say also that in addition to time it takes some financial resources to make this happen, the logistics of getting the committee together, of commissioning papers, of conducting workshops, is not a trivial expense. Having said that it is the case that committee members serve without pay, without being compensated, that’s another of the steps to prevent sort of bias or financial interest. We do carry around a little sign saying will work for food, and our basic unit of pay I think is the chocolate chip cookie. The committees gather information in a wide variety of ways, we conduct workshops such as this where people who have been active in the field come and present discussion of their work, commission papers, and occasionally the committees will go out and conduct site visits or do direct observations.

After gathering the data the strong push is to achieve a full consensus of the committee. On rare occasions there are dissenting reports but for the most part the committees I’ve been involved with, notwithstanding some diverse and widely firmly held positions, end up achieving a statement that everyone can sign on to. One other important aspect of what the committee produces is is not just as in the outcome of a meta-analysis, sort of a factual statement about the relationship between measures of general ability and job performance or some other sort of factual outcome so much as the beginning by placing in context the issues and the findings, drawing specific conclusions from all the information that’s been gathered, and perhaps most importantly making recommendations for action, including of course further research, but also for policy actions that Congress or the various agencies might take in response to the issues at hand.

When the NRC committees write up their conclusions and recommendations there’s help and assistance from the reporting staff that focuses on talking to a diverse audience. I mean we’re frequently talking with policy makers more than with just ourselves, and so it’s important that principles be clearly explained and that the audience, a wide audience be engaged.

Then the final step in the committee process is that everything that the committee comes up with gets further reviewed by additional peers, usually people that were of the same nature as those who served on the committee itself. In my experience it’s always been five or more reviewers, you get the reviews back labeled A, B, C, D, E, F, G, H, I, etc. The reviews are taken very seriously, there is a review coordinator who is charged with seeing that the committee in fact responds to all of the questions and criticisms raised in the review, and the report is not released until the review coordinator signs off and says yes, you have addressed the issues that were addressed. There also is usually a natural member of the National Academy who’s a liaison that serves with the review coordinator to ensure that the evidence base for the conclusions has been thoroughly reviewed and in fact is acceptable.

So the question, and again, as I said the reason for talking a little bit about this is to ask you to think about ways in which the professional consensus building can play a larger role in knowledge accumulation, and the ways might range from simply highlighting important and persistent findings to suggesting applications for these findings, fruitful evidence for further work, and we’ll leave it as an exercise for you to think about and talk about how this application can be broadened.