MR. HAKEL: I want to talk about peering at peer review. The nice thing about peering at peer review from my perspective as an industrial and organizational psychologist is that I have been involved in issues of basic scientific research and application in industry, for hiring employees, promoting employees, training employees for my entire career.

So, it gives me a different perspective. It is nice to be at the table as a peer in this particular presentation.

What I am going to do, instead of making you see these slides, is just tell you what is on the slides. I want very quickly to give a critique of peer review, and then talk about the goals we are pursuing and talk a little bit about project A, which Laurie Wise knows about in great detail, because he is one of the people who was on that contract for the U.S. Army. I think it has some lessons for educational research.

I want to talk about applying the science of learning, make a couple of recommendations and talk about some grand challenges.

On to the first part, the critique. There are many interesting critiques. I had planned to show you the double helix structure of DNA.

The point here is simply that that manuscript was rejected from publication the first time it was submitted. It is easy to find other examples like that. The first law of thermodynamics. Camparino's article in 1995 has lots of citation classics that were rejected the first time out.

Then, in Chubin and Hackett's wonderful book -- I told Ed this morning how much I enjoyed that book -- I only came across it two-and-a-half weeks ago when I got the invitation for this presentation.

One must never say anything new in a grant applications. It is about the communities that we work in.

Margaret Eisenhart also has an article in Research in Science Education, 2002, detailing the vulnerability to politicization, favoritism and strategic maneuvering, the narrow range of perspectives, peer reviewers are defenders of the status quo, reviews depend explicitly on conscientiousness of the reviewer.

There are inconsistencies in review panels, where somebody likes it and somebody else doesn't, or you come back a second time around, you get a lower score than you did the first time when you were encouraged to re-submit.

Errors of omission and commission in terms of what gets above the pay line and what is below it. Then, of course, there is the ever-familiar impact of author status characteristics.

When I was a kid I learned the parable of the blind men and the elephant, John G. Sax's wonderful poem. I had planned to recite the entire thing to you. I will spare you that, given the time here.

Hal Argees(?) has a wonderful article in the January issue of Psychological Science, 2003. No psychologist is going to tell me how to evaluate proposals in my field, is one objection to peer review.

There is a science of decision making. Hal is president of the Society for Judgements and Decision Making, and characterizes a number of the objections to peer review.

Everyone can play this game, and they can play it with their gut, in fact. You have to know with your gut how to play it, and the whole key issue is making sure that the decisions come out the way you want them to come out.

In military officer effectiveness ratings over the decades, there has been lots of research on how to set up the efficiency reports so that you get the right answers.

You wouldn't be surprised to know that all the generals had exemplary ratings, and the colonels are just a little bit below that and so on.

Nevertheless, they are predictive over the long run in terms of who is promoted and who is effective in their roles.

Some other objections Argees has is, scientific data is not relevant. So, in peer review, you can dismiss entire fields of research as being irrelevant to the task that is being done.

The final objection, we don't want criteria.

Meta analysis is the new tool that has come along in the last decade, and I have a plug that I didn't drop in the instant data here, to try to get down into the five to seven-minute time limit. It is from Bernard Kline in Science 10 years, 12 years ago.

While meta analysis may be a step toward an objective summarization of the state of art in a given scientific area, I get the impression that at least some folks think the method somehow will facilitate arriving at scientific truth. At its very best, meta analysis cannot determine what is true. It can only determine what is.

If we constantly keep in mind that, had meta analysis been around at the time, it would have confirmed that planet earth was the center of the universe, we will be alright.

Sax's parable simply puts the emphasis on inter-rater agreement, inter-observer agreement as a critical kind of thing.

We heard from Whitehurst this morning about democracy is the worst form of government except for all those others that have been tried.

Let me turn to talking about Project A and some of the lessons that might come up from it that might be relevant in this domain as well.

Chubin and Hackett have a quote that begins one of the chapters: Science policy is turning out to be rather more difficult than science itself. Some of the difficulties may be arising because we are not sufficiently aware of the influence of science policy on the way research is done. Thank you.

It is science policy where basic science and applied science meet the difficulties of being in the real world, that is provoking a lot of difficulty.

Project A is a longitudinal study. There is a book that I commend to you by Campbell and Knapp. It originated in the aftermath of the Vietnam War, horrible attrition in the enlisted ranks. The all volunteer force wasn't working well.

There were persistent differences in adverse impacts of the EEOC guidelines. Then, there was a particular problem in the mis-norming of the armed services vocational aptitude battery.

There was an inadvertent natural experiment, in which 100,000 people were admitted into the military that shouldn't have been admitted, and the question was, how well did they perform.

So, Project A was a response to these forces, these general forces. For me, looking from the outside, it doesn't take too much imagination to think that No Child Behind testing requirements are going to lead to downstream -- maybe in the distant future, maybe in the near future -- need for a Project E dealing with educational issues.

So, Project A is cross section designs, longitudinal designs, altogether, 50,000 soldiers tested, relating predictors to multiple criteria.

For me, that is the key piece in this. It is not just how well did people do in training, how well did training predict performance on the job, but how did that carry through longitudinal research.

Educational work starting with cohorts in first grade or preschool, all the way through PhD seems to me to be something that needs to be looked at in the long run.

Project A is relevant here. There was a contractor consortium, and Laurie Wise was a member of the AEIR group at that time. He is now president of HumRRO.

The governance advisory groups are the piece that is important for peer review in this context. There were three of them.

I chaired the scientific advisory group. There were nine of us that met with the contractor over the entire life of the project.

The so what is that peer review continued, not just in the award of the project, but went on for many, many years as this got carried out. So, peer review of ongoing grants, ongoing projects, is something that I would recommend.

The other thing from Project A that is critical in my field is the finding that performance is multidimensional.

We all know that. I knew that when I was rejected from sandlot baseball teams or picked last. I was a pretty good hitter. I was never a very good fielder, and so, the decision would be, I knew, before I had the technical vocabulary, that performance was multi-dimensional. Project A put together causal models of performance, and those are in the literature.

Switching to the science of learning for just a few minutes, Anne Rowe was a clinical psychologist who, in 1960, talked about why educate.

Her point was, it is very, very global, much more than I would think about it, simply that we should be asking how do people develop competency in living, how do people become people.

This is where the schools come in. They shouldn't be skills oriented, they shouldn't be knowledge oriented and so on.

The key issue for peer review is, what is the level at which we are going to look at these objectives. I dropped out some slides about the Institute for Educational Science, its objectives, the NCER as a unit, an educational research center, and then a commissioner.

I simply want to talk about the science of learning, as something that is now evolving, and a key publication is How People Learn.

The models that appear in terms of learner centered, knowledge centered, assessment centered and community centered learning is something that is a critical contribution to research overall, and we need to find ways to instatiate it and play it out in many different places.

One of the items that was not included in that book was research by Kluger and DeNisi on learning from feedback.

Everybody who is a supervisor is told that feedback is very important. If you have ever had somebody who gives well meaning performance reviews, helpful hints to somebody has sometimes gotten a fight on their hands.

Kluger and DeNisi show that, even though there has been research on this for a long, long time, about a third of the time feedback makes the performance worse.

So, my concern for the institute would be in providing feedback on grant applications, to make sure that this actually works out well.

Time is going too quick, so I am going to skip over some of these. Chubin and Hackett have some process improvements that they recommend for peer review. I would like to endorse some of those, and call your attention back to the book from 1990.

Principle investigators being allowed to write rejoinders to their reviews before the award of publications. In some sense, the scientific advisory group for Project A's ongoing peer review during the conduct of a project might fit into this.

Reviewers signing their reviews, openly rewarding reviews, is important as well.

Normalization or standardization of ratings is another one. I disagree with recommendations in strengthening the standards, but that will come up tomorrow morning for discussion.

Disaggregated ratings and disaggregated panels is something that Hal Argees writes about, and I invite your attention to the rationale for that.

More generally, calibration and training of reviewers are things that are important.

Where it gets to be really difficult is when you have eight or 10 different focused topics, as the IES does not, and we will skip over the Gates Foundation, and come to design principle four.

In the Shavelson and Towne book, there is the issue of trying to create a focused and balanced portfolio of research that addresses short, medium and long-term issues of importance for policy and practice.

The trick is going to be to get conversations going across the boundaries that are there. Ed Hackett started it in response to a question to address that issue.

I think it is awfully important to have openness to creativity and openness to creative practice, and design a system that gets to those. Thank you very much.