MR. RAUDENBUSH: I want you to know that I developed my comments completely independently of Margaret. Yet, you will discover some inter-discussant reliability here.

Bob Floden and I co-ran a task force at Michigan State several years ago. It is interesting to hear how you went.

In thinking about this talk, I got to thinking about the tasks engaged with methodological training at the places I have been, the University of Michigan, Michigan State, and where I was a student, at Harvard.

They seem a little bit daunting upon reflection, and I identified three of them. The first is to develop what might be called broad literacy across an amazingly diverse array of methodological perspectives, including surveys, experiments, ethnographies, multi-site case studies, as well as assessment and psychometrics.

This is distinctive in schools of education. Economists don't have to know anything about ethnography and cultural anthropologists don't have to know anything about time series, but we expect doctoral students in education to be literate across this array of topics.

Secondly, we have to ensure that our students develop a high level of expertise in something. They have to be really good if they are going to be successful in classroom ethnography, or in the analysis of large scale, longitudinal data sets collected by the National Center for Education Statistics, or you can fill in the blank.

This is a task that I feel we can't do alone. We have to rely on the rest of the university to help us with this, because we don't and we can't expect to have the expertise within a school of education to foster this expert development in all these diverse areas.

The third task is to produce the methodological leadership of tomorrow, to do the methodological research that has to be done.

This is very timely now, because we are operating at a time that there are higher than ever stakes on the kinds of assessments we are using. They are more pervasive. Therefore, the requisite demands on reliability and validity are tremendously high.

There is actually, I think, rather than worrying about our public influence, there is a tremendous public demand for knowing what works, which translates, in my world, into causal inference, and to knowing when it works and for whom, which I translate into so-called causal generalization. We also might be interested in knowing how it works and why it works, which requires other methodologies.

By the way, this is a task that I don't think we can outsource. We need collaboration with the rest of the university, but we cannot outsource, and I will say more about that.

When we face daunting challenges, sort of where do we go and what do we do. I guess, in my view, I think we look at our history, we go back to our roots.

Now getting a little bit more specifically into my own roots, I looked, then, back over a century of research on psychometrics and educational statistics.

What I see, in contrast to what people often seem to bemoan in educational research, is a century of tremendous public influence and the development of tremendous methodological breakthroughs in sophistication.

There is probably no innovation in the history of social science that has had a broader impact on society than testing.

This, and the methodological work, much of it has been done in schools of education. By the way, you might bemoan this fact and despise this fact, but the influence is undeniable.

The only possible competitor, I think, would be the development of the sample survey, in terms of its influence.

Along with the public influence has been a series of fundamental contributions to statistical theory and practice.

Going back a century to the work of Spearman and Pearson on correlation, or Sir Francis Gaulton, his eugenics tendencies not withstanding, developing factor analysis, moving along into the mid-century, looking at the work of Lee Cronbach, the school of education, Stanford, on classical test theory reliability, generalizability theory, that has had pervasive effects on social science.

More recently, many of the most widely used modeling techniques that are just now having big influences across the disciplines outside of education, include item response theory, developed primarily by educators and educational psychologists working with them, people like Melvin Novick, Darryl Bach, Benjamin Wright, and many others. I could name Ron Hamilton, Swami Nathan, many that I could add, Brian Junker more recently.

Structural equation modeling, an area that is probably weighted a little bit more toward psychology, but some of its major contributors, such as Bank Mutan(?), coming out of education.

In my own area, hierarchical linear models, which have had a vast influence on social science research, the work that has been done there has been primarily done in schools of education here, as well as at the University of London and in the Netherlands, at Gronican University and the University of Tuente(?). The most widely used books in this area come from those countries, all from schools of education.

Even when I think about a man whom I greatly admire, and I think is one of the two or three greatest statisticians in the last 50 years -- Donald Ruben -- his work on missing data and very closely related causal inference -- I don't know if you connect those, he showed those are the same problem.

Donald Ruben, of course, did his work at the Harvard Department of Statistics, and then went to ETS, where he developed the missing data and causal inference methods.

He then went to Chicago, where he was jointly appointed in the department of education there and the department of statistics, before returning to become the chair of the department of statistics at Harvard.

It is really a quite wonderful tradition. We have a lot to draw on. Why is it that this group has been so influential and, I think, in many ways successful?

It can't be because of size. When I compare biostatistics to educational statistics, it is like comparing an elephant to a mosquito in terms of size.

It seems to have to do, rather, with some other things. One is the pressing practical nature, the fact that you can be sued for bad methodology, which explains why ETS knows more about test bias than anyone.

The fact that research in education is very challenging, it poses actually challenges that are greater than those in other disciplines.

Then the third fact, the people who have been trained as statisticians with PhDs in statistics have been hired in reasonably large numbers in schools of education.

As a result of that, there is more knowledge about probability in mathematical statistics in many schools of education than there would be in a department of psychology or sociology, where a PhD in statistics has no hope for getting a job.

Moreover, there is a history of collaboration in which people trained in education learn a great deal about probability and mathematical statistics in departments of statistics, and work closely with statisticians.

So, I think we have a great deal to draw on in terms of our history, and I think we can learn lessons from our history that we can apply to the current challenges.

What are they? The biggest challenge -- well, the assessment challenge, the psychometric challenge, goes on. It won't die.

The new challenge is to discover, as I said, what works, for whom, and under what conditions, which really involves causal inference.

It involves a need to link those who invent educational innovation and design practices and test them in local experimental settings, for those to collaborate with statisticians and very rigorously trained people in large scale evaluations of these things.

So, I would like to, with these challenges in mind and this history in mind, make four recommendations that we might think about, four goals that we might set.

The first is to train a new generation of quantitative researchers and methodologists in education. Actually, I am going to focus on methodologists.

I want to emphasize here that we cannot simply apply that which is handed down to us by biostatistics or econometrics.

The reason is that causal inference, which is really at the core of this, is different in education than it is in medicine. This is something I have been working on pretty hard lately.

In medicine, a key assumption is, if I am comparing a medical treatment to heart bypass surgery, if you are assigned to heart bypass surgery, it has no effect on my outcomes.

In education, if you are retained in grade, I have different classmates. That is a fundamental difference. That means that who else is assigned to what treatments affects the outcomes of each person.

The second thing is that, in medical research, if we are comparing two drugs, we assume that if doctor A gives a drug and doctor B gives a drug, it is going to have the same effect.

We can never make that assumption in education, because everything that we do instructionally, our key treatments or our structural treatments have to be delivered by -- that is too strong a word -- they have to be enacted by teachers with students in classrooms.

That means that the agents are crucial, and the potential outcomes of the students depend on who the agents are and who the classmates are, rather than this simple potential outcome idea that, if there are two treatments, I have two potential outcomes.

So, we have to work closely with people who have led the way in causal inference and other disciplines, but we have to be actively engaged with the subject matter of education in order to do the methodology right.

The second recommendation is that we have to reject the idea that anyone can teach statistics, along with the closely related idea that teaching statistics to the broad mass of students is an irritating thing that methodological people sometimes have to do.

In order to illustrate this point, I would like to talk a little bit about lineage, which seems to be so important, since we don't train people to teach in higher education. We have to rely on role models.

My role models were Dick Light and Tony Brike(?), masterful teachers at Harvard. Dick Light's role model was Fred Mosteller, who is probably the most famous teacher, someone who made it a lifelong calling to make this subject matter accessible to the public, while doing cutting edge research, of course, himself.

Tony's mentor was William Cochran, another renown statistician, who made his life calling also making his ideas publicly accessible to the broad public.

Continuing that lineage at the Harvard School of Eduction are Judith Singer, whose mentor was Fred Mosteller and, by the way, Dick Light was on her committee and, by the way, trained in a department of statistics, working with John Willet, from the school of education at Stanford, who had very rigorous training in probability or mathematical statistics.

Willet and Singer exemplified the kind of teaching I am talking about, because it is informed by deep understanding of subject matter and a deep commitment to make the ideas accessible. We have to pursue this very explicitly.

Third, we have to think about how we coordinate quantitative and qualitative methods. I think that Margaret's ideas make great sense, and I could give another talk on this, but I am not going to.

Fourthly, I think we have got to think about the possibility of reorganizing how we do methodology. The traditional model -- well, of course, I said it was so successful. So, why do I want to change it?

The traditional model involves having quantitative people, particularly psychometricians, embedded within a department of educational psychology, which is embedded within a large college or school of education.

The model there is a self sufficient sort of idea, that this group would have all of the expertise needed for the methodological challenges.

I don't think that will work any more, and we are not doing that now at Michigan. For one thing, we are not big enough to do it. We don't have enough people. We don't have the size to do it. I think we are actually better off not doing it.

Instead, what we have developed is a dual degree program in education and statistics working with people in the department of statistics, as well as other quantitative social scientists around campus.

Our doctoral students who are doing methodology have two social networks. They have a social network in a school of education where they learn about the subject matter and they talk to people who, if they haven't themselves taught, have taught in classroom, and they are in seminars where these issues that David Labaree talked about are discussed.

They also have a campus wide social network, which includes other students in dual degree programs in sociology and statistics, psychology and statistics, and the departments of statistics and biostatistics. There is a campus wide network.

This creates the possibility of rapid dissemination of methodological innovations throughout the social sciences, from which I have found, in the last couple of years, educators have a great deal to learn and an enormous amount to contribute.

I think that is pretty much what I want to say. We can't do it alone. We have to work closely with our closely with our colleagues across the disciplines, across the university, but we absolutely cannot outsource this, either. Thanks.

[Applause.]