Workshop on Understanding and Promoting Knowledge Accumulation in Education:

Tools and Strategies for Education Research

Day 1 – June 30, 2003

Remarks by Dr David Cohen

DR. DAVID COHEN: I am speaking from a piece that is coming out in Educational Evaluation of Policy Analysis this summer that Steve Raudenbush(?) and Deborah Ball and I wrote together, and that piece on instruction resources and research or some combination of those three words is part of a large study that Steve and Deborah and Brian Rome(?) and I have been working on, a very large study of educational interventions in the high-poverty elementary schools in the U.S.

So a lot of what I have to say arises from our efforts to figure out how to think about the relationships among resources, loosely understood, what happens in schools and student performance.

In order to do this study - this is a longitudinal study tracking schools, tracking interventions into schools and their development, tracking schools’ progress in implementing the intervention, tracking teaching and learning and linking that to kids’ performance, literally linking it in specific classrooms on specific days.

What I have to say arises from our efforts to figure out how this animal works. So we are researchers who have been deeply engaged with practice, both in the sense of trying to understand it, to instrument it and observe it, and in the sense of trying to get it right, to do that in a valid way, which meant that although some of us had various kinds of experience with educational practice before launching into the study, we have to take much more of a - in the study than most of us have done before, and I am going to talk about this - about some of our ideas from the perspective that I did understand from the committee, that is to say knowledge - about education.

There is no doubt that knowledge has accumulated. I think it would be fair to say that there are boxcar loads of studies. If it were all collected on the Washington Mall, it would be a very impressive accumulation, especially if we went back behind Coleman et al and included studies that had been done in the ‘20s and ‘30s of which there were many.

The question I understood the committee to have was not about accumulation, not about the heaping-up of research, but about cumulation - that is to say, systematic knowledge-building - and I think, at least in the area that Sunny and I are talking about, there has been much more accumulation than there has been cumulation, and I don’t want to do any more than stick my rhetorical toe in the cumulative water, but I mean to refer, when I say “cumulation,” knowledge-building, including revision, within a scholarly community, such that one can - looking backward - see the growth of systematic understanding.

Now, that always entails dispute. Dispute is either the meat or potatoes of knowledge accumulation, but dispute within a knowledge-building community is different than dispute across community boundaries, and my casual, empirical observation of educational research suggests that it is much more like the second than the first - many, many ships passing in the night from which one observes screams of outrage - (laughter) - aimed at some other ship. (Laughter). This can be entertaining but it can also be dismaying. I am more on the dismay side.

As a consequence of this situation, there is lots of incoherence on enormously important issues, and in incoherence I include ignorance. For example, listening to Sunny talk about - very, very briefly - the history of research on school effects - what she called the “production function literature”, one fact one finding that Jim Coleman and McPartland and York and Carol Hobson made in their study, which Mike Smith very ably excavated in his re-analysis of the data that was published in the Moynihan and Mostow (?), was that approximately 18 percent of the variance in school achievement lay among schools. Eighty-two percent of the variance in school achievement lay within schools. This is a finding which has been repeated, roughly speaking, between 15 to 20 percent among schools. Coleman and his colleagues were dealing only with the 18 percent. They were concerned only with school-to-school differences in performance. That is a hugely-important fact, which almost everybody - I know that this is not true of Sunny, but almost everybody who has some acquaintance with this literature doesn’t understand. Most of the variability in student achievement was not on the table in that study and it has not been on the table in many other studies.

Second, another - and I offer this not in a critical spirit, but to illustrate some of the big problems in this field. Almost all of these studies, as Sunny said, are cross-sectional. They are looking at the relationship between inputs and very, very occasionally processes and outcomes, where you are looking at a snapshot at one point in time, and when one asks the question that Sunny was talking about about the relationships between inputs and outcomes in that kind of a framework, one finds that, you know, there is something somewhere between a modest effect of school resources and a trivial effect of school resources.

If, however, one looks at value antedated, still staying with the variance among schools, and asks what fraction of the variance in outcome is accounted for by school resources - there are very few studies like this or there were up until very recently - the answer is 50 or more percent of the variance in academic growth is accounted for by school resources. Now, that is one huge hell of a difference, which most of my colleagues - there may be a kind of a time warp in Michigan - most of my colleagues are completely ignorant about.

Now, in a field in which there was a community that was arguing about knowledge growth, about accumulation, I don’t think that these rudimentary and hugely important findings would be so relatively little known by members of the community.

One reason that several of the opening panel members suggest is the fragmentation of the field. It is a bunch of disciplines and other more practically oriented things. Another, as Sid Winter and some others on the opening panel said, is an unfortunate inattention to practice by many, many researchers, and a third is the lack of much of a theoretical frame around which knowledge might accumulate - and by accumulate, I mean - I include revision. Nothing is forever.

So I want to talk about that last one, about theory, and my remarks are not part of some set of methodological preferences. I am not - we didn’t start out on this study with a set of methodological preferences. When Steve Raudenbush began graduate work at the Harvard Graduate School of Education, he was a qualitative researcher working with Sarah Lawrence Lightfoot. He only happened onto quantitative work because Sarah and Tony Bragg were teaching a course together, and Steve is one of the very able statisticians I know who really understands and can do qualitative research. The same thing is true of Brian Rome.

So we didn’t come to this with a sort of a mind set about what kind of inquiry is best or most valid. We came to it, rather, asking what kinds of inquiry are most likely to build knowledge in the field in which we want to work, and so we spent a lot of time rather conventionally, reading back through work that our colleagues had done over the past 20 or 30 years, and on the resources question, we found - as I think some of you would agree - we found that in most of the school-effects research that there was a kind of conventional, often assumed, but sometimes explicit model of how the world worked, that there are outcomes for which we use achievement - norm-referenced, standardized-achievement test scores as a stand-in. There are resources, and sometimes, recently, there are intervening influences like teachers, but very, very rarely does anyone get to practice this, as Sunny was suggesting.

And so in these conventional models, the resource is the cause of the outcome. That is the way the equations are written, and there is a sense in which that is the way economics and some parts of sociology work. I don’t think you could call it a theory, and many people who work in this field, when they talk and often when they write will say that, “We are testing the operating assumptions of the system,” and that certainly is what Rick Hanashek thought he was doing when he started out many years ago.

We decided that that is not plausible, that even if one thinks about educational resources apart from any particular intervention, any particular school, any particular system, it is not plausible to think of them as a cause of anything, because they only become active when they are used, and they only are used in instruction. Class size doesn’t cause anything. Class size is a resource - well, perhaps a congeries of resources - which may or may not be used by the teachers and students in question may or may not be used effectively.

The causal agent or agents are those who are acting in instruction, including the materials that they use, including the environments in which they work which includes everything from their principals to people in the community, state standards assessments and you name it. This is the United States, whatever else might be said of it, the environment is booming and buzzing with educational activity.

A couple of examples of how we think this works. When Steve and Brian and I, and many of our colleagues do a regression analyses trying to predict student achievement, we enter variables like parents’ education. Now, we don’t really think that parents’ education causes students’ achievement. When we sit down to explain to ourselves what we mean by that association, it is that parents with more education are more likely to do certain kinds of educational things than parents with much less education. So the variable, the sociological or economic variable proxies for a set of behaviors that have to do with the generation and use of resources that act in instruction.

So resources count as they are used, and they are used in social interactions in and around classrooms. So we think that observing those resources and validly measuring their effects would have to take those conditions, these two conditions into account. Some instructional research does. The first that we found was some studies by Gail Einhart(?) and Bill Cooley back in the ‘70s, every elegant studies, but most doesn’t. As Sunny said, practices are just left untouched in most of this research.

A further point about these interactions which led us down the road that we had not expected to walk is that they are laced with judgment, with what we call calibration.

The math teacher who responds to Johnny’s solution to a problem responds based not just on what she thinks the answer to the problem is, but on what she thinks about Johnny’s ability and a bunch of other things, and Johnny responds to the tasks that his teacher sets similarly.

If that is so, and it seems self-evidently so, then resources are used in instruction in ways that give rise to an enormous problem that economists call indegeneity(?), that the teacher’s judgment of the student’s ability, which is presumably related to the outcome, is implicated in what resources she applies or offers to that student and vice versa. So there is a terrible sort of problem at the core of efforts to unpack this thing that we call instruction, and there is a lot, and anybody who has ever crouched in the classroom in those chairs designed for people much smaller than we, it is on plain display every day.

So there’s some very interesting research and psychiatry and medicine about this kind of situation. Those researchers call it “dynamic treatment regimes,” and several very talented methodologists have satisfied themselves that trying to estimate the effects of doses of treatments on responses with standard sort of regression analysis doesn’t work, precisely because of this - because the treatment itself is based on estimates of the patient’s condition, and that is exactly what teachers do every day.

So we concluded, somewhat to our surprise, that if one wants to generate valid estimates of the effects of resources, we have to start asking the questions that we have been inclined to ask, like do resources matter? I mean, you can understand how the question would come up in the political process. You can understand how it might have come up in a country that was always ambivalent about spending on schools, but the question is much too gross. Algebra does not spring spontaneously to the mind of 12-year-olds or, alternatively, imagine that schooling only could be purchased on the open market. Now, I am sure you are all familiar with the parallel and slightly diverging trajectories of advantaged, disadvantaged students’ achievement as they march through the grades. Well, that is a state-subsidized compulsory system. Suppose those kids’ parents could only get schooling on the open market. Those lines would diverge much, much - I mean, but it’s - I think it’s indisputable. The achievement trajectories would diverge much more greatly. So resources do matter, and when we study them, we study them in this kind of a system in which a huge amount of the inequality resources have been eliminated.

So we concluded that the best way to do research on the effects of resources is to decide - and this, again, somebody was suggesting in the book - decide what outcomes matter. What is the job that you want done in the school in mathematics? Construct a valid and reliable measure of that outcome. Define a regime, a treatment, an approach to instruction that seems likely, based on all kinds of research and developmental trials, to be reasonably related to that outcome, and try it, and, ideally, under experimental or quasi-experimental conditions, but use other sorts of research to figure out what is going on in there. Do not do your classical blinded or non-blinded clinical trial, and then vary the resource constraints, and inform that variation with what you have learned about dynamics and effect.

Now, we often - it is partly because we have concluded that it makes sense and partly because we think it will be heuristically useful - we hope it will be heuristically useful - to the field. It will not cure all the problems of research on educational resources. We think it would focus attention on really consequential issues - what approaches to instruction are most helpful for what sorts of kids in what sorts of situations in what sorts of subjects?

We think it would tend to ground research in an understanding of practice, because one couldn’t do this kind of work without deeply understanding practice, and we think, within certain limitations, it also would offer a frame on which knowledge could build and cumulate, rather than simply accumulating.

Thank you.