Workshop on Understanding and Promoting Knowledge Accumulation in Education:

Tools and Strategies for Education Research

Day 1 – June 30, 2003

Remarks by Dr Helen Ladd

DR. HELEN LADD: I was asked specifically to talk about the relationship between resources and student achievement. I am, in fact, going to devote much of my talk to production functions, but given the more general topic that was assigned to me, I felt it was important, first, for me to figure out what the various research questions were related to the linkage between resources and the student achievement.

So I came up with just a very simplified model to help me think about that, and I hope we can help you think about relationships as well. In this simple model I’ve got family background, characteristics of the students is a given, and then I’ve got these three areas; resources, practices, institutional context - with arrows drawn to student achievement as the outcome goal and the educationists would talk about other outcome goals as well, but I want to keep things as simple as possible.

What I would like to do is just have you keep this analytic very simple framework in your head as I talk about three specific research questions about the relationship between resources and outcomes such as achievement that we might care about.

The first question is what is the impact of school resources on educational outcomes? I call that the effects question and that is the one that I will spend most of my time talking about.

This is the production function literature. I have specified here the typical type production function that many economists or others estimate. We have achievement which could be at the individual level or at the district level or at the school level. People estimate. There’s lots of different ways. As a function of school resources, I have in mind variables such as class size there or teacher quality and then family background characteristics and the characteristics of peers.

Now, and I have also included a lagged achievement variable that is important to get at some notion of value added and to take account of the cumulative aspect of the education production process.

The key thing to notice about this typical production function formulation of response to the key research question is that practices are not controlled for, nor is the institutional context explicitly controlled for. So that means, you know, the question - even the basic research question if we are going to use this type of formulation is a bit ambiguous. Are we thinking conceptually of holding practices constant or not? So we go out and look - use this sort of a model to think about something like class size, the question is are teacher practices, such as whether the teacher uses an interactive teaching form or a lecture format, controlled for or not?

Now, in fact, what happens in most of the cross-sectional analysis using this sort of a model is that what we are picking up with resource measure like class size is the effects of class size as they are correlated with practices or institutional context.

So I will come back and talk in more detail about this whole set of questions related to the effects literature, but let me just quickly specify two other research questions that one might ask about this link between resources and outcome.

One might start at the other end and ask the question as follows: Given some outcomes that we care about, what resources would be needed to achieve those outcomes? I refer to that as the adequacy question, and that is an issue that a lot of economists in the late– - well, sort of throughout the ‘90s and early in the 2000s have been devoting some research, mainly using expenditure data at the district level. So these are people like John Yinger(?) and Bill Delcome(?) and Hom Downs(?), all economists working in this area.

Basically, they start out with data at the district level, and they have expenditure data per pupil, and I actually have labeled this desired outcome - think about it, in fact, as outcomes, a number of outcome variables, such as percentage of students who - at the elementary school level, who are at grade level or some other - you can have a whole - outcome measured, and then some cost factors where a cost factor is something like the percentage of children in poverty or the percentage of special-needs students in the district, and what these economists do is estimate this equation using actual data and then go back and plug in some desired outcome levels. You want everybody - you want 80 percent of the kids to be at grade level. You plug in the characteristics, the actual characteristics - those are the cost factors in that district - to come up with the level of expenditures per pupil required to achieve those outcomes.

Now, there are lots and lots of technical issues associated with this, a lot of work recently that economists have done, but I do want to just sort of highlight the different focus from the straight effects focus that I referred to earlier.

I believe David Cohen will be talking about this from an instructional point of view and my reading of his work with Ralm(?) Bush and Ball is that what they are trying to conceptualize is something very similar, but in an instructional context. So they start with the notion of instructional goals, ask the question what instructional strategies are most productive in reaching those goals and then how much did those instructional strategies cost, what are the resources required. I think that is really very similar, but in a more micro within schools with an instructional context as opposed to the district level approach of economists.

Now, I am not going to say anything more about that, but David, in his talk, will elaborate, I believe, on his approach.

Let me go to the third research question, which I refer to as the “Productivity Question,” and I am not going to talk much about this today, but I think it is important to put this in the context of this whole discussion of link between resources and outcomes, such as student achievement.

Here the notion is you start out with a given level of resources and what can we do to make those resources more productive toward the goal of educational outcomes, achievement, earnings or whatever measure we are using.

I would put the whole effective-schools literature of the 1970s and ‘80s into this question: In my reading of that literature, which emerged out of the Coleman Report which - saying money doesn’t matter. Researchers are going out finding schools, schools that seem to be effective in raising achievement of disadvantaged kids, and then going to those schools, those “effective schools,” and trying to figure out what practices are common to those schools that seem to account for their effectiveness in raising student achievement.

Now, my own view - and I’ll throw in my views along the way and we can debate or discuss them - is there are some real problems with that whole effective-schools research, because there is no real serious effort to determine causality of effects as opposed to just correlation.

Now, I would put a whole lot of other research into this category of thinking about the productivity of resources. There is a nice paper - a 1992 paper by David Monk, educational evaluation of policy analysis, where he talks about productivity research and the need to get down to the classroom, but a lot of this research is focused on this question of how to make resources more effective.

I have included in this a lot of the work that is in the National Research Council volume that Janet Hanson and I edited called Making Money Matter. A lot of that is about this third question.

So with that as background, let me go back to the first research question - go to the next slide - and I’ll, quite quickly, go through a summary of the types of research of this effects-literature type, and, presumably, most of you will be familiar with Coleman at the top, and then I’ve got Production Function, Literature, a little bit on Hennessey(?) class-size experience and then some on teacher effects where the particular focus is using state administrative data.

Let’s start out with the Coleman Report. We have to start there. That is the sort of starting point for all of this research. As many of you know, that was based on a survey commissioned by Congress as part of the Civil Rights Act of 1964, more than half-a-million students from more than 3,000 schools.

Now, the interesting thing to me, as I went back and looked at this more closely, was to find that the charge to the U.S. Office of Education read as follows, to conduct a survey and report on the “lack of availability of equal educational opportunities by reason of race in public educational institutions in the U.S. and its territories,” and I’ve left out a few things there.

A couple of things are interesting about that. They sort of thought they knew the answer, and then, secondly, they did not ask - they did not specifically charge the U.S. Office of Education to look at the link between resources and achievement, but, in fact, the authors of the Coleman Report did go that next step in a very short period of time to not only gather the data on resources, gather the data on outcomes and then to try to make the link between the two.

The main conclusions of the Coleman Report, as I see it, are, first, that most blacks and whites attended different schools. Second, the school resources didn’t differ much between blacks and whites. So there is the resource variability issue, and then third, and very importantly, that the variation in school resources, which wasn’t all that great between the blacks and whites, was not very important in explaining the variation in student achievement, and, finally, that the variation in family background and peer groups of the students were very important, were much more important in explaining this variation.

So this Coleman Report was interpreted as saying schools don’t make a difference, but there have been lots of criticisms of the data gathering and the methodological approach. A lot of the analysis was of the production function where the link between resources and achievement was based on a variance analysis where it mattered which set of variables you put into the model first, and it was always the case that they put the family-background variables in before the school resources, but you got a lot of correlation between those. So separating out those effects is very hard.

Despite its flaws, though, and I think there were a lot I think, the Coleman Report clearly has to have a huge impact on research, among other things, that it generated the effect of schools research that I referred to earlier, and that was a response to the apparent conclusion that schools don’t matter.

But it also generated - provided the data for - follow-up studies and then generated what is typically referred to as the “Education Production Research.” So let me turn to that.

That education production research is largely done by economists, not exclusively, but largely, and some of that education production function research is not focused on student achievement as the outcome goal. Economists, as you know, like to think about education as an investment in human capital and the ultimate outcome goal for many economists is earnings or wages. So there is a lot of research. The most famous article is by Tory Kruger(?) in the early ‘90s looking at the effects of education quality in the 1930s and ‘40s on the earnings of workers as of the 1980 census.

But, in any case - and there are some issues about that earnings research - let me focus instead on the achievement-related research. A plethora of studies during the 1970s and 1980s, mainly based on the opportunistic data sets. So it is existing administrative data set or whatever is available, not newly-generated data, other than the use of some data from the Coleman study.

Lots of different types of analysis. Some of the analyses were based on variation across school districts, some across schools. Some using some national data sets were used as the unit of observation of the individual student.

In all of this research, there is some general recognition of the importance of some sort of value-added research, and that is why, when I put up the equation earlier, I had a lagged-achievement variable to emphasize the valued-added specification. That is generally preferred - generally accepted as the preferred specification.

As resources in these models, economists typically use focus on those measures of resources that are observable, and that costs money. So the focus of a lot of this research is on resources, such as teacher-pupil ratios, which cost money; more teachers to increase that ratio; teacher characteristics, such as years of experience or whether the teacher has a bachelor’s degree, both of which cost money given the way salary schedules are set up.

Now, some studies do look at other variables as well - for example, teacher test scores - but a lot of the research is focused on the budgeted resources or the resources that cost money.

Now, the results of a lot of these studies, as many of you know, have been summarized several times by Rick Hanashek(?). His first summary came out in the Journal of Economic Literature, Prestigious Economics journal in 1986, and his latest summary that I know of is 1997 in EPA.

What is interesting, the first review of all this study, these production-function studies, published production-function studies included 33 studies and then the 1997 included 90 studies. So there are a lot of studies that have been published during the period that he was looking at.

If you look at the Hanashek articles, though, you won’t see these highly-emphasized numbers of 33 and 90, what you’ll see is much bigger numbers like 90 estimates or 390 estimates, because what Hanashek does in his overview of these studies is that he looks at the individual estimates of the effects of each of these teacher resources, and that is going - each of these school resources, whether they are class size or teacher quality, and I’ll come back to that point in a minute.

The conclusion that Hanashek comes up with, using a vote-counting type of analysis - he looks at all of these estimates and asks whether these estimates of the effects of resources are positive, negative or statistically insignificant and concludes that most of them are not positive and/or not statistically significant. So he says budgeted resources - or he concludes that budgeted resources don’t matter.

This work, I would argue, has been very influential. He has written a lot of articles. It has been used in school-finance cases, and Hanashek is always cited by the plaintiffs who don’t want to spend more - or not the plaintiffs, but the defendants who don’t want to spend more money on education. So it comes up over and over again, and it has had an impact, not only in U.S., but around the world. I’ve been traveling quite a bit recently, and people always say, “Well, as we know, money doesn’t matter,” Hanashek.

Now, there are at least three main criticisms of these Hanashek studies. One is that he does his analysis based on all the published studies, whether the studies are high-quality or not do and there are some standards or whether studies are better than others, but he didn’t want to set those standards, so he includes them all.

Now, to his credit, in later summaries of the research, he has narrowed the set of studies, in various ways, to account for that criticism.

A second criticism of the Hanashek review of this literature comes from the methodologists, Hedges, Lane and Greenwald, who said, “Look, this isn’t real data analysis. This is vote-counting, looking at all these estimates. What we really ought to be doing is thinking about this in a much more statistically sophisticated way, thinking about putting all of these studies together as if they were one big study in looking at the statistical significance and size of the impacts.” Hedges, Lane and Greenwald conclude that resources do matter. So there is a different result, and it is based on exactly the same set of studies that Hanashek worked with.

There is a much more recent criticism by Alan Kruger from Princeton, and Alan is looking specifically at the Hanashek results relating to class size, and I haven’t seen a similar analysis for other resource effects, but I think it is a very important criticism that Kruger makes.

He says, “Look, what Hanashek is doing is misstating the estimates, because he is using estimates, rather than studies as his unit of observation, and we all know, as we are trying to publish our work, if you don’t have highly-significant results, what you do is present lots and lots of results to convince people that you have tried everything and you don’t have significant outcomes.”

So what Alan Kruger did is, went back, took all the studies that Rick Hanashek used in his summaries and re-weighted them, using the study as the unit of observation, and then provided, in addition, some empirical evidence of his assertion that the studies that have lots of estimates are those studies that are likely to have insignificant or odd results. And he, Alan Kruger, does find that the resources in this case on class size does matter, based on these non-experimental sorts of designs.

There are a whole lot of technical issues that come up in all of these production-function studies. I don’t want to understate them by not talking about them very much, but there’s endogenously question if you are looking at teacher characteristics or measures of teacher quality, you can’t assume that they are exogenous to the type of school that the student is in. There are definitions of variables; using people-teacher ratios or true class sizes. There are limitations of cross-sectional - of using cross-sectional analysis and general understanding that longitudinal analysis is much better. There is the issue of the nesting of students within classrooms, within schools, and their techniques, the whole hierarchical linear modeling technique by sociologists, and economists use generalizing squares to deal with those differences.

I was pleased to see recently in the new book by Judith Winger Singer(?) and John Louette(?) a way to sort of bring sociologists - the methodology of sociologists and the methodology of economists together into a common framework, so that we can talk to each other more so than in the past.

There are other things I could say about the production function, having to do with estimates from the international evidence or other issues about whether, in fact, there is a stable production function, but let me leave that aside and move on to the third type of research here, which is the experimental research, and the key example here, related to resources, is an example you are all familiar with, which is the Tennessee class-size experiment, which took place between 1985 and 1989, and then is still - there have been follow-ups and also reevaluation of the initial work, all these early studies, and then Alan Kruger, again, did a nice reevaluation - nice from my perspective as an economist - a careful re-analysis that came out in 1999 in the Quarterly Journal of Economics.

The conclusions from this are clear, at least to me, and that is that there are some effects - positive effects - from having smaller class size for minority and disadvantaged students. So that is the substantive conclusion that comes out, but the other conclusions, I think, are equally important.

In terms of internal validity, this experiment is really quite good, not perfect, as you have some students dropping out, but there are other issues as well, the main one being how do you take this to scale, and we have evidence from California where you try to take this smaller class size to scale, and there are lots of other resources you need to worry about, like when you’re talking about the teachers, and, in particular, the movement of teachers between urban areas and suburban areas, and so, in fact, when you take it to scale, it may be that minority kids who are trying to help with the policy are not the ones who are helped.

Very quickly, because I only have about three minutes left. Teacher effects. I want to mention this briefly, there’s a lot of new work going on, some of it by me and colleagues using North Carolina State administrative data, some by Bill Saunders(?) using detailed state administrative data from Tennessee. The problem I have with Bill’s standards work is that he uses it in a proprietary way he is now at schools, SAS, so it’s hard to get into his models, and then there’s very good work by Hanashek, Cain and Ribkin(?), based on Texas Data.

Now, this use of administrative data, I think, is really important here, because for Texas and North Carolina and also for Tennessee, you’ve got data over time on student test scores. So you’ve got longitudinal data. You have teacher data. In Texas, they can match the teacher data to the grade of the student in a particular school. In North Carolina, through some fiddling around at other activity report data, we can, in many cases, match teachers to individual classrooms of the students.

Now, there are some limitations of this data, this state administrative data. It would be nice to have better measures of family background characteristics, and we just don’t have data, in many cases. Be nice to know the educational levels of the parent. Usually what we have is whether the family is eligible - whether the child is eligible for cross-match(?). Now, it sure would be nice if states could do more with these data sets.

Let me turn to the conclusions - so I am ending here. My interpretation of what has been going on here. I think, over time, starting with the Coleman Report and to the present, we are making progress in understanding how resources affect outcomes. Once again, I want to emphasize, we are not controlling for practices or institutional contest. I do think the literature - and this is my own interpretation, and I got a bit off track with the Coleman and with the Hanashek summaries - my own biased or based on my own research view is that resources matter, given average practices and the average institutional context.

Second conclusion is that the state administrative data sets do have great potential, be helpful for education researchers to work more closely with the states to see if we can build in to some of them some additional questions at the time students are taking tests, so that we can do a little more about family.

And, then, finally, I want to end with this potential for more experiments related to resources, and I suspect David may pick up on this as well. If we are really focusing on this resource question and want to learn more than we have to date about the relationship between resources and practices, what I wanted to see is experiments that have various combinations of resources and various combinations of practices or incentives built into them in order to change those practices along with resources.

So I guess what I’m saying is it is important that those of us who start with an economist perspective and not used to thinking about practices need to bring a few more practices in, but those people who also who are starting from the practice in the instructional side need to pay attention to the resource side as well.