Workshop on Understanding and Promoting Knowledge Accumulation in Education:

Tools and Strategies for Education Research

Day 1 – June 30, 2003

Remarks by Dr. Kenji Hakuta

KENJI HAKUTA: Thank you for the introduction. So I’m the guy with the fancy job title and no money. It feels a bit like seeing my life flash in front of my eyes here, looking at about half of you that I recognize from various things that I have been involved in. And it’s a bit of a scary thought, trying to do this meta-talk about a meta-issue or meta-perspective, and what I understand is that I have sort of three charges. One is to hit on some of the issues that were raised in this book.

And the second, I was asked to be a bit specific and relate some of these issues to a field that I have worked in, which is the education of the English-language-learners, also called “Bilingual Education,” and so I’ll do a little bit of that.

Then I will raise some questions about knowledge accumulation, something that some people in this room know quite a bit more than I do and have thought more about, but the questions that are raised should stimulate conversation.

I do want to say a little bit about the policy context, just so we have some perspective. This phantom sponsor that Laurie referred to was something that I was also involved in, the National Educational Research Policy and Priorities Board, and it was our discussions with Michael Feuer and Lisa Towne and others at the beginning of the discussions that led to the reauthorization of OERI that stimulated the thinking that triggered the idea that the academy would be a good place to go to for advice on the question of what it means to have scientific research in education.

At that time, there was some thought into the definition of education research, but not a lot. If you think that the definitions in “No Child Left Behind” and in the Institute for Education Sciences proposal were kind of harsh and draconian, there were probably some even more extreme. What might be considered the sort of methodologically-fundamentalist sort of ideas floating around at that time. Some of them you may know, one of these I attribute to Gerry Sroufe, who showed me one of the amendments, which was the definition of research is evaluate using randomized experiments in which individuals, entities, programs or activities are randomly assigned to different variations, including control conditions compared to relative effects of these variations.

There was a - Carnine and Meeder had proposed a set of principals to be put into law at that point where the science studies were to have these conditions, including a minimum of 12 per condition.

So the fact that there were organizations and bodies, Meeder is one I guess, which would be called a legislative advocate, which worked with Doug Carnine and had organized to try to get some of these ideas and push the boundaries of definition of research.

And so that was kind of part of the context in which our board was working and felt that we alone were hardly able to address it.

We had, at that time, been formulating some policy proposals which we hoped that the Academy would agree with. With the help of Emerson Elliott, who is also in this room, our board worked our way through how to respond to and deal with the issue of randomized experiments.

So we addressed science. The power of science comes from a combination of strong theory and data that bear on the theory. This implies endorsement of explicit ideas and agreed-upon methods for exploring and testing these ideas based on observation that is internal and external consistency. Experiments, as a classification of research, should not be scatter-shot or universal. Rather, they should be justified by a cumulative record of rigorous, naturalistic observation and piloting. This requires knowledge of context, in addition to adherence to scientific canons.

While experiments in education may not be used as frequently as they should as a preferred method, means for investigation, for a variety of reasons, perhaps, but availability of funds is surely one such reason, science should not be equated with experiments.

So the idea that trying to legislate things was there as part of our thinking, and I think the first reaction I had when I saw these various sort of fairly draconian ideas was a scene from Woody Allen’s Bananas, which I like to quote from once in a while, in which the dictator takes over and then he gets up and announces to the population that the official language of San Marcos will be Swedish - (laughter) - and that underwear will be changed every hour on the hour and will be worn and changed every hour, and that underwear will be worn on the outside, so that we can check. - (laughter) - That was the first sort of thought that occurred to me about some of these proposals, that while there are some really legitimate reasons why this methodological issue is important and it’s really pushing us, it really is one of those where you say, “Oh, my God. It’s come to this.”

In that context we were delighted by the deliberations and the conclusions of Scientific Research in Education, which while there were six principles, I kind of boiled them down to four and sort of theoretically-grounded empirical questions; methods appropriate to the question, which was one of the important issues around methodological fundamentalism; emphasis on generalization and replication, which is, again, one of the themes that we need to address in our workshop; and open community and transparency.

And I’ll hit on a few of these issues, fulfilling the first charge from Lisa on what I am supposed to talk about, but I think it is important to note that the SRE was important in pointing out the interplay of theory, methods, measures and data as defining science, and that education fits into this in many respects. So the point that Bob made, that the report shows some existence proofs of areas in education where this has happened, it can happen and so forth.

One of the highlights I would like to point to is the emphasis on community, that the research community, it sort of shows up in many different parts of the report, that it is not just the methods that enable knowledge to accumulate. Methods is an important part of it, obviously, but that should not be what defines it, but especially the critiques and the questioning that happen in science that - you know - the thought that you should be your own worst critic or harshest critic of your own ideas and ideas that you believe in as being really quite fundamental.

The contested terrain. I noticed that there is another book that has that word in there somewhere, but the idea that education is in a contested area in which various members of the public - as Carl Kaestle said, everyone has been to fourth grade and therefore can feel like they can participate in debates about education, and that, in that sense, you would do that much more than the public might comment on forms of medical treatments, although even in medical treatments, as you know, the public does feel like it can voice itself. So it’s not entirely different from education, that education is contextually localized and complex.

You hope that it is complex if you think about the fact that - you know - this is the argument that - you know - what happens in California shouldn’t happen in the rest of the country and that it’s unique and different, and if you have been following what is going on in California, you will hope that this is actually true - (laughter) - that what happens there should not resonate around the rest of the system, and, yet, I think many of us have sat around enough tables in discussions of kind of local educational issues with others concerned about similar issues to realize, “God, you know, everyone thinks that their problem is unique and yet the commonality is in how much people think that their problems are unique” - (laughter) - and this is not just true of the US, but if you sit around tables of OECD or groups, you know, from other countries that also is - at least from my experience has been the case.

The persistent belief that research does not matter. The report - I just put in page 96 just to show Lisa that I read the report carefully - is presented as a discouraging notion of educational research having an effect, that people think, “Oh, it’s not going to matter anyway,” so people will listen to anything in education, and, therefore, - at least from the consumer side, the demand side, the demand for quality research is a push, and so that is seen also as another issue that - scientific quality in education.

And then science is also about revealing and understanding what we don’t know as it is about what we do know. So I think the report addresses it more in the context of the fact that social phenomena have a larger error variance in measurement, because of different context and so forth, and I think it is even more pronounced than that, that, you know, fundamental importance of science is in letting us proceed and attend to what we have not yet been able to understand and explain. So that by knowing - having a clearer sense of what - you know - what we do know, we also can get a clarity on what we don’t know, and, certainly, in education, there is plenty that we don’t know, and so that this sort of - the complimentary aspects of what we do know, versus what we don’t know, I think, is quite important. I’ll come back to this, though, in a point that I’ll hopefully clarify this in a bit.

So the workshops and this session goals, as I understood it from Lisa’s marching orders, were how to promote knowledge accumulation in education by understanding it better - “it” is the accumulation - through analogies with other fields, specific instances, and by looking at the particular advantages and disadvantages in the field of education, and I tried to stress advantages as well as disadvantages. I think we - in education, we have some things that other fields would love to have, like the fact that the public is really interested in it, and, in that sense, it has the potential - and it hasn’t really attained that potential, but it really does have some of the potential that - what’s contributed to the successes of places like NIH, where - you know - the consumers really do care about it. It’s just that the problem is that disconnect that the consumers don’t think that research would have much to say in addressing those issues. So there are some advantages.

Now, but to sort of - some of my own - connecting it to some of the work I have done with the education of language-minority students, I just offer these two slides as sort of bookends to the field that I have been working in.

‘74 represents a slide from Chinese schoolchildren in San Francisco, who were the basis of a lawsuit that resulted in a Supreme Court decision in 1974, called Lau v. Nichols, which said that it is a district and state responsibility to address the needs of children who arrive in school not speaking English, without prescribing any particular approach.

1998 was a Californian named Ron Unz who passed a ballot initiative - in California we have this way of doing ballot initiatives and governing through that method - which passed an initiative called “English for the Children,” otherwise known as Proposition 227, which banned the use of native language in instructing English-language learners, except under certain conditions - and speaking of localized, you know, that proposition has passed in Arizona, Massachusetts and is very active in many states, and so what does happen in California occasionally does reach other states.

So - and this is kind of a - represents roughly a 30-year history. Before Lau, there was the federal - Congress passed the Bilingual Education Act, which recognized the presence of English-language learners - otherwise, at that time, were referred to as “LEPS,” Limited English Proficiency students - and that, through a series of both Congress and the Supreme Court and other federal courts, has developed a framework for what it means to provide adequate or appropriate instruction for English-language learners, and I raise this because I think this is a framework that if you have it kind of legally in place, provides an opportunity for research to play a role in an important educational issue - educating English-language learners.

This was a Fifth Circuit Court decision, but it has pretty much been interpreted as the - kind of, in a sense, interpretation of Lau, which was much broader, and they said basically a program has to be based on sound educational theory - so there is a role for research - that they must be implemented effectively with adequate resources and personnel - so if you are going to have a bilingual program, it would be helpful to have a bilingual teacher - and - those of you who know this field know how difficult that is, so that is the - and after a trial period, the program must be evaluated as effective in overcoming language handicaps. If the results are not satisfactory, the implementation or the theory must be reexamined. So there is a little loop there.

So it is actually a very rational theory written by a Texas judge, who thought - you know, just thought about what does it mean to have a framework that isn’t prescriptive, saying you have to have bilingual or not, and, yet, I think it was part of the Congress that did it, that issue of educating language-minority students became an issue that was either bilingual or English only and defined as that, and so a lot of research was set up around asking that question, “Which is the better method?” and there is actually quite a bit of accumulation of research that tried to address this, not always using randomized experiment. Actually, there is a - I think - a research process going forward in the Department of Ed right now that requires randomization to address this question, but from what the meta-analyses of these studies would show, this is pretty much what they will find, and this is just an illustration from some data that I have collected, but this just serves to show the point, which is this shows bilingual versus English-only program kids in comparable schools over grades one through five, and it shows a slight advantage in English-reading comprehension for kids who have been in transitional bilingual programs, programs that start literacy in Spanish and then transition into English, and it is a statistically reliable effect, in most cases, about - you know - a .2 standard deviation effect, on average, and that is pretty much what - you know - I’d say that that would be the conclusion of most reasonable people who do meta-analyses in this area, and that was accomplished by investing in studies that do the comparisons, and, now, there is a randomized component to it that is going forward, and my prediction would be that that - about a .2 to .3 effect size, if properly implemented, is what they will find.

Now, the white space up above that is where they would be in age equivalent to native English speakers - okay? - is where they would be. So you can see that there is a gap there of about one year at first grade, and then at fifth grade it is about - you know - two to three years.

And so I have kind of used this to talk about some of the opportunity costs of - you know - letting at least some of the policy and political issues drive a research agenda that what - and the field is, right now, I think, shifting from talking about the gap - the difference between these two and saying, “Well, you know, what is accounting for that white space there?” and, you know, people are looking at things like poverty and so forth, a lot of the Title 1 sorts of issues, but that was an issue that the field really did not pay attention to. It would always talk about the poverty of the kids and so forth, but really did not address, in part, because of the grooves that were set by the policy context.

So kind of going back to this Castaneda thing, what the field - at least, hopefully - will do, as it moves along, is to say, “Okay. So we - you know - we have had this one theory about native language versus English only that accounts for this bit of explained variance, but there is a huge amount that hasn’t been explained, so we need to go back and start really - oops, that’s my 20-minute mark - start addressing those issues.

So let me just - I’ll come back to some of these issues as I get a little bit more meta maybe here of how to promote accumulation, and I’ll just raise some questions, and these are just literally - I mean, you know, this is why this was such a difficult presentation to prepare and sort of scary, because I’m just as ignorant as everybody else in this room, I think, about how to do this.

But some of the questions are what you might call supply-side questions. Where do we invest in the knowledge production? Do we invest in individuals, in institutions or ideas? So - you know, I would say that my life flashing in front of my eyes, Ellen Lagemann, who, when she was President of the Spencer Foundation, you know, we worried a lot about what is the best unit of investment. As she walked into the door as President of the foundation, she realized that a large amount of money had been already invested for her by her predecessors in schools of education in the form of what is called “Research Training Grants,” and several in this room are recipients of that - grant, but that was an investment in schools of education to promote research. Is that the right way? Should there be individuals? Should there be other strategies?

How do we combine the strengths of the disciplines and the strengths of the professional schools? And, most importantly, probably, how do we build community? and the whole sort of set of activities that CORE seems to be addressing is this illusive thing called - you know - a research community and a set of peers and processes for quality control that if it doesn’t do it, Congress will do it for you is a way of thinking about it. So you will have to have - of 12, and, you know, change your underwear every hour. (Laughter).

And so demand-side questions: How do we train attention to quality, in addition to content, both in the courts of law and public opinion? So the - you know - consumers want to know more about reading or want to know more about civics education, but the claim of the report, which I would agree with, is that the public does not distinguish between a report from some conservative or liberal advocacy group versus a report coming out of a peer-review journal and so forth. So how do we train attention on quality?

Who qualifies as an expert? What is sound educational theory? These are sort of the Castaneda issues, that first prong of, you know, that has - based in sound, fury, and I have been in courts as a witness on these issues, and it is somewhat surprising. It is really not that different from if you serve on juries on DUI cases or other things like that of what an expert witness is. All right? So there has to be some reform or some good thinking about what would qualify somebody as an expert in a lot of these cases.

How do you get the media more knowledgeable about educational research and quality issues I think is extremely important, although there is - if you have been to the education writers association meetings, you’ll realize that there has been a - you know, I’ve been going to these periodically. Dramatic change in the last 10 or 15 years in - first, you can just notice by the age of the people that are there, that it used to be the youngest writers writing, and nowadays, it is really much more of the experienced reporters who are starting to show up, and so there is certainly a maturing of that group that is really very promising, but how to connect with writers?

The contested nature is how to turn this into a productive advantage, so we do have - you know - I mean, the fact that it is contested also means that there are a lot of people interested in it. So this is the - if you really made good use of it, maybe there is some good to come of it as well as just worrying about the fact that it is contested and, therefore, what can we do about it or there is nothing we can do about it.

The cultural-diversity issue. The report - probably the pages that I had the most difficulty with in this report were the pages referring to sort of the diversity of the student population presenting a problem for generalization, and so I guess I ask is this truly a big obstacle for knowledge accumulation and generalization?

For example, commonalities may be in differences across social classes rather than numbers of languages or cultures represented. The fact that we are confounding, in our society, diversity with socio-economic - racial and ethnic and cultural diversity with socio-economic diversity may be part of the problem.

I think of this, as I move to Merced, which is a very rural community in California. It is what this writer, Gerald Haslam refers to as “the other California,” inland California, and the differences between moving my 14-year-old daughter from Stanford to Merced are profound. I mean, I feel like she would probably be more at home moving to London or Paris than she would moving to Merced, and so that says, I think, you know, that because socio-economic and cultural diversity are so confounded here, we tend to confuse those issues, and maybe the commonalities are really in what is contained in the socio-economic dimension much more than the cultural or the linguistic, and I sort of say this as a linguist. So -

Okay. How to target accumulation. Accumulations lets us know how much more we have to know, like the white space in that bilingual versus English-only thing. Like science. So how do we periodically look at the policy-driven questions and evaluate whether we have been accumulating knowledge around the best questions? So - you know - we managed to accumulate a lot of knowledge around this fairly narrow question, which turns out to be just a very small part of the story, and so what kind of processes are there to allow the field to be a little more reflective about what it has been doing and whether the targeting is appropriate?

Thank you.