Workshop on Understanding and Promoting Knowledge Accumulation in Education:

Tools and Strategies for Education Research

Day 2 – July 1, 2003

Agenda Item: Wrap-up: Summary of Themes and Concluding Comments - Dr. Laurie Wise

DR. LAURIE WISE: Thank you for that summary, Bob. Bob I think has done an excellent job of covering and encompassing the whole of the discussion and the dialogue from the beginning of yesterday, up until the end of his talk just now. I won’t try and do that. I do want to say just a few words about what this conference has meant to me and then leave plenty of time to open it up to ask what its meant to you and what things you would particularly like the committee to hear and to take away from this and to share with us things. The sort of crystallizing vision for me is that we began with the fact the field of educational research is largely viewed by outsiders as being divided and unproductive. I think we live in an age of great explosion of information technology and I think the old saw is kind of if the auto industry was as productive as the electronics industry in developing the microchips we’d all be driving Mercedes that cost 15 cents and got a 100,000 miles to the gallon. What’s happened in information technology has been just dramatic, what’s happened in education is not perceived has having changed much, years and years, it costs more now as does everything, but are students learning any more.

We had a discussion yesterday that talked about a couple of case examples, most notably I thought the production function discussion, and we also had a discussion later that talked about some common measures. And I think this illustrates one of the challenges that we need to come together to confront. In a production function we’re primarily looking at the impact of resources, maybe class size but resources primarily, controlling for family background, maybe controlling for prior achievement and on predicting future achievement. Now what’s missing? What’s missing is any description or discussion of curriculum and instruction, that is what is it that education is about, what’s missing from that equation is the treatment. Now David Cohen talked to us about the difficulties in the work and the complexity of doing an adequate job of describing what really goes on in the classroom, what really is the substance of education. But the general perception is that there’s not a common language, a common way of talking about it.

The whole premise of No Child Left Behind is that we’re going to measure outcomes, we’re going to measure achievement. Does it ask that we measure anything about the instruction or what’s being delivered? No, that’s sort of too hard of a problem I think to be confronted. So I think that there’s been this, what I think is a false dichotomy between quantitative, qualitative and the two worlds will never come together. Well the whole idea of how were we going to describe curriculum and instruction really requires the best of all of our methods to do good descriptive work and then to put it in the context where we can sort of count, where we can begin to ask questions about generalization and begin to get more systematic results about the it depends of the situation. So I think this is an enormously important topic for us as to how we can do a better job of both accumulating and communicating to ourselves, across the gap between research and practice, and to the public and policy makers at large, basic principles of educational research.

We did spend some time I think talking about several avenues that hopefully will be very fruitful, from increasing work on common measures so that we can do replication, so that you have similar measures across a variety of studies and a variety of context and you can begin to look where things generalize and don’t more systematically than we have to a large extent. Data sharing is another way that is going to promote advancement in the field and then the final discussion today about different ways of summing up and of getting from just an accumulation of knowledge to really a synthesis and some general findings.

So I’d look on this with some mixture of sort of despair and hope, it’s a challenging problem that we’re confronting, it’s a very serious one if educational research are going to be taken seriously by the policy community. But at the same time I think we’ve had a very dynamic discussion about several approaches that we can working on that will help greatly to advance the field and I would first like to thank Bob and the committee for putting together this great dialogue --

[Applause.]

And the speakers who were brought in just really I think were very stimulating and the set of topics actually came together much better than I think any of else realized they might. But I’d also like to thank all of you who’ve come to participate in this workshop and I’d like to now I think invite your comments, both ranging from what the past day and a half have meant to you to what further suggestions should we make sure that we hear among the many messages that we had over the past day and a half.

DR. PIERCE HAMMOND: I’m Pierce Hammond from the U.S. Department of Education. I want to recommend that you avoid the temptation to be precious. It seems to me that explicating the word accumulation is a good idea and you’ve seized on a dimension that we can have, we can get carried away with the quantity when we ought to be looking more at the quality. And I’d recommend that you go further into that and think about the interaction among the different kinds of studies that can be done as a field starts to open up and develop, how do those who were inclined to more qualitative methodologies talk to those that are quantitative. But avoid the temptation if you would to invent a new jargon and to be part of the in group, it’s precious. And furthermore, it adds a veil to research, you were talking about influencing policy makers, imagine going in front of a Congressional committee and explaining the difference as you saw it between accumulation and cumulation. I think you want to work to demystify this, not add another precious layer on top of it. Thank you.

DR. GIL GARCIA: Gil Garcia, IES. I think it’s very, very good and necessary to have discussions like we had yesterday and today, but I think it’s equally important to ask the question why are we accumulating any body of knowledge, why are we interested in doing this? And in our case, certainly in the education arena the questions are multiple under this general need to ask. Well, what’s the reason for accumulating knowledge? And I can think of at least five reasons why we need to continue to accumulate knowledge in the disciplines that define and describe education and this field of education.

For example, I was privy to a working draft of a document being developed by one of our research centers, it’s basically going to be the framework for the next generation of high stakes assessments for English language learners. And one of the points that struck me immediately was the absence of any real discussion on the need to look at instruction in the classroom. The kind of language, the style of language, the sequence of language, the management of language that the teacher uses in instruction, and coupled with that what students in fact are doing in the classroom, again with academic language as well as the dialects that define social interactions and learning, in particular science and math and whatever kids are supposed to do. So it occurred to me that with the absence of a discussion of a plan to look very carefully at a sample of schools or classrooms, or to look at the literature on instruction and how that needs to inform, not only the format but the content of high stakes assessments, is an opportunity that was missed, at least in the draft, but I assure you it’s not going to be missed in reality because we’re on to this particular problem.

But if you look at the arena in addition to this of school reform then I think the question always has to be well, what do we need information on, do we need additional information on school reform frameworks or models or strategies, do we need more information on the theory of change at the institutional level? Do we need more information on how to manage the long term implementation of change agents in communities like San Diego and elsewhere? And by the same token I think we need to accumulate knowledge that we can do such things as the following in the discussions that we’ve been having lately.

In the context of the National Literacy Panel, one of the questions that I asked the panel is we’re talking about the emerging work that they are charged with developing was are you going to be, for example, are you going to be in a position to enhance, to enlarge, to embellish our understanding of the concept of transition, of transfer? When we’re talking about kids transferring language skills from one language to another are you going to give the field a better understanding, a more comprehensive understanding of what we mean when we talk about transfer? Or even the whole notion of development, cognitive and linguistic development over time. Are you going to push the field to get busier about trying to understand all of the dynamics of these concepts because they have tremendous implications for how we go about doing our business? So it comes back to this well, what’s the reason for accumulating knowledge?

DR. ROBERT FLODEN: Thank you.

DR. ROBERT FLODEN: I confess I was sitting there listening to the two of you summing up and doing a beautiful job of it, and I kept thinking to myself this has been a really interesting two days, and I’ve been to many other similarly interesting two day meetings over years where we’ve talked about many of the same issues. The thing that strikes me as different today, which makes what we have to do much more urgent, is that the context is different. We are operating it seems to me in a context that is so hostile to education research, to schools of education, to the complexity of what teachers are trying to do in classrooms that, you know we keep saying context matters in education, well context matters for what we’re doing. And I don’t know how we do this but it seems to me that this report somehow, I mean our final report assuming we write one, really has to have a chain of activities that could somehow begin to build infrastructure in the field because if we leave here and we’ve written another interesting report and we still have a field that’s totally syntrifical(?) as opposed to coming together, we won’t be any better off. And given the hostility of the context, I mean Jerry Schroof(?) was telling me this morning about a new study that people are clamoring for of ed schools that apparently, maybe you can tell us, or Lynn Chaney is going to be involved --

DR. ELLEN LAGEMANN: But there are all sorts of possibilities looming that are really very frightening. So I think we really have a very difficult and urgent problem and I hope when we get to recommendations we can think about real action steps.

PARTICIPANT: One of the things that I heard in the background underneath a lot of what was said yesterday and today is in my mind related to our overarching theme of community and who’s talking to who, who’s working with whom, teachers working with researchers, yes or no, researchers working with other researchers, yes or no, but to really think about that whole sense of collegiality and community building that is so important to be able to cumulate knowledge as opposed to all the disparate points of light out there doing their own thing.

DR. BARBARA SCHNEIDER: I want to follow up a little bit on that because I kind of had a little different take on things Bob than you did in terms of results with context, because most people always have started out with the concept that context matters, I mean we’ve been working in this field for a long time, we know context matters. But if we just think that we have results and that these results matter so much by context then it comes to the point that we don’t have any kind of findings that cut across all context. And I think that that’s an issue because I think we do have some things. I think that David and Harris yesterday when we they were talking about issues of class size and how it really is one of the things that it uncovered about classroom management was really important, and there are other kinds of things that we also know from a whole bunch of studies about knowledge. And I think that this really gets to some of the questions about the fact, about ed schools and how educational research is being devalued, which is that we’re just a bunch of desperate little studies that don’t come together in any serious kind of way. And I don’t think that that’s true, I think that we have knowledge and yes, context matters, but the way the context matters is very small, these aren’t huge things that we’re talking about in terms of making gigantic differences depending upon whether or not we’re looking at one classroom in a suburb or another classroom that’s in the city. So I think we have to be careful because it gets the impression that we are in fact a field that is so terribly diverse that there isn’t any kind of universal findings that cut across many of these kinds of situations that we look at.

DR. ROBERT FLODEN: I would agree with that, and I think that each of the case studies that we have and some of what was reported today would support that and what Barbara Rogoff was saying was both about context mattering but also about large scale changes in understanding of how people learn.

DR. BARBARA ROGOFF: I was just going to amplify that a little bit, I think that context matters in a big way, but that doesn’t mean there aren’t regularities, in fact I think the challenge for us is to be describing the regularities in the variability rather than assuming that the kind of generalities that we can make ignore context, which is the way that in the past it’s sometimes been cast, we should be trying to figure out which aspects of context matter for what and come up with descriptions of the regularities that give us some ways to proceed that incorporate the variability’s that context represent. So I think that we need to go beyond saying context matters and actually going beyond saying it depends, to say it depends on what, how, and we know some of the answers to that and I think that was some of the things that we’ve been discussing here, but I also think that we need to push ourselves to get beyond the dichotomy between saying universal laws and contextual specificity to say the universal laws involve contextual specificity in ways that allow us to make generalizations that are true to the complexity of the world.

DR. LAURIE WISE: One of the things that I would add to this is this idea of community, it needs to be not just narrow little communities of people that work together in a field, that talk to each other but don’t make known to the more diverse range of educational researchers and to the public and to students, these more general principles that, the things that we do know.

DR. REX COSTANZA: I’m Rex Costanza from the National Education Association, and just a couple of observations. One on the whole follow-up on some of this really has to do with dissemination and I was absolutely thrilled with the whole set of discussions but the session yesterday afternoon on dissemination actually disappointed me to some degree because people talked about confidentiality problems with dissemination but not a new paradigm, a new way to disseminate the information. Related to that, the information about relating the research to practice, many people talked about, or several people talked about the need to get more involved with practice and understand it and I’m sure that people that we represent, the teachers, would agree with that. But I think new ways need to be found to do that, there were some clues perhaps, situation in San Diego where they’re using the web to interact between the teachers and the researchers may be a clue but this whole area about dissemination, the difficulty of putting it into practice, that was discussed a couple of times, that it just atrophies unless you keep working at getting these ideas and practices in place. So much needs to be done there that I suspect there really is a good bit of knowledge that we could do a whole lot better with if we had a way to put it out there.

DR. DANIEL BERCH: Dan Berch, NICHD. At the risk of making a statement that sounds as Pierce was saying, precious, I think there’s a distinction that hasn’t been raised yet exactly, the closest was David Cohen’s distinction between cumulation and accumulation. But I think I’d like to distinguish between the accumulation of research findings and the accretion of knowledge, which seems to me a much larger issue and that’s where notions of theory I think are brought to bear, so I’m not minimizing the importance of the meta-analyses and other types of approaches to synthesis but those tend to be fairly focused efforts. And while they certainly contribute to the accretion of knowledge I’m not sure that we’ve made a lot of headway in discussing that sort of issue.

I guess a second point that I’d like to make is this, we’ve been sort of using terms like external validity and generalizability interchangeably and there was a paper a number of years ago, the American Psychologist by Binagi(?) and Crowder(?), it was actually called the Bankruptcy of Everyday Memory, and in there they make a distinction between external validity of research methods and generalizability of results in a sort of two by two table, high and low for each of those, and argued that there are cases in which you can’t have low external validity of research methods and high generalizability of the findings. I think it’s a useful paper to revisit because we’re all making assumptions that argue that external validity has to be high otherwise you’re not going to have generalizability of results. I think, well, I’ll leave it at that. Thank you.

DR. ROBERT FLODEN: Thank you all very much, we’ll try to build a community that goes beyond just the small groups and helps us figure out how to continue to improve the ways in which we can increase our knowledge about education. Thanks a lot.