Workshop on Understanding and Promoting Knowledge Accumulation in Education:

Tools and Strategies for Education Research

Day 2 – July 1, 2003

Agenda Item: Wrap-up - Summary of Themes and Concluding Comments – Dr. Robert Floden

DR. ROBERT FLODEN: We are going to get going here, this is the final session; Laurie and I are going to try to look back over the last couple of days and pull things together in some way, quite a challenge. So let me start with challenges because I think that’s where we started yesterday was with some of the challenges. We’re trying to think about what we’re now calling cumulation of knowledge in education research and there are a number of features about educational enterprise and about research in the field that pose challenges to cumulation. One of them is the fragmentation of research in the field, the number of different things that people are doing, things they’re looking at, ways they’re approaching things, assumptions that they start from, and the sheer diversity in the field, although that has its advantages as well, poses a challenge for cumulation of knowledge in the field.

Another challenge is what we’ll now call Mehan’s Law, which results vary across context, so if there’s one thing we know it’s that we don’t know about anything that’s general; it’s one of these paradoxical things. Things vary across context, which makes it difficult to accumulate knowledge about things in general, although perhaps within a particular context it’s possible to continue to learn more and more.

And we want knowledge that will support improvements in a variety of contexts, and we’re about education, we’re about trying to make improvements, and as much as possible to draw from research and use the research in as wide a range of context as possible, which takes us back to the theme of this workshop, which is on the cumulation of knowledge. As I listen to the presentations across the two days one of the things that we had charged people to talk about, particularly the people on the panel on the first day was about some of these terms like cumulation and accumulation and I notice that we continue to use these terms but it’s increasingly clear that they get used in different ways, so replication is a replication of something that means doing the same thing over and over again, does it mean making some changes in the study and seeing what you learn when a few things change, how much change would be possible for it still to be a replication. So replication seems important but we’re not quite sure how we want to think about replication. The same can be said about generalization or the process of scaling up.

One of the challenges is that as a field it often seems as though researchers in different areas are, and I won’t quite get this right, but David Cohen said something like ships passing each other and shooting each other as they go by, not quite passing in the night, which may be a little worse than passing in the night because they’re not just passing in the night but trying to destroy each other on their way by. And some of the things that Ken Howell had to say about different approaches to research and how if you take any one particular approach there are flaws with that, and as he’s written about in the past we get in sometimes to these debates about well, should it be quantitative or qualitative, should it be a randomized field trial, should it be something else, and rather than finding a common ground to engage around what’s most appropriate for answering a particular question what we seem to do sometimes is to be criticizing each other, perhaps as I think Harris Cooper said, overly much, spending too much time doing that.

There’s a challenge in the field of capacity, of people knowing enough to do this cumulation, to do the high quality work that would make it something that we’d want to cumulate together with other things. And there’s a need for capacity, and I was really struck with what Bud Mehan said of both having the training to be able to look for what you’re supposed to be looking for in a study, to write down the right things, but to do that and at the same time be able to step back and notice what he called the emergent phenomena. This is a pretty sophisticated level of knowledge of how to do research, that you’re both able to know what it is you’re supposed to do and to follow the procedures as strictly as need be but still have the judgment and the distance to be able to notice something beyond the protocol that might be important.

We have a challenge we talked about yesterday afternoon of the need to attend to privacy concerns, while at the same time wanting to be able to lay work open to scrutiny, to be able to do secondary analysis, repeated analysis, the forms of replication that come from looking at the same data set and either addressing different questions or addressing the same question from a different approach.

Ellen Lagemann said yesterday in response to what Barbara Rogoff was saying that it seems like daunting work and indeed the whole two days makes accumulating knowledge seem like daunting work given all these challenges. Cumulating knowledge is possible but not for the faint of heart. We did have, though, when we started talking about particular things that people have worked on and especially as people look back over the history of the development of a line of work, you can see that despite the fact that this is daunting that people are successful. We had a number of examples we looked at that showed how a knowledge evolves or accumulates over time, the work on production functions and economics showed a progression as people used more sophisticated models, tried to add more things into their analysis, the work on describing classroom practice, David Cohen gave a good description of how they spent a couple of years trying to learn from what had been done before so they could do work that would provide a better representation of classroom practice so they could get more valid and reliable answers to questions about the connections between practice and student achievement.

We have progress that Barbara Rogoff traced in our understandings of how people learn, progress in methodology and in meta-analysis for example, progress in Success For All as Bob traced the history of work showing how they got to the understandings they have now of what aspects of the program are helpful. And this is, what people were describing was not just accumulation, not just saying we’ve done a lot of things and now we’ve got a lot more studies than we had before, but that as we listen to these accounts they were clearly accounts where things were accumulating, that is people were fitting things in, making sense of more of the puzzle, and often replacing one idea with another, having something that they thought was appropriate so people thought that it was just working in cooperative groups initially that seemed like a good thing but on closer study the thought that just being in groups was going to be a good thing was thrown out and replaced by the idea that you needed more structures, and particular sorts of structures if students were going to learn.

Similarly, in the methodological work Harris Cooper talked about the developments in meta-analysis, increasing statistical sophistication and changes in views about what are appropriate ways of accumulating knowledge across studies. Now all this sounds a lot like descriptions of growth in scientific knowledge in other fields, that is you have things where people are doing work, they’re doing different sorts of work at different times and there is cumulation which includes sometimes just getting more detailed understanding of something you understood pretty well and other times it involves rejecting one idea and replacing it with another.

We also got a chance to compare how cumulation of knowledge in education works in some other fields, medicine, business, biology, were the ones that we focused on most. And one of the things that surprised me when we put that panel together, medicine, business, and biology, I thought well, medicine and business will sound a lot like education, they’re both sort of studied, they’re both fields of study rather than disciplines, but biology will sound really different. But repeatedly Jay Labov said well, you know things are really more complicated in biology than that, you can’t treat this as though this were a simple unified field where people all agree on methods of working and where knowledge is cumulating in some straightforward way. References to learning from the people’s who live in the Amazon about their understandings of the way the lifecycle works, not to accept that uncritically but drawing on that for ideas about new things to explore makes it clear that a range of fields face some of the complexity, some of these challenges that education does.

Controversies exist in other fields as well as in education, we’re not the only field in which the political winds blowing one way or another affect what it is that people want to do work on and the sort of challenges that are posed to the conclusions that people are drawing. In general I found that we need to get beyond a comment, someone made this comment yesterday, the overly simple ideas that we have about other fields, like medicine in the medical model, that too often in our conversations other fields are brought up in an overly simple view of what happens in other fields is brought in and held up as a model when if you look at them things are more complicated there and the set of challenges have some similarities at least to those in education.

This discussion of other fields also repeatedly came back to the idea of paying attention to practice in a practical field as a source of insights, that in education we need to pay attention to what people on the ground, in classrooms or in administrative offices, or in state departments of education know in some way as a source of insights, as a source of information about what problems are worth working on that need further investigation. So knowing that things in other fields are also complicated gives some solace and also suggests that we might look to the other fields for some ideas about what to do but I’m again reminded of Mehan’s Law, context matters and the context of education research isn’t the same as the context of research in medicine, business, or biology.

So a few thoughts on what we might do to move forward, to try to do a better job of whatever this thing called cumulating knowledge is. And I find in listening to the discussion people often trying to move from arguments that often pose things as extremes back to finding some mixture, to looking at a range of things in trying to find some Aristotelian middle ground. So one of the things that, one of these tensions where you could think about two different extremes is well, how should the field be structured in terms of what gets worked on. One extreme is what we sometimes think is happening in education, which is people are working on such a wide range of things that not much gets done in terms of cumulation because everybody is working on their own unique thing. The other extreme is well, let’s just agree on what the two important topics are, and make everybody work on those. Wouldn’t we make a lot more progress if everybody just worked on the same two things? Well, maybe, but that’s not going to happen and it’s probably not the best use of people because you’ve got to somehow provide both a sense of ownership for what people are working on and incentives for them to pursue things in which they have some interest. But something that provides more sustained focus in some areas, say perhaps by focusing much of the investment of the resources in education research in a relatively limited set of things, would probably help us in cumulating knowledge, unless you have a group of people working on something it’s hard to get the criticism from peers that’s helpful in moving things forward and it’s hard to get a sense of moving, of cumulation of knowledge if you’re the only one working in the area.

Another thing where we need to get some balance is between opening our work to scrutiny by a community of scholars, an important thing, and on the one extreme would be well, everything everybody does should be the data should be up on the net so everybody else can look at it. Balancing both with concerns for privacy and with as Gary Natriello pointed out, the work that is needed to make your work transparent to other people. It isn’t that easy to let someone else in on everything that you’ve done, it takes a lot of work to do that, and probably not everything we ever do is worthy of the investment to make it open to the full community of scholars. But some balance there is needed and probably just thinking about where we are now something that provides more critical scrutiny of at least some of the aspects of what we do would be desirable.

We need a balance between empirical investigation and theoretical development. Barbara Rogoff mentioned this as a thread in work on learning, David Cohen bemoaned the lack of theoretical progress in work on schooling and instruction, it seems like there’s either a cycle of back and forth or some balance that’s needed. You both need to use theory to guide data collection and analysis, but it’s important to remain open for emergent phenomena and as NCES does to provide data collection that’s a little broader than just the things that you think are theoretically central, so you can permit the answering of some unanticipated questions.

We need in the field to provide some balance between uniformity and flexibility; we talked about common measures and the importance of having them. People also repeatedly brought up the need to maintain some flexibility to change things to fit the context or as our understanding of an area grow over time. The need for structure, though, is important both in say things like common measures and in working towards agreement on what are better and worse ways of studying particular questions, in a way that gives people understanding of why some, what the advantages are of some ways over others, rather than just saying well, randomized experiments are good or they’re not good, or case studies are good or they’re not good, but to develop some understanding of why you might do things in a particular way because just as educators will proceduralize anything so researchers, and one of the things that often happens in research methods courses is you get the sense that people know the steps to follow but when you look at the work that students first turn out, not later in their careers but immediately after taking a course, it looks like they’re just going through the motions without any sense of why you would do one thing rather than another.

We need to recognize that the variability of educational phenomenon by context is both a resource and a challenge, I introduced it first as a challenge but of course variability is one of the things that we learn from and the fact that different, teachers will teach differently in different context provides some of the variation you need to understand why some approaches to teaching work in some context and others work better in other context. And one of the great strengths of education is that people do figure out how to adapt things to the particular context and as Bob Slavin says it may take some rigidity to get people to make an initial move from what it is they’ve always done, but once you’ve provided some loosening up so they can see a different way of doing things and can see the advantages of that, people are good at adapting things to the context in which they’re working.

So I think cumulation is possible and scaling up of educational innovations is possible, but not for the faint of heart. We need to marshal and focus resources and we also need to build capacity to use them well. We need structure for the field that supports data collection and a sustained attention to areas of inquiry, we need both research training, and I’m thinking about research training, research training spans from figuring out what the right questions are. Someone said in the panel before the most important part of research is getting the question right and that’s one of the things that probably isn’t taught as well as others in research methods courses in part because it’s hard to figure out how to teach people to ask the right question than it is to tell them which statistical test would be most appropriate. So we need both research training and we need structures that would support ongoing discussions of theories, findings, measures, and methods.