Workshop on Understanding and Promoting Knowledge Accumulation in Education:
Tools and Strategies for Education Research

Day 2 – June 30, 2003

Agenda Item: Ways of Taking Stock: Replication, Scaling up, Meta-Analysis,
Professional Consensus-Building - Dr. Hugh Mehan

DR. HUGH MEHAN: The overall theme of this workshop following on to yesterday afternoon’s workshop is ways of taking stock, and as someone said I’ll discuss scaling up as a tool or a strategy for knowledge accumulation based on my experience in qualitative research in the study of school reform, or what I prefer to talk about is school improvement. I rather think about school improvement because school reform doesn’t necessarily mean that issues of student excellence or equity, social justice are obtained.

In framing my remarks I’m going to call upon two different studies that I’ve been involved in in recent years. One is the study of school reform efforts, including the scaling up of a program that started in San Diego, the AVID program, which stands for the Achievement Via Individual Determination, which is kind of what I call a tracking program. And I’m participating in another set of studies concerned with CSRD programs that also were involved in scaling up so we have scaling up in two senses of the term, one, school improvement activities started small and got big, and two, research projects that started small, have traveled with them to get bigger. So as the programs traveled from one school to many so did our research.

Second, and really is my current work also concerned with school reform, is a study of a program, a reform effort that started in one city, New York City, and has moved to San Diego in the form of Tony Alvarado(?) who was superintendent in New York City, District Two, and moved to San Diego and actually now there’s a termination to that time, from 1998 to 2002, and actually has had his employment terminated. So lots of people have had their jobs change in not just moving from one position to another like Ken Jihakuda(?) and Harris Cooper, but subjects of study have also had their employment terminated from one time to another.

But that issue of studying an effort that has traveled from one city to another has posed an interesting set of challenges. And interestingly, both of those studies have been funded by a combination of Spencer and OERI and my benefactors happen to be both, both of them happen to be here today, so I want to acknowledge Gill Garcia and Ellen Lagemann because our Spencer work took place during the time that she was at Spencer, so I think there’s an interesting set of convergences there.

There are a whole set of research challenges that are posed when studies that start small and then get bigger that I would like to talk about. Any time that a study gets big like that a whole set of methodological issues are placed before researchers that are different then the more classical ethnographic form of research. If you can think of classical anthropological work, and although I’m a sociologist I tend to do more ethnographic work, my anthropological colleagues kind of see me as a fellow traveler in that respect, classical ethnographic work as you know has been at the village level, or the small society level. And when brought into educational research the unit of analysis has also tended to be small, a classroom for a year, a school for a year, analogous to the village for the year. But when programs that you’re studying, such as a reform effort, get big, then the research enterprise has to get big along with it, and that poses significant challenges to researchers who are used to working on small scale.

And as we were concerned with issues of standard in rigor in research then we have to attend to those movements in making sure that the standards that are associated with the small easily controlled study in the sense that a person has got control of the research from top to bottom is taken into account, because the big issue is that as research goes to scale, gets larger, extensive teams of researchers have to be involved in the project. It is no longer possible for a single researcher to have control of the work in the classical sense of Malinowski(?) tenting in the Trobrian(?) Islands where he set up his tent and then gathered his field notes and then did all those observations. It isn’t possible to do that when you have a research project that goes across several cities, several states, many different periods of time, so issues of scale pose particular problems. And although the work that we’re doing is not on the scale of Linda Burton’s work that you mentioned yesterday, Barbara, in the Three Cities Study, I’ve talked to her about some of these issues and they do carry over. There are a whole set of particular problems that I would like to mention.

One of them has to do with trainings of teams and orienting researchers to a set of research questions so that when they enter the field they are oriented to the same sets of issues that the researcher thinks is important, the PI thinks is important. And this has a double edge because you do want your research team to be oriented to the same set of issues and you want them to be focusing and gathering data against a protocol, which is the general way in which this is done, and that’s good because people will then come back with the field work, that field notes that you think are important. But that can be limiting, too, because they can be so faithful, so literal in the gathering of that data, that they do not see the emergency phenomenon that is also crucial in any kind of field research. I think that is the thing that I have learned the most over several years of field research, that you go in studying a particular phenomenon and all too often, or all so often, you come out with information that is considerably different than when you went in and that’s the value of that enterprise. So familiarity orienting questions has to be balanced against a blindness to knew phenomenon.

Once materials are gathered then a new challenge is presented, and that is trying to get consistency in material treatment. And there’s been some interesting technical development, programs like Nudist(?) for example, enable field notes to be gathered and assessed across huge batches of data, which are very interesting. Now this faces a particular challenge, again back to what Barbara was saying yesterday afternoon, in trying to make information publicly available, data sets where you can mask names and the rest present particular challenges putting field notes on publicly available data sets, presents a whole set of other problems because you can mask a name but the description itself, if it’s rich and thick like it should be, could also reveal confidentiality considerations that we want to avoid. What I’ve done to deal with issues of that sort is having my team go out, gather data against a set of protocols, and then trying replicating the old fashioned way of doing ethnographic work, have each of them do a case study of their particular setting. It might be a school or a classroom that they’ve been studying in Kentucky or some other place, and then we try to find themes that go across those cases and that becomes the end product.

Another note of difference in such studies is the published result, it’s not unusual for an ethnographic study in the classical form to be reported in article form. But I have found that it’s difficult in doing this kind of larger scale work to reduce it down to an article or even a few, so rather the monograph now becomes the unit of publication, different than the article. It’s not quite a book in the sense of a synthesis, like Barbara’s new book on culture and cognition, as a review of and synthesis of lots of work. The monograph is an intermediary form of publication that presents its own form of challenges.

One of the things that it’s often been said about qualitative research is that it doesn’t come up with any basic principles, that there aren’t any rigorous laws or rules about its findings, and I found that that isn’t true. I found out that I have at least one abiding rule or law that I can talk about in our kind of work, and that is it depends. You’ve heard a lot about context dependence yesterday, but I think that that statement can be reduced to that basic law. So when anyone says that the kind of qualitative work that we do across any kind of these settings doesn’t produce substantial findings, I say you’re wrong, the basic law is it depends.

Now what I think we have with the kind of work that we’ve done over these two sets of studies is an existence proof. A proof that it is possible to scale up research from one small particular enterprise, one particular small case in one arena and move it across to other large situations, away from that original center of research to many other sites, so at least at that level we have the proof that we can carry out that kind of work and then draw implications from it for other kinds of studies.

Now I think that there are a whole set of interesting relationships between quantitative and qualitative research that we should take into account, and some of these were discussed yesterday. In unfortunate relationships quantitative and qualitative research has been carried out autonomously, where there’s been no dialogue between the two. It’s also possible that qualitative research can be useful in assisting large scale quantitative research in that a phenomenon is discovered in a small case study that can then be used to generate a hypothesis that can be tested out in larger scale enterprises.

There’s also another relationship between qualitative and quantitative research and that is that when quantitative research finds a certain relationship between factors then that relationship can be explored for what we heard yesterday talked about as processes, practices, or mechanisms. And that has been the approach that I have taken, that third approach, that either other research has formed relationships perhaps between background characteristics of students and certain kinds of educational outcomes, or in the case of our AVID research where we found that a particular program seemed to have educational results where students who participated in the program seemed to do well.

Well, that was a finding that required, in fact encouraged us to do a greater exploration in depth to see what was behind that seemingly set of relationships such that ethnic minority students were going on to college at rates that far surpassed local and national averages. African American youth were going on to college at a much greater than were expected by local and national averages. So with that idea of that quantitative finding went into first a local consideration and investigated in the local context, by that I mean San Diego, we looked at 18 out of 24 high schools in the local area that had these programs, which is already a large investigation from the classical ethnographic point of view. Then as that program got traction and spread throughout the state of California and then the United States we scaled up our research in the ways I said in the beginning but keeping in mind that we were motivated by a finding that came from our quantitative investigation, the relationship between program intervention and students outcome, and I think that that third way of mixing methods between quantitative and qualitative is very fruitful. And part of that is based on the assumption that it is fruitful and helpful to determine what’s going on in an intervention before we come to the conclusion of did this intervention work.

So we’re trying to take the ethnographic motivation of what’s going on here and apply that to intervention studies in its own terms and then determine, that is to say if a program such as AVID, one of the ones we have studies, is vitally concerned that its study skills program is the one that is making the difference, we went investigating that to be sure but what we found is a result of our studies both in a local context and a national context was yes, that study skills program was helpful but other phenomenon emerged from that study which had to do with interestingly peer relationships and parent/student relationships which were not part of the official dogma of the program. So by simply asking the question what’s going on here we were able to determine information that was helpful before deciding whether the intervention worked, and we have found that that strategy is particularly helpful.

One other point that I’ll make in conclusion is that, I can see you coming, this research inevitably puts researchers in personal contact with the subjects or the participants in research, and in classical ethnography researchers are instructed to be flies on the wall. I haven’t found that strategy to be productive, indeed impossible, it may be possible when you don’t speak the language of the people you’re studying but when you’re involved in research in your own communities people know you, you know them, they’re asking questions about your research, and I think there’s an ethical responsibility to engage the participants of research in ways that are dialogic and not, that are fraught with the power relationships that are built into the classical approach to ethnography. This has been a particular challenge because in both of our recent studies of school improvement we have taken the challenge that Ellen has laid down when she was with Spencer to make our research useful to the participants which of necessity requires engagement with the participants, not after the study is done but while the study is unfolding and that places particular strains on research that has a certain engagement or intimacy in the project.

In some cases we have found that the people when we were investigating the research settings have been very pleased that the reports that we give them either in miniature form as we’re going along or reports at the end. In other cases superintendents have been loath toward the findings that we have provided because they do not match the public relations positions that the school district wishes to put out. And one of the superintendents with which we have worked sees research more as the public relations tool then as a basic research tool, the results of which could be helpful in formulating policy. And those kinds of strains put a particular, those kinds of conditions put a particular strain on the research that we’ve been involved in as we scale up research from one setting to many.

So those are a couple of comments, there are many, many more to be made about this but within the timeframe I think those are the ones that I’ll stay with. So thank you very much.