Workshop on Understanding and Promoting Knowledge Accumulation in Education:

Tools and Strategies for Education Research

Day 1 – June 30, 2003

Framing the Issues Moderated Discussion

DR. ROBERT FLODEN: Now, this next phase, I am going to moderate the discussion here. I’m going to pose some questions to the individual members of the panel here, but the panel members should feel free after someone has spoken to jump in and add things if they want to. So it won’t just be Q&A from me, but I’ll moderate the discussion.

And we are going to do that until about 10 o’clock. . And then we’ll shift to questions from the committee and then questions from the floor.

So there were a lot of different points brought up in these two presentations. Let me offer a short list of things that I think we might continue in this moderated discussion. One is context. The context - specificity of investigation or research and what sort of a challenge that is in these different fields, and how fields manage to try to grow knowledge, despite differences in context, if it’s a problem.

Second one is more focus on methodology cases, and work with individual cases has been brought up and in the fields that we have here - in particular cases play a big part in sort of work that gets done. So how the different fields think about what you learn from a series of cases, perhaps in contrast to an experiment.

Third, Kenji brought up the contested nature of some issues and the sort of mixed - the two-edged sword of public attention. There certainly have been areas in these different fields where there has been that sort of public attention. I think smoking(?) research is one -- business area there are lots of politically-contested issues, and I’m not sure in biology --, but how does that play out in terms of the field’s ability to accumulate knowledge - whatever that might mean?

Fourth, something I didn’t hear mentioned much in the first two presentations, which is the role of theory as something to hang knowledge as it accumulates, how that works and it is something that comes up in the scientific research volume and I wonder how it plays out in this field.

So let me start with business and go to the cases question - anything one hears about how business is taught, it’s around cases. Does research and business also revolve around cases, and, if so, how does one learn from a series of cases?

DR.SYDNEY WINTER: That is a very good question. There is a lot of case-oriented research in business, and I think it is very valuable, indeed.

I think it would be largely consistent, perhaps, with the second of Ken’s types of methodology; that is to say I think many of us believe that the cases are an indispensable part of the background to quantitative research and basically insists that our dissertation students attend to that point.

I think the case is out, however, when you consider the fact that business schools are, to a large extent, about trying to create effective managers. I think the cases have a different kind of value which is to more or less, you know, give people a vision of what it means to be pushing the levers that are available for controlling a management situation, and that allows me segue slightly into another point I wanted to make, which is that there is a whole area of understanding not the accumulation of scientific knowledge, but the accumulation of the knowledge that is used in practice – that is the accumulation of shop(?) or the school, and I would argue that there is a connection between these which is not widely recognized and which should be recognized in terms of the different kinds of encounters that represent the variability of the situation - context dependence. Uncontrolled and unexplained variation is the enemy from a managerial point of view, because it threatens output and quality, and is also the enemy from the inferential or methodological point of view, because it threatens generalized ability and accumulation.

And so I think it would be productive, in fact, to engage in a large inquiry now into the connections between the accumulation at the level of actual practice and the accumulation of scientific knowledge.

DR. ROBERT FLODEN: So just say a few more words about that, so how does that work then? So how do you learn from one case to the next from practical knowledge? People go back and say, “Well, you learned this from a study of GM, and, now - “

DR.SYDNEY WINTER: Well, you can read a - it’s a lengthy point, you know. If you can read a case on how somebody has made knowledge accumulate in practical or how somebody has a plot across the implementation gap and made some academic discovery work in practice, and reading that case is the very short form of an apprenticeship. You know, you get some exposure to the details, and you get some basis for understanding what quantities of evidence might be able to do for you.

So I think that it is a different kind of knowing and a different kind of knowledge that is being formed in the scientific knowledge, but there is an important practical connection in the two.

DR. DAVID MCQUEEN: Well, I listened to all this. It is quite interesting. I really think that the issues that have been brought up actually deserve discussion for at least the next week or so, because so many critical ones, and beliefs and misbeliefs that I think have been presented that really need to be looked at, but I think one of the big issues here is that separating out in your field is different from practice versus research, which is - it’s really a key concept.

If you look at practicing in medicine, the old saying is - the old saying in medical school is “See one. Do one. Teach one,” when it comes to the surgeon or something like that, you know, and that is knowledge (?), but, you know, it hardly represents any randomized clinical trials or anything like that. It is highly empirical and observational, and so I think that the research issues are very different than that to be distinguished directly from practice.

And the other thing that strikes me is that a couple of the assumptions that I read and heard about knowledge and its accumulation, that, I think, is challengeable in itself, whether knowledge accumulates and whether knowledge is just superceded by another background and that is an issue that I think is quite important.

And then with respect to the field of biomedicine, it is a huge field ranging from medical research to public health practice, and I would say in a great percentage of that field of effort, the use of RCTs is probably not terribly relevant, and they have been forward to the sine qua non of elegance(?) of evidenced-based by some groups, but, basically, a large number of groups of people working on the evidence issue that basically casts the RCT aside in principle approach to studying knowledge. That’s just a general comment.

DR. ROBERT FLODEN: So, Jay, when I think about biology, I think about this human genome project - take an example where it looks like people are adding to knowledge, not just superceding, but adding, that would be an example. Is that characteristic of biology?

DR. JAY LABOV: As Kenji pointed out, biology is so diverse, and there are so many different approaches to how people conduct biological research and the accumulation of knowledge that it is really very difficult to generalize. I would say in the case of the human genome, that’s probably one example where - as Chapter 2 in the book points out - there has been this accumulation of knowledge over many, many, many years, and a great deal of money invested which is also different, I think, than the educational research - So that there are a lot of different ways in which this knowledge accumulates.

On the other hand, other fields of biology, I think sometimes - as we were talking about here - the wisdom of practice, if you will, is very influential. The whole issue of extension services in universities are not only to provide farmers and practitioners with the knowledge of research, but also actually be out in the field and gaining practical knowledge about what is working, what is different, what isn’t working. So the whole field of ethno-botany, for example, is an attempt to try to speak to the people, for example, in the world’s tropic rainforests who live there to understand the accumulated knowledge that they have through various ways of folklore, of passing a lot of information verbally. To try to understand the kinds of things that they have been using which then can help guide the kinds of research in trying to find new compounds for various kinds of purposes.

So there is a dynamic interplay, and I think a lot of this depends on the field that we are talking about and the ultimate nature of what is trying to be discovered.

DR. ROBERT FLODEN: So when you talk about ethno-botany and learning from the accumulated knowledge of the people who have been doing things, then that seems to me to raise the possibility of sliding on a slippery slope where everybody has accumulated knowledge the whole time and there is nothing that distinguishes scientific research from anything else.

DR. JAY LABOV: Well, no, I think another part of science is that you go down many false paths, and that many of the things that you have set out to do don’t work the way that you expect them to work, and in the case of ethno-botany, I think that is probably a classic example. There are many kinds of folklore about various kinds of things that people use for food, for medicine, for other sorts of things, and when you actually test them you find out scientifically that there is really no basis for what is going on. You know, conventional wisdom could be tested in many ways.

But on the other hand, it provides some real insights into thinking about things that we never really expected to have some kind of biological property, for example.

DR. ROBERT FLODEN: One of the issues that Kanji spoke about was like contact and how the variation in contacts of the way things work in education is a challenge for getting knowledge that is general or for cumulative knowledge.

So let’s take the case of medicine. Is the same sort of thing true in medicine, that you have broad contextual variations that are a challenge or for trying to learn things in general about people and how to make them well?

DR. DAVID MCQUEEN: One thing that he said that I thought was really on target was that education is a field rather than a discipline, and this is one of the main problems in the whole issue around what is knowledge, what is evidence and so on, but if you are looking at a field rather than a discipline, you often have no theory base or a solid theory base. So if you look at medicine, yes, there is a theory based in biology and chemistry and molecular biology, but for medicine itself, theory is a unique concept, and the same could be said with education perhaps. I mean - but in other areas, health promotion and public health. These are fields of action that are charactered by doing things, by doing interventions, and you sort of grab and take theories and concepts from wherever, and this is an important concept - if you don’t have a discipline, it is very difficult to send up a theoretical base where you have hypothesis tested and causal relationships and all of those nice things that are demanded by a strict scientific model.

DR.SYDNEY WINTER: Back on the question of practice, I tend to think that the disabilities referred to in education and education research, a lot of the context dependents are actually a lot more prevalent in the world than education researchers may believe. Certainly, if you look into business activity, certainly there are many things which are not subject to these particular kinds of disability, but whenever there are these human beings around, especially when they are around and they are called “employees” or they are called “investors” or they are called “customers.” They turn out to be quite complex and quite diverse, and they cause the same kinds of doubts about context dependence and the same kinds of frustration in trying to make a thing work in Site B that works in Site A that you report in education research.

I think the issues here are very general, very parallel, and I think will be very productive to address them in that way.

DR. ROBERT FLODEN: Jay, you are nodding. Is something similar true in biology?

DR. JAY LABOV: Absolutely. I think if you take, for example, that in physiological experiments there was a great deal of criticism 25 years ago because the National Institutes of Health, for example, were funding research that focused primarily on subjects that were white and male, and we know that in human beings, women are different than men in many ways, and we also find the same kinds of things happening in many species of animals, where males and females, if you begin to test using only one type of animal or one gender, you lose a tremendous amount of information, and you can’t see the entire picture.

So that also provides a lot of confounding problems to your theory or your hypothesis when you begin to realize that you need to test, you need different kinds of subjects. There are different strains, genetic strains for example - that also confound the kind of theoretical construct that you have because they simply don’t behave or their physiology is simply different, and without beginning to understand this variability, it is very difficult to construct these grand schemes of how these things work.

DR. ROBERT FLODEN: I’m going to turn now to the contested nature of one of the things that Kenji points out is a challenge and also brings with it public attention, which is may be a benefit, and I’m wondering, in the other fields, whether the contested nature of some issue raises difficulties for trying to figure out what you know, and, if so, how the fields try to address those.

So let me go to medicine first, because I think of this in the case of smoking which was a contested area for a long time, perhaps it is still contested. How is our understanding of smoking effected by the fact that it was a politically contested question?

DR. DAVID MCQUEEN: Well, that is a great question.

I think that the contested issue of this area, smoking, is a good example. It brings in a broad idea that there is perhaps something out there called objective knowledge with regard to causality, dependent smoking and, say, lung cancer, that there is probably some objective reality there.

The problem is that almost all knowledge becomes qualitative in the sense as soon as you introduce the observer and then it becomes subjective knowledge, and the observer is both - the whole event of smoking and lung cancer, as well as all the interested parties and the observers, whether they are the scientists or the people concerned about smoking.

So you don’t - so that is what makes it very complicated, and makes it contested, and, in fact, one could argue that the cause of lung cancer is actually not smoking. The cause of lung cancer is the production of cigarettes. (Laughter). So it depends on whether you want to stick to your cause.

DR. JAY LABOV: Let’s blame the victim, right?

How about in business? Any contested issues in trying to understand things about businesses?

DR.SYDNEY WINTER: Well, there are, of course, violently contested areas in corporate governance right now, which produces a definite political tinge to that conversation, but I think the - probably more - the broader issue is not just a question of being politically contested, but things that are parts of larger social processes that are going on upsetting the applecart all the time, and I have been very struck in trying to supervise the disertation of students doing work in particular business areas of how it is hard to get through two or three years of work on the subject without having some merger or some regulatory blowup or who knows what, come along and mess up your research site. There is a great difficulty in just coping with the degree to which the situation is continually being transformed by considerations that are outside of the scope of your scientific inquiry, and that seems to be a valid analogous to the problem of the contested nature of education.

DR. ROBERT FLODEN: So, Jay, in biology is this a cleaner area, people just get to do research without this political interference?

DR. JAY LABOV: Never. Well, I mean, I suppose at one level, at the purest level of research, basic research where people really aren’t sure where it is going to go, where others haven’t recognized the potential for it to go somewhere that has political or social or medical consequences, I suppose one could pursue that, but, you know, you just think about the whole issue of evolution in biology. That is one the things - the good news is that it will keep me employed here at the National Academy of Sciences for the rest of my life and beyond that.

I mean, I think there are both internal and external controversies that you have to think about. The controversies within the field are always going on among biologists and palaeontologists and people in geology, you know, Darwin’s idea of incremental change in evolution, versus Steven Jay Gould’s idea of punctuating equilibrium, where things remain relatively static for a long time and then, in relatively short periods of time, change overnight.

Now, that, on the outside, is viewed as controversy, so that people begin to think the biologists don’t accept the theory of evolution, which is not the case at all. The idea is that biologists, most scientists accept the idea that evolution has occurred. The question is what are the mechanisms, and those kinds of controversies generate a great deal of excitement and keeps the field going, but, on the other hand, when it is translated to people who are not in the field, there are many social and political consequences, and I think evolution is a perfect example.