Workshop on Understanding and Promoting Knowledge Accumulation in Education:

Tools and Strategies for Education Research

Day 1 – June 30, 2003

Framing the Issues Question and Answers

DR. ROBERT FLODEN: Okay. We are going to shift now to a question and answer session, and we will start with questions from the members of the committee. Members of the committee?

DR. ELLEN LAGEMANN: This is a question primarily for Kenji and Ken, but for both of you.

Kenji talked quite a lot about the issue of community in educational services, which I think is very important, but you didn’t use the word “standards,” and I think one of the issues that is closely related to community is the lack of common standards in education research, and I would be curious to hear both Kenji and Ken and others comment on it. I wonder from an interpretist point of view where the issues of standards exist?

DR. KENJI HAKUTA: I think I would just like to distinguish between whether the standards are top down or bottom up, and that what you really want is the standards that are accepted and generated from the community, and borrowing from other related communities and so forth, rather than being composed from the top down.

And the other thing that I would just say is that as a linguist of sorts, many things of sorts, but a linguist of sorts as well - that linguists talk about things like speech communities, emphasize not just the commonalities and common standards, but also the very ability that occurs around the standards that are also an important part of the definition of a community is also - teach the field or the community impact a lot, and so I think what you want is both kind of a bottom up and common accepted standards as well as variability around those standards that is sort of accepted.

DR. KENNETH HOWE: Well, I think part of the question of specificity of the standards – I don’t know who actually proposed some standards in 1990 or something, which many people found useful, but I think they are extremely general, and, insofar as you can make standards that are general – that is what the standards do in a certain sense. Standards may often be a response to an outside force - maybe there was something wrong. Somehow you have to lay down what the principles and standards you can follow should be.

It would be interesting to find out what the other folks on the panel think about that question, the question of standards, for example, in biology or standards in medical research, standards in business. Is there a similar issue there?

DR. ROBERT FLODEN: You all want to respond to that?

DR.SYDNEY WINTER: I think there certainly is such an issue when there is an issue of passiveness, as you say, I think to a large extent when the relevant standards are going up in communities and they are implemented in various ways, in personnel reviews and article reviews and so on, and mostly go relatively uncodified, although some people act as if they thought they were codified and widely published and therefore clearly authority.

I was wondering, on the question of standards, whether the question of data sharing links into that subject, whether people, other than the authors of the original study, get a good crack at the original data, which I think is really quite a fundamental issue about the way research is organized and there is some trend to create better standards on that.

DR. ROBERT FLODEN: Dick, are you still there? I think we lost him.

Other members of the committee?

DR. DAVID MCQUEEN: I just have a quick comment on the standards, from a medicine view(?) of standards all over the place, and guides to political practice, guides to preventive practice, guides to community approaches to public health, and these are big, fat, thick books constantly being revised by armies of workers. (Laughter). So it’s an industry, I would say.

DR. KENNETH HOWE: There was a recent story in The Times - says that physicians don’t follow those.

DR. DAVID MCQUEEN: The Times might say that. (Laughter).

DR. DAVID KLAHR: My name is David Klahr. I’m a new kid on the block on the committee – this is my first meeting.

David McQueen said something about education as a field of action, and I would like to raise a question related to that, and that is to get your reaction, any of the committee members, to the view of education not as a science, but as an engineering discipline, and if you look at a successful engineering discipline - let’s take spacecraft design for example. It is based on basic research, research in, you know, material sciences and research on all kinds of stuff, but, eventually, you build spacecraft and you don’t do by design. You take your best intuitions. You put something together. There is tremendous variability from one space shuttle to another, even in the same series, as now we all know in the Challenger series, there’s many different vehicles, even though they are all of the same general type, and the application there and the expectations are quite different than the basic sciences, and there is knowledge accumulation in that field. I don’t think anybody could deny that.

So is that a useful analogy? Can you look at the kinds of knowledge accumulation in the engineering field and use that as a model instead of knowledge accumulation in basic microbiology?

DR. DAVID MCQUEEN: I’ll respond, because I think that is a very good point, and I think that this is the issue of practice versus research and basic science versus empirical, everyday, practical issues.

My analogy I always liked - and this, I think, has very much to do with the whole area of complexity. These fields are very complex. Basically, classical research science, in a way, tries to reduce everything down to its simplest components, so we can study that little micro-component, and that is why it is very successful at doing that, because it takes out the rest of the world and just focuses on a very specific problem, and then it does extremely well.

But our problem is at the other end, that everything is incredibly complex, and, you know, I always think this when I - the two analogies I think of, one is when I get inside a 747, I always wonder, “How is this thing actually going to get off the ground?” And I am always amazed every time that, in fact, it does, and it lands as well. I mean, this is really incredible, when you think about it, because when you think of all the engineering that is in that plane.

And the other analogy I like to use is what I call my centipedal question, and that was a centipede, you know, with all its many legs, was walking along and another insect came up to it and asked the centipede, you know, with all those legs, how could it walk, and the centipede thought about it and could never walk again. (Laughter).

DR. KENNETH HOWE: One thing about the engineering model is I think partly correct, but one of the big questions in education is what we are trying to understand, the quality the excellence, citizens, workers. So every things an important part of what this controversy is about and what sorts of research to undertake, because we have to find quality and so forth.

And secondly there is a question of how this effects various recipients - When we engineer a certain sort of curriculum or instruction, how does it effect the recipients? And recipients effectively differentiate. These may be problems in engineering as well. I

DR. MARGARET EISENHART: David, you mentioned the distinction between research and practice, and you brought it up, actually a couple of times, and I think that it would be fair to say that a general perception is that the relationship between research and practice works better in medicine than it does in education, and I am just wondering if you and some of the others where you have a sense of research in practice being important activities that are going on in your field, what could you say about the relationship between research and practice that would be helpful to us in education?

DR. DAVID MCQUEEN: Gee, I don’t know if I could say anything that’s helpful. I think the issue in the whole area of medicine is that it is resources. The resources that are given over to both the practice of medicine and the underlying research that goes on at NIH or in universities and the CDC and places like this, those resources are just enormous and probably a comparison to education, particularly the resources forward for basic research ties to all of the biological, chemical, physics, lots of research areas.

So, to a certain extent, I think it is a contextual problem, if you want to look at it that way, that medicine is an area that is highly valued in society at such a level that most societies are willing to invest enormous amounts of money in basic research, and basic research gets tied to applied research because they are both areas with lots of resources. To me, that’s the simple answer.

So it’s a question of making education have the same kind of value in society as medicine has.

DR. HELEN LADD: I just want to follow up on that. Isn’t it true, though, in medicine that there is something called “translational research” -

DR. DAVID MCQUEEN: Oh, sure.

DR. HELEN LADD: - where people quite exclusively set out to take basic research and figure out the applications? Because I think that’s what were missing in education.-

DR. DAVID MCQUEEN: Well, sure, and, in fact, that is an area that has risen particularly around - after the Cochrane efforts and the CORE evidence debates came up, but, yes, in fact, at the CDC, we have - actually, we have divisions that are called “Division of Translation.” For example, our diabetes division is not called “Diabetes Division.” It is called the Division of Diabetes Translation, which specifically has this task of this intermediary between what is the best of basic research that we have with respect to what we know about diabetes and its causation - and what do we know about application, how to get that message to people who have diabetes, and could possibly have diabetes, and so that is - I think that is a very useful distinction, and it is that link between research and practice that could be developed.

DR. ROBERT FLODEN: And is that link a matter of communication, figuring out what to say to the people who have diabetes or is it to come up with practical treatments that doctors might use with patients?

DR. DAVID MCQUEEN: Well, it’s a mixture. It depends on where you want to go. I mean, the issue is - the focus is on the translation, you know, how do you get it. So, I mean, if one issue, for example, is you know from evidence that you have collected that there are lots of people with undiagnosed diabetes, then the translation problem becomes one of how do you connect that group to - that don’t know about diabetes, how do you make it manifest to them? And so that becomes an educational issue, and so there are lots of areas like this, and sometimes it is very clinical and technical that there is some new device that is available and how do you get that knowledge out there to the general public. So, yes, it is very concrete.

DR. ROBERT FLODEN: We are going to open up now for questions from the floor - and, please, give your name.

MR. PIERCE HAMMONDS: Good morning. I am Pierce Hammonds from the U.S. Department of Education.

I want to ask you a question about transparency and obligation for transparency and its benefits and perhaps its costs.

Kanji had said that he didn’t think that having standards imposed - that is, top - you know - standards - was a good idea.

From the point of view of the public, the academy is top down, and so bottom up to you is top down to somebody else, and that goes to the transparency point.

If we are concerned that the public thinks that research is a waste of time, they don’t pay any attention to it. “They are playing games again. We don’t understand this. People don’t agree.” Isn’t it better if they understand the standards as well as other things?

Furthermore, if we are concerned that there is a certain amount of imposition that is happening, that games are being played with the definitions of research on those who are imposing them from on high, because they have the power to do so, transparency would make it harder for those games to be played, as well, would it not, and wouldn’t that be a benefit? But then what would the cost be of opening that up?

Thank you.

DR. KENJI HAKUTA: Well, yes, I think that the issue that we saw by trial being imposed legislatively was really a symptom rather than having any causal factor, and I think it really did - you do speak to that, which is that the public doesn’t feel - whoever the public is - the public doesn’t feel like this is - it is being regulated sufficiently, and somehow the public feels that NIH study sections do it more - better and they are safe in that process, even though that process itself - having served on the study section, I can tell you that, it lacks transparency to anybody outside of people who are in the room, but I think that - that is where we want to get to, and I think part of the issue is, you know, what is the most effective way by which we can get to there being transparency in the process.

DR. HARRIS COOPER: My name is Harris Cooper and I wanted to encourage more discussion for the committee to consider more fully the notion of the medical model and perhaps - we heard a lot here this morning about disabusing people of what the medical model is or actually curicature(?) that we hear people use in education relative path medical research actually operates and that the medical community, medical research doesn’t really even operate under the medical model as we think about it in education.

We have heard about concerns about dose response issues, concerns about genetics, age, gender differences and how these interact with medical findings. We have heard about use of multiple methods in medicine. Certainly, we are all very familiar with the SARS epidemic or scare or whatever you would call it. Clearly, that is a branch in medical research that leads to policies and practices that have very little to do with experimental work. We have heard scaling issues in medicine and issues of accumulation.

So I think there is a real disconnect between this thing called a “medical model,” as education researchers in medical models as actually implemented and disabusing this pristine notion about what goes on in medicine would clearly be a benefit.

DR. FLODEN: That seems to have been more a statement than a question. (Laughter).

SPEAKER: I have been listening very attentively, and I have a number of things that I am trying to juggle in my mind, but the idea of accumulating in a field, as opposed to a discipline, obviously poses some challenges, and I am also intrigued by the notion in biology that controversy has served to stimulate growth in knowledge and accumulation of knowledge.

One of the concerns I have had over the last couple of -- whatever is the tone in education and the sort of decisiveness or the lack of respect across disciplines, and I worry that we’ll never get to the point of really accumulating in this field if that tone isn’t somehow addressed, and one of the gentlemen here pointed out the issue of resources. It may be just a fight over resources that makes these debates in education seemingly so vicious or so negative.

I’m wondering, number one, whether in other fields you see that kind of tone, and, number two, if you have always seen some kinds of ways in which to bring people together, even if they have very different perspectives, values and points of view on knowledge as an issue?

DR. FLODEN: Sid, you want to take a crack at this from the point of view of business?

DR. SYDNEY WINTER: I think it would be pretty optimistic to think that we have ways of actually closing some of these gaps of controversy. Certainly, it often seems to me that it is part of a step forward even to get the two sides or the five sides or however many there are, into the same room together, so at least they can hear what each other has to say and to perhaps expose some less committed audience to discussion.

I often feel that on many of these issues, you know, the trains pass, the ships pass completely in the night and you don’t get any real engagement in the controversy. So while we are setting standards, we could use some standards for intelligent debate. I’d say that is a very pervasive issue in this society.

DR. GARCIA: Gil Garcia of the Institute Education Sciences.

When I look back at the 20-plus years of research that was launched by the federal government on linguistically- and culturally-diverse students, at that time, we began exploring this very diverse field of bilingual education. We started with a population of students that we referred to as “limited-English speaking,” and as years went by and our research accumulated or our understanding of who these kids were and what their problems were and what the educational solution is for their success needed to be, we changed our definition to “limited-English proficient,” because it captured conceptually and practically a more robust picture of these students.

And then I shift forward to right now, the Biliteracy Research Initiative that IES and NICHD are cofunding that I believe is going to contribute immensely in at least one more year, but already has, and one of their contributions is the development of this very robust protocol for an instrument for collecting in-depth features and characteristics of the population of English-language learners as they are now known are today, and it goes back to, in part, Kenji’s comment about context.

In the case of bilingual education and serving English-language learners, clearly the accumulated knowledge has told us that we had better pay not only close attention to this context, but we had better understand it before we even begin to think about making assertions about who these kids are, what these kids need and what teachers need to know.

And I think what is implied in this body of research and it is not unique, per se, is that, in fact, we were following standards 20-plus years ago for conducting research, and we were, in fact, building on what we understood about the theory of teaching and learning in this context of limited-English-speaking students to English-language learners.

So I guess my point is that I would invite the committee and I would invite the panelists to reconsider the loose statement that, in fact, in education or education research either, we don’t have very many standards or the standards that we have are very sloppy, and while I would be the first one to say that we’ve got a long ways to go in education research, I’m not so sure that they are either very few or generally sloppy.

DR. ROBERT FLODEN: Perhaps on that optimistic note - (laughter) - we should call a close to this session. Please join me in thanking the panelists. (Applause).