DR. TOWNE: Are there questions?

Let me just cue our friends in the spill over room if they would like to come in and join us and ask questions. Please feel free to do so now and the usual rules apply. Please go to the mike, identify who you are and direct your question.

DR. FREY: I am Karin Frey and I wanted to come back to the question of what should be our objectives in training students in multiple methods and I find myself really drawn to the idea that within the expectation of specialization we really want students to have a basic understanding and an appreciation for multiple methods, and I really want to stress the word "appreciation."

This morning when Steve Raudenbush was talking about people who consider themselves philosophical opponents of quantitative methods feel isolated in the current climate, and I don't think that is an exaggeration. I know people who do feel isolated and for reasons I can appreciate but why in Gordon's name are we graduating people who consider themselves philosophical opponents of a particular method? I would like to suggest that to the extent that we have people graduating from our programs who consider themselves philosophical opponents as opposed to people who say, "Oh, that is not the right methodology for that question," I would suggest that we have failed. Anyway, let me say a dirty word that is underlying your question and that is epistemology. No one has dared to utter it yet today but frankly I think that is what is at issue here are some very core fundamental epistemological differences that tend to characterize some of these fault lines, again,those philosophical ideas but also current faculty.

Do you want to comment?

DR. PETERSON: That is the first category on our list.

DR. HEDGES: I would like to say that I don't, it is interesting that I don't see that fault line as keenly in some of the disciplines that have both quantitative and qualitative practitioners as we often do in education.

I know in sociology and I was going to say in my own department, but which one is that?

DR. TOWNE: Which mailbox, Larry?

DR. HEDGES: Yes, wearing my sociology hat I know that we have folks who mostly do qualitative work and folks who mostly do quantitative work and quite a lot of people who do both and my sense is not that there is a sense of epistemological opposition from one camp to another. I think there is a healthy understanding of what can be gained by methods that people aren't primarily practitioners of and I am not sure what accounts for that, for the larger perhaps not perfect but larger degree of tolerance that we often see in the disciplines. I suspect that is true in parts of psychology, too, that there are lots of folks who do very qualitative work and they are not pariahs because of their choice of methods for the problems that they work on.

I don't know why there is more tolerance within the disciplines. Maybe it has something to do with having a more defined structure in which they operate,but that is an interesting question.

MS. MILLER: I think the real question we are looking at is what, oh, I am sorry, I am Erin Miller. I am a doctoral candidate from the University of Virginia and I think that the real question is what is the outcome of graduate training, talking about outcomes and how to define them and from my perspective as far as methods of course it is empowerment. You want to come out with the empowerment that a breadth of resources come from and power to ask the right questions of the right people and feel comfortable going to different disciplines the way they can go to the quantitative experts who have done research in something they are interested in and speak intelligently with them, go to the qualitative experts and speak intelligently with them. That empowerment is really what we want to get to and breadth. You need breadth to do that but depth is also important and I think that is self-chosen, that students choose where they want to go and you need breadth to be able to make an educated choice on where to go.

DR. PETERSON: I remember a doctoral student at Michigan State University who was actually in language and literacy, Sara McCarthy who worked on a research project. We did actually some research together on literacy classrooms. She wasn't actually my doctoral student but she really hated statistics. She had to take a statistics course at Michigan State. So, she was a very qualitative researcher. She got to the University of Texas and got put on a committee of a doctor student in reading and it was a quantitative dissertation, and she called me up in great anxiety. She said, "I have got to somehow get a crash course in statistics so I can understand this dissertation because I have got to be a thoughtful member of this committee," and she said, "Why didn't you tell me this, Penelope?" I said, "Well, you know you took statistics, right?" She said, "I didn't know I was really going to have to understand it so I could be on this committee," but you know you do, right? I mean you get put on these committees or you are asked to be on those committees and they often use methods that you yourself are not trained in.

DR. HEDGES: I would go further. I always describe what I am trying to accomplish in graduate courses in statistics or at least the ones that everybody is subjected to as a departmental requirement and I talk about it in terms of literacy and it very much fits the empowerment mode. You can't be empowered and illiterate in the language that in effect part of your discipline or part of your research area takes in.

DR. GORGIA I am J. Gorgia from the American Psychological Association and I would like to hear a few more specifics about how you would actually set up a program like the program you are going to set up about what that would look like and as important as I think it is to construct programs like this and as philosophically in agreement as I am with actually using mixed methods I lack the confidence that it can be pulled out so easily. I went to one of these handful of good schools, teacher's college and my doctoral preparation actually had a mandatory introductory year-long course on epistemology, a 12-credit course. It was an extensive part of the program and it was team taught but the camps were, current faculty even more than we are now are socialized in a culture that you pick an extreme and you are good at that and that is what you are rewarded for in an academic setting and so that the qualitative person didn't know what a regression analysis or T test was and the quantitative person gave a couple of lectures and that was it, and so how do you actually in that sort of setting where the quantitative person when he said that he was in special education did the qualitative class because he was the only one who knew anything about it and did the quantitative course in the curriculum teaching because he was the only one who knew anything about that and how can you actually construct a program thoughtfully where you do have camps that you can pretty much guess how they vote almost by whether they do quantitative or qualitative or it is apolitically charged and where there is disagreement with each other and where they are not as fully literate as you would like in each other's specialization?

DR. PETERSON: I think you have to create a culture of collegiality where people don't disagree. I actually think in our school I would say that there is very good agreement in both doctoral program that it is important for students to be able to have some familiarity with quantitative methods and qualitative methods and they have to take at least two courses in qualitative and two courses in quantitative and I think that there is respect, and I don't think there is polarization. I may be totally Pollyannaish but Greg is not. I don't think there is. I think you have to have a culture and create that kind of culture and for the faculty who understand less about quantitative that might mean coming to understand more about quantitative methods and vice versa, and I think it helps to have people like Greg who have done studies like he was talking about this morning and Tom Cook, you know, but if you have a few of those people I think it makes a big difference.

I don't know what you would say.

DR. HEDGES: I think I would say the same thing that if we model for students intolerance and maybe I will put it a little more strongly, bigotry then we can't be surprised that our students might behave that way themselves.

DR. KING: I have a comment and a question. I am Karen King from the National Science Foundation and Michigan State University. My comment is that I don't know if you are aware that the Carnegie Foundation for the Advancement of Teaching has a program called the Carnegie initiative on the doctorate that is looking across several disciplines, one of which is education, but they are doing some research. So, there might be some data one day given the speed at which research is done.

My question is around the issue of the apprenticeship because Michigan State is one of the institutions that is part of the DID for education. I have actually been doing a lot of work in this area and the concern of this apprenticeship and particularly in colleges of education where they have a large demand for doing teacher preparation I know currently you are not in one of those types of programs but the demand for graduate students to apprentice in other aspects of the work that they may be called on to do as a future faculty member besides research and often that demand becomes overwhelming because you have to serve the undergraduate population and particularly at a land grant institution like Michigan State versus the demand to move forward a research program and apprentice in that way, and so the demands of apprenticeship start to become divergent in some ways and may not allow for enough opportunity to learn and apprentice in research settings.

DR. HEDGES: I don't know. I think that is directed to me, at least in part. I have a response to that and I think it is that the situation is really a difficult one. I think it takes a lot of time to learn how to be a good researcher and it is not just classroom time. I do believe it is apprenticeship time. I know I needed it and I think most of the students that I have had in the last 20 years have needed it, too.

I think that we have to bite the bullet in education and recognize that it is just as hard to train education researchers as it is to train sociology researchers and psychology researchers or mathematicians and statisticians and I will go further.

I think we have to realize it is that difficult and I think the question is not whether people will get training to do this kind of work and whether people will do this kind of work. The question is whether or not people in schools of education will get the kind of training they need to do it because frankly I think one of the realities of education research with its importance now is that education research is increasingly a target of opportunity by people trained in disciplines and if we are not there to do the kinds of educational research that is going to inform policy and move ahead our knowledge in the area it is not that it won't be done, it is that somebody else will do it, and my sense is that there are some really substantial advantages to multi-method, multi-disciplinary research that is located someplace near practitioners and not in departments of economics or sociology or psychology not because any of those departments, not solely located, not because any of those departments are bad or they have bad scientists. It is just that their perspective doesn't include a lot of things that I think are really valuable about schools of education, namely contact with the profession in a way that disciplines don't have.

So, I think the difficulties you pose are absolutely there and I think they are the challenge that we have to meet and I think the answer is not that because there are those difficulties it is easier to do educational research and it requires less. I think the problem is that it is at least as hard to learn how to do education research at a world class level as it is to do research in the disciplines and unfortunately that means that the amount of research training including apprenticeship that is required is going to have to be there, and if there are other demands that students have to meet we have to find a way of helping them negotiate that. I don't know what the right answer is. Maybe the answer is that there are certain kinds of functions that we typically --that many schools typically use cheap graduate student labor to accomplish that need to be professionalized.

DR. PETERSON: My answer and actually I said this at another meeting of school of education deans and I am not sure that they agreed but my board of visitors loved this answer and advisers who are mainly from the corporate sector and the answer is focus. I don't think we can be everything to everybody and I think many schools of education try to do that.

I think you have got to stop doing some things and I guess we do have a luxury maybe as a private institution to be able to do that more but I think that public institutions could probably also do that more because you end up just adding faculty and taking on the next function, the next function and the next function and we can't do everything well.

So, I think what our school has done is we have tried to figure out a comparative advantage and we are kind of a one-of-a-kind school of education. We seem to do all the social policy and learning sciences and we really have a strong commitment to training researchers in that area. We do do teacher education but we don't see that as our major role, but I think every school could do that, school of education and social policy or whatever. You say, you know, what is our comparative advantage; what do we want to be known for?

DR. MAXWELL: Joe Maxwell from George Mason, again. I want to talk about some of the things that I think I learned from teaching mixed methods courses over a period of about 10 years.

For several years I taught an advanced mixed methods course. Most of the students were doctoral students who had already had at least two quantitative and two qualitative courses. It was a very exciting course to teach. It was actually created by student initiative but it was also frustrating because many of the students just had no idea how to combine methods. They were perfectly competent in qualitative. They were perfectly competent quantitative but they could not put the two together conceptually and see how to make a coherent integration of them.

Then when I came to George Mason I was serendipitously put in charge of the introductory research methods course for doctoral students which was supposed to be a beginning course with no prerequisites but taught at an advanced sophisticated level and what I did with that course was focus on conceptualizing both quantitative and qualitative inquiry. I talk about epistemology. I talk about two different world views, have them read articles both qualitative and quantitative and mixed methods and critique them, design both a quantitative and qualitative study and I think that is more important than having people be able to do a variety of kinds of research because it fits them to collaborate with people from different perspectives, and I would like to get your reaction to that.

DR. HEDGES: I like that approach because I think that the approach to mixed methods that I favor is not everybody learning to be completely competent in every method but learning how to work with other people in teams that have different competencies, doing some of that perhaps yourself but I think that is a much more realistic vision of how to generate capacity to do mixed methods work in our community than the everybody can be everything kind of model.

DR.PETERSON: I agree. I think developing that kind of appreciation for both kinds of, you know multiple kinds of methods is really important. It sounds interesting.

DR. LESGOLD: Alan Lesgold from the University of Pittsburgh. You know educational problems are around the same complexity as medical problems. I think we could learn a few things from looking at the medical side of the house.

First of all the really ground-breaking stuff almost inevitably is done by people who know how to make their own tools. Some subset of the people that we produce are going to have to be good at everything.

Secondly, because it takes a long time to get really good at multi-method approaches and because so much of medicine involves multiple methods what happens? First of all, people spend some time in practice. Secondly, we have virtually universal postdoctoral fellowships. Third, increasingly major medical schools put people outside the tenure stream for a couple of years before they go into the tenure stream so that the tenure system essentially adapts to the fact that they need more time and when you think about those kinds of things what it suggests is that the medical world at least believes that if multiple methods are important to solving key problems that we have to train people to work that way and they have got to start working that way right away rather than having the assumption that they first have to get to the core of their career and then start trying to figure out how to deal with multiple methods. I think we have to learn how to do the same thing.

DR. ROLLIN: My name is Steve Rollin from Florida State University and part of my question was sort of triggered by a comment you made, Larry about your review of postdoctoral applications and I was thinking you know just about every profession I can think of whether it is massage therapy or psychology or medicine or law or accounting, I am a licensed psychologist, I am required to have 40 hours every 2 years of continuing education but in fact that is not the case. When I even look at the numbers of sabbaticals that are taken at my own institution faculty members don't typically take sabbaticals. They seem to be so busy and maybe there is a sense of being indispensable that I can't take myself away from the work that I do.

I wonder how much of what we are faced with is sort of a culture that doesn't encourage us to learn and we become so expert in our own little areas that in fact when it comes to thinking about new ideas and new arenas well it is a nice thing for my students, and so if we began to think about a culture within our own institutions that said, "Look the expectation for you as a faculty member is no different than for you as an attorney or a physician or a teacher, and I would even include teachers, or the expectation is that you get 40 hours of continuing education or something whatever the number might be." We can argue about that number but how do we do that? How do you get faculty members? I agree. I don't think we can do away with the tenure system and I am not sure I am even for that personally but on the other hand I really want to encourage the idea that we really need to be responsible for our own education.

DR. TOWNE: Let me turn this a little bit. This gets to the core idea of thinking about professional development as a continuum for education researchers. You know, how do we think about this both before and after graduate school since obviously training and learning don't begin or end there?

DR.PETERSON: We have three of our 22 faculty on sabbatical this year. Greg Duncan is at the Russell Sage Foundation. Right now he is here today. Faye Cook is at Science Pro(?) University in Paris and Carole Lee is at the Center for Advanced Studies. I agree completely. I think we have to create a culture that you know we are going to continue to learn.

One of the things to think about is once we give our students PhDs they are not done learning and sort of take responsibility for mentoring them, continuing to mentor them and you know we end up switching our PhDs among a bunch of universities, sometimes the same set of universities. They have our assistant professors you know and so to take some responsibility for thinking about them and sort of a life course approach to training educational researchers.

DR. HEDGES: I think it is a problem we have to address. I think it is kind of a scandal the way in which most faculty find out about something one of their colleagues is doing is they ask their students to go ask them and report back and I probably have done that myself. So, it is clear, I think it would be useful. I mean the answer to questions like that always seem to me to involve incentives of various kinds.

I mean in principle we have incentives for productivity and for various other things professors are supposed to do. It would be interesting to think it would be possible to introduce incentives for doing something new and different, learning something new and different. It is possible that that might have good effects.

DR. TOWNE: Jim, how about you give us the last question and then we will wrap up and go to the next panel.

DR. GRIFFIN: Jim Griffin, US Department of Education and I actually wanted to end this session, and it kind of blends into the next one anyway with a lot of good news I think in terms of training in this area an in multiple methods.

The Institute of Education Sciences actually funded the American Psychological Association to fund 13 postdoctoral research fellowships many of whom I hope are here today, if you would actually stand up and be recognized? Come on.

(Applause.)

DR. GRIFFIN: Great and also we funded our first round as you heard of predoctoral and these are predoctoral interdisciplinary research training programs of which we have Northwestern represented, James Spilan but also Greg Duncan. Penelope is here. I think actually James stayed back because he is the only one at Northwestern who is minding the shop while everyone else is here at this meeting. We have Carnegie Mellon represented here. David Klahr is the project director. We have Vanderbilt University, David Cordray, University of Virginia, Bob Piante, Florida State University, Chris Lonergan. So, these are the first five we have funded. We actually have an RFA. It has closed. We have the applications now for a second round of predoctoral training fellowships and now we have also started a new one and again the applications are in for postdoctoral training fellowships.

So, I think there is a real recognition within IES that we have to carry our own water here in terms of training people and really providing meaningful support, and we have purposefully made the predoctoral program interdisciplinary just to get at a lot of the issues both to be able to draw on the disciplinary departments and as you heard at Northwestern and I think they all echoed this to be able to draw on student sources but also the methods that people bring from their home disciplines to at least get an exposure and an appreciation for it and hopefully it is something that they will carry in their toolbox as they go out into the world to actually face the challenges we all do in doing research in education settings. So, good news.

DR. TOWNE: Thank you.

Please join me in thanking the panelists.