DR. TOWNE: Welcome back, everybody. My name is Lisa Towne. I am a program officer here in the Center for Education at the National Academies and I am going to be doing my best at moderating a small panel discussion about the topic of training through this lens of a multiple methods paradigm. So, let me go ahead and just very briefly introduce the two panelists, refer you to the bio sketches in your packet for more information.

On my immediate right is Larry Hedges who is a professor in the Departments of Sociology, Psychology and Public Policy Studies at the University of Chicago a multiple mailbox appointment. He is perhaps best known for his methodological work in combining evidence from multiple empirical studies in a range of fields and since Dr. Hedges is not housed in a school of education he has a different set of experiences and insights about training issues as compared to our second panelists who is Penelope Peterson. She is currently Dean of the School of Education and Social Policy as well as Eleanor R. Baldwin Professor of Education at Northwestern University.

Dr. Peterson has spent substantial amounts of time training and working in four schools of education in addition to Northwestern, also, Stanford, Wisconsin and Michigan State and so has a broad range of experience in schools of education in particular from which to draw on.

Our conversation about training is going to focus primarily on graduate training in university settings in particular. We are not going to have any presentations in this one. I am going to do my best to keep a good conversation going with my two panelists and then open it up to discussion. I know there are many of you from different universities and schools that could lend expertise and insight to this issue as well.

So, I will do my best to keep enough time for that. A couple of quick overview points just to point out that some of the questions and ideas that we are going to be trying to draw on in this session come from a couple of external publications that I wanted to draw your attention to.

One is the paper by Steve Raudenbush. We have tried to take that paper and draw some of the core ideas that have been discussed into this conversation about training.

Another is the NRC publication Advancing Scientific Research in Education that Marty mentioned this morning. This publication did take up the topic of training at a broad level. So, some of those ideas will be interwoven throughout.

I, also, wanted to draw your attention to a recent report that was put together for the National Science Foundation that Felice Levine had a hand in on education and training in the social and behavioral sciences writ large.

All of these things are coming together at a similar time to make this conversation particularly timely.

Speaking of Felice Levine I am going to invoke her words to kick off the first question here to our panelists and in the talk that she recently gave she argued that training for education research is diverse. The question for us is what is their training for diversity, training for methodological diversity. So, this is the what is question. In your experience is there what Raudenbush calls for in his paper which is, and I quote, adopting a self-conscious and explicitly articulated strategy for combining information for multiple methods? Does that core idea pervade training for education researchers in these settings?

Let me start with Penelope?

DR.PETERSON: I was trying to figure out why I was invited to be on this panel and one of the reasons might have been because actually both Larry and I got our PhDs at Stanford as did Pat Forgione, as did Brian Rowan by the way and my training as well as Larry's was heavily quantitative.

I think even though I was in educational psychology I basically had 3 or 4 years of quantitative statistics and in fact when I got hired at the University of Wisconsin, Madison they wanted to put me in the measurement and quantitative methods area in addition to the cognition area. I declined but they said that I knew a lot. So, I want to Wisconsin actually. The department I was in was really heavily quantitative research, experimental psychological research and in the mid-eighties I got together with Elizabeth Fenamen and Tom Carpenter. The department thought it was a little interesting or strange because they were across the alley which might have been across the river and they did stuff in mathematics education and we actually in the mid-eighties did a randomized experiment in which we assigned 40 first grade teachers to an experimental or control group. The control group actually got the treatment a year later and the experimental group we gave research findings and talked to them about the new research that Tom Carpenter and Jim Greenough had done looking at how first graders learn addition and subtraction facts and lo and behold we measured their achievement of the experimental groups at the end of the year, the kids and they did better on not only a problem-solving test but a standardized test which was very interesting. We got a main effect of the treatment but I was very interested in trying to figure out what had happened in the intervention groups because even in the intervention groups there were some teachers that were much more effective.

So, I did a case study. I had never done a case study before but I was trying to understand what was going on and I sort of taught myself and Liz and Tom helped to do this case study and in 1989, actually I published what I think, and this might have been another reason I was asked to be on this on panel although I bet Rena didn't even know this in the Journal of Ed Psych we published what I think was the first set of mixed methods study to be published or we actually quantitative analysis, correlational analysis of the experimental intervention group and case studies with two teachers.

But at that time at Wisconsin we definitely did not have a mixed methods kind of approach to the training. When I went to Michigan State I think at Michigan State this was in the early nineties we began taking very seriously what it would mean to teach both quantitative and qualitative methods and since I have come to Northwestern we have also been taking that very seriously and I can say a little more about that in a few minutes but I would like to hear what Larry's experience has been.

DR. HEDGES: I guess my experience about training for diversity is that the sort of training that I think I got as a graduate student wasn't particularly diverse. I had similar experiences to Penelope in that regard but I would like to sort of raise one point about training that I think for me is very important about training to become researchers and that is that the most important, perhaps not the most important but certainly it is very important training that I think we provide at Chicago for students and I think Stanford provided when I was a graduate student was training on the job, apprenticeship in research projects and I think most of what I have learned about interdisciplinary and multi-method kind of research I have learned in the context of major ongoing research projects, some of which have been mine, some of which have been projects that I was involved in but they were primarily the brain child of somebody else and I think when we talk about training we have to remember that even within a single method an awful lot of important kinds of training occur as apprenticeship, even in statistics. I mean I don't think that the people who take my statistics courses are able to go out and do quantitative research very effectively until they have had some apprenticeship in how to do that and I think research projects and involvement in research projects provides a really crucial aspect of training in doing research and the chance in the best of all possible worlds to be exposed to people who are using different methods and they are using them precisely because the kinds of questions that need to be investigated require methods that are different than perhaps the methods that they have been trained to use.

DR. TOWNE: I am going to hold off on this idea of apprenticeship just for one second and back up for perhaps a preliminary question. It is not clear to me what you would characterize as the goal of multiple methods training. This is something that Susan Bodilly this morning, touched on very briefly. Are we talking about training all people who are going to be the future generation of education researchers to be themselves literate in conducting and implementing a range of multiple methods which suggests quite a bit of breadth or are we talking about developing sort of a minimum competence to judge the worth or value or respect other kinds of methods while developing methodological specialization in one or two? I mean it would help I think to think about what really are we talking about in the training context of what is the goal of the multiple methods framework.

DR. PETERSON: I was telling Lisa earlier that I was struck by this in this session that interestingly we talk about doing research on teaching and learning and coming up with best practices curriculum in K-12 classrooms but we have virtually no research on the training of our own graduate students, let alone the training of methods for our graduate students.

Several years ago our faculty got concerned about how well we were doing in training our graduate students and we didn't have any data of course but we did survey a number of our doctoral students who had recently graduated. We had them actually write an essay or write to us about the things that they found positive in their experience and negative and the things they would like changed and interestingly just about all of them said that they thought the methods training needed to be improved and so we actually had a retreat and we launched a task force to look at method. Greg Duncan was on the task force. He might have even chaired it at one point during its work. No, you didn't? He doesn't want to take credit for that. He built that quantitative perspective and qualitative perspective and they decided that first of all they needed to lay out the goals and the concepts and the level of understanding, familiarity, being able to apply these.

So, I had this on my computer and I decided just to show this to you. We have two programs in our school, human development, social policy, two doctoral programs, and so the committee created this. This is a survey they gave to each faculty member in our school and the faculty members needed to say should all doctoral students have, you know, what level of familiarity with each of these all learning science students or a student who works with me should and then they had to rate it on zero not necessarily have any familiarity, some familiarity and applied experience would be being able to understand it as well as being able to apply it in a research context.

So, what we have up here is just actually the first page and she is going to scroll through slowly. There are actually more than 110 items here which gives you a sense of if everybody says that the students have to have some familiarity let alone being able to apply them and these are mixed methods and they are qualitative and quantitative with the idea that our doctoral students should know and understand and be able to do these. So, it is really continued research and how much you can do this in depth. This is a work in progress I would say. We started this a couple of years ago and I was just telling Larry that it actually led us to hire, one of his students, Spiros Contintopolos who taught HLM last year.

DR. HEDGES: A good decision by the way.

DR. PETERSON: A great decision and we also, he is teaching a stat course this year. We also instigated a new introduction to research class which Alani Mindara is teaching, just finished teaching this fall. Now, we don't know how this is going to work. As I say it is a work in progress but I would say that a first step in really thinking about training for mixed methods is what our committee did and I had nothing to do with it actually. They spontaneously did this which is to lay out what are the goals, you know of our training and what do we want our graduate students to be able to know and be able to do.

DR. HEDGES: I would add in terms of these goals I think that when I just looked briefly over Penelope's list and realized that this is pretty daunting and I think that one of the challenges for doing mixed method research is to think about how that kind of research is best done and in some fields at least there is a tradition of working in large research teams with people who have somewhat different skills and specialties and methodological specialties in particular and think about the context in which mixed methods research may get done and at least one of those contexts, not the only one but at least one of those contexts is the sort of large project that involves people with very different kinds of specialties all of which are sort of pulling together to investigate a line of research, often using different kinds of methods that complement one another. This calls for, this is a different model than you have to be competent in every one of the methods that might be used, competent enough to do it well but the model of research teams that have collectively a mixed method approach is one that requires at least a sort of passing familiarity with the methods that are available and a certain respect for other methods, a recognition that whatever you happen to be a specialist in isn't the only thing that is useful in doing education research and in fact may be in a sense crippled if it is not paired with some other things that involve different kinds of methodological approaches and in that, well, I could go on, but --

DR. TOWNE: It is a good foray into then thinking about what in fact would multiple method, optimal multiple methods training actually look like. You have been talking a little bit about some possible examples of research experiences that graduate students could engage in to get this kind of on-the-ground sort of experience. So, why don't you both talk a little bit more broadly about what this might look like both in terms of research experiences as well as course work?

DR. PETERSON; I just wanted to give one example. We have been involved in the Spencer research training grant program for the last, well, we have been involved and some other institutions have been involved for 10 years. I think we have been involved for about 7 years and there are about 10 or 11 institutions, some of them in this room that are involved

One of the components of those programs that have developed, each school actually developed a different training program and again there is not a whole lot of research on these. However, we have a session of ARA trying to bring together five programs that do we hope have some data, but one of the commonalities that developed across a number of these institutions is that several of them developed pro seminars and the key characteristic of the pro seminar was to have two people paired who taught it coming from different methods or different disciplines and interesting enough this was not always really easy. In 1999, I attended actually a conference at Stanford that was organized by the Stanford research training grant, center research training grant and Rich Shavelson which was actually a great conference and they have a very nice write-up which I wish people would circulate this because it has been in a drawer at Spencer but I got it out of the drawer and one of the things I remember being talked about, in fact, there is a nice quote in here by somebody who is teaching one of these pro seminars. It was mixed methods and this faculty member and I don't remember who it was actually told this story about how the person with whom she taught made sense of the world by, quote, reducing it down to a discrete number of variables while she tended to understand the world holistically. This faculty member described this experience as quote, it is hard at two levels. It is hard intellectual work to find a common language between people who are teaching and it causes a lot of frustration because we feel like we are not talking about the same thing.

Now ,she was very frustrated but somebody in the audience listening to this then commented, "It sounds like you did find a language based on your description. It is not like the two approaches are isomorphic but it does sound like you found a way to talk across these distinctions that are quite clear. It delineates differences and possible points of moving forward."

That was in 1999. One of the programs that did actually do some research is the University of Wisconsin and they have a very nice research study up on their web site which I also brought a copy of but you can go to the Wisconsin web site under DRP which is their doctoral research program and read their study. It is quite hefty and they have a lot of very interesting data.

They surveyed the 48 students who were Spencer fellows and compared it with doctoral students who were not but one of the most negative comments of the students about the program was actually on their PRO seminar which was taught by a qualitative researcher and a quantitative researcher and I don't know the insides and outs of the PRO seminar but they comment on some of them found it a valuable opportunity to learn about different approaches but others said that it didn't cover both quantitative and qualitative perspectives enough, not extensively enough.

They also said that there was a lot of actually often disagreement in the seminar which I guess they found quite disquieting. So, I mean I think some of this minimal data that we have which I say is very minimal with people who have tried to d this mixed methods kind of teaching in a PRO seminar is that it is difficult and that is about as much as I can say about it right now.

DR. HEDGES: I guess I would add that I am going to go back to the issue of training in some kind of specific research context because I think methods training without any context has a way of being awfully sterile and I just think it is very hard to do that. Now what a lot of us do who do methods training to try to bring content in, the content we bring in in a classroom context is often quite artificial and so not a problem to anybody who cares about the class. If you are lucky one or two people care about it and most of the rest really could care less. So, it is barely an improvement over no content at all.

I think that one of the places in which people do get engaged with the problem that they are investigating is when they feel like they are part of a research team and when they actually can begin to be researchers, to do the work of researchers, and I think it is in that context that you can make not only methods of one type but you can make methods of any type I think meaningful and interesting to students when they begin to see how to use them to answer questions that they care about and they begin to see that methods that perhaps they didn't think were very valuable actually have important virtues. They may choose not to primarily be that kind of researcher in the future but they begin to see the virtues of different research approaches and in a sense I suspect that that is a more satisfactory way of inculcating sort of values about multiple methods research in students and actually giving them the experience of what that kind of work is like, and I don't think that can easily be done just in a classroom context or even for a seminar kind of context.

DR.TOWNE: So, in talking about the research experience and working on research projects as a way of really sort of establishing this ethos of valuing and understanding the role of multiple methods that helps in a somewhat serendipitous way to get at my question to Penelope which is how do you choose among the 110 different methods that you had listed on that five-page document. Now, in a research setting it is set by the question. That was an argument that Raudenbush made and has made in any number of other kinds of contexts but in terms of course work, I mean, Penelope, could you speculate? Do you think that sort of having to make the decision about which methods to feature in a class is part of the issue here and how do you go about choosing that?

DR. PETERSON: Larry teaches courses like this. I think he should answer and I was going to ask him if we should give up courses completely and maybe entirely go to an apprenticeship model, research apprenticeship.

DR.TOWNE: An even better question.

DR. PETERSON: It would put you out of a job, wouldn't it?

DR. HEDGES: No, I think you actually have to have people who know how to do research running the apprenticeships. I am not sure I would go so far as to get rid of course work entirely but one of the things I think I would do is try to assure graduate students to write of a research apprenticeship as part of their educational program, and maybe write is too strong but nonetheless I really do see it as desirable and frankly I see virtue in course work. For certain kinds of technical things I think that statistics courses and experimental design courses and measurement courses serve useful functions. There is a certain amount of content that you can sort of bludgeon students into learning, but I also don't have an illusions that somebody who has taken one of my courses for example, is useful for research until they have learned a little bit more and this is sort of a classic. I had a friend who was an engineer and was disappointed to be told after getting his master's degree from a pretty good university by the company that hired him that he wouldn't be good for anything for 6 months until he learned how to be an engineer. He was under the illusion that he learned something in his 6 years of higher education in engineering and they were wise enough to know that he knew enough to be dangerous and needed to actually learn how to be a practicing engineer.

It is not that course work wasn't valuable. Undoubtedly he couldn't have learned sort of practical skills if you like of being an engineer without it but I think the situation is something close to true in graduate training in the social sciences and education.

You raised the question could we do away with course work altogether. It is a radical suggestion, but I could imagine graduate programs that were structured around projects.

DR.PETERSON: Actually I don't think our faculty would want to do it, would we, Greg?

DR. HEDGES: I think it would be harder to do. It would be much harder to do.

DR. TOWNE: Let us push on this idea of apprenticeships. We don't have to have necessarily agree on whether it should be the only part of graduate training but let us assume for the moment that we all agree that it is at least important as part of it.

In Dr. Raudenbush's paper that we talked about this morning, he touched on this idea of the multiple methods framework really assuming a fairly cohesive community of scholars who are themselves fluent in multiple methods as they apply to different educational research investigations.

Now, if we connect that idea with this notion of an apprenticeship being very important let me ask another provocative question. Do you think that the current faculty, these are not people who are being trained; these are people who doing the training and who are populating the education research literature as we speak, are they themselves do you think as a group willing and able and those are two different questions to be the mentors and the models for the next generations of researchers to actually work in a multiple methods framework?

DR. PETERSON: I think that is a big challenge we face. Actually Luis Gomez and I were talking about this the other day. He came into my office and you know there are very few people actually in our school that are fluent in both qualitative and quantitative methods if you want to call them that and so then if you have a graduate student who is working with just one faculty member who only does qualitative methods then you know and they go to maybe Spiros for help in quantitative methods but they still really have an adviser who just really knows and understands one and so then I mean it would be interesting to think about actually having courses and we have had some of these courses like short courses in the summer where faculty could learn HLM but you know then people would have to be interested and willing to do that, sort of retrain themselves.

DR. HEDGES: I guess I tend to think about that question and have different answers depending on what your conception of multiple methods research is. If your conception is working in a team in which you don't necessarily have full command of all the methods that may be used in that team but a certain amount of respect and ability to work with people using different methods I think the answer might be a little different than if the question was if your conception of multiple methods research is training students to do multiple methods at the same level of competence.

To be honest I am actually concerned about the ability of the, not the best schools of education. I think in a sense Penelope's school is one of those places in which this is less of a problem but as I look nationally at the capacity to do cutting edge quantitative research and that is something I know more about than qualitative research I think that the capacity is very limited.

If you look at something like NAEP just to choose an aspect of quantitative research like assessment the methods that are used in the national assessment are not particularly new. They have been using the same methods for about 20 years now, but if you ask how many schools of education or for that matter how many economics departments can train their students to handle, just to do secondary analyses of NAEP and with all of its complexities the answer is a very, very small number. You can probably count them on one hand and that includes all of the good places, the very best schools of education, and that is not and I just want to emphasize it, that is not a brand new technology now. It is 20 years old. It is now widely used. Every international survey uses the same kinds of methods. It is an example but probably not the only example of places where the ability to train people even in one method, even in one domain of methods shall we say is reasonably limited and I have another source of data that I guess I will cite. It is something slightly better than anecdote but not much.

For about 8 years I sat on the NAE Spencer Postdoc Selection Committee and have been part of a group that read a couple of hundred of applications each year and one of the things that continued to be disappointing to I think every person on the committee was the quality of the applications that came from people who had graduated from or were faculty in schools of education. The thing about it was that everybody on the committee wanted, you know was prejudiced towards those kinds of candidates and our experience in evaluating their proposals for research was that disappointingly we found ourselves less able to award postdoctoral fellowships and be fair about it to people whose training was in schools of education than to people whose training was in disciplinary-based departments.

So, that anecdote suggests to me that there is something amiss in the methods training because in many cases the problems were problems of method, a misfit between the question and the problem, problems of situating the concrete research problem in a larger research literature. So, you ask yourself the question well, suppose they did what they said and they found what they said they hoped to find, so what? That experience sort of led me to feel that there is perhaps a lack of capacity in training for qualitative research as well because many of the proposals we got were for qualitative work that we thought didn't pass muster as qualitative. It wasn't pro-quantitative bias. It was really more of a --

DR. TOWNE: And they were people other than Larry Hedges evaluating these?

DR. HEDGES: I was kind of the minority person on there. Every once in a while they would give me something like a philosophy project to read. There were probably more qualitative folks than quantitative.

DR. TOWNE: Let us talk about the question of incentives. This is a theme that pervaded the conversation this morning and that has to do with some of the long-term entrenched incentive systems for academics, particularly with respect to tenure and promotion for example that at least the hypothesis this morning was whether that seemed to cut right up against the kind of collaborative project-based work that Larry and others are talking about here. What about that? Is there an opportunity here given the fact that some universities are really waving the banner for interdisciplinary-based research to really fundamentally rethink notions of tenure and promotion or is that wildly naive?

DR. PETERSON: I gave my 2 cents this morning. Larry, why don't you go for it?

DR. HEDGES: I don't see the tenure system going away and I don't see it changing very radically, and again this comes back to the question of how do you participate in multi-method research. I think the idea that a brand new assistant professor who may have pretty good training in one methodological specialty should in their first 6 years learn two or three other methodological specialties to a high degree of perfection and learn how to be a professor and publish the requisite amount of stuff to get tenure is a pretty daunting kind of requirement to impose on anybody.

On the other hand there may be ways in which people can participate in work that has interdisciplinary aspects and has multi-method aspects in which their primary contribution is in one part of the project and that may be a more feasible thing for younger scholars to aspire to rather than trying to do the whole ball of wax.

My comments on the degree to which I expect current social organization of academia to persist are just based on the fact that academia is a very, very conservative organization. You can't get people to agree on when to have a faculty meeting. How can you get them to agree to abolish tenure?

DR. TOWNE: I am absolutely determined to end this on a positive note. So, let me ask Penelope to just describe a grant that she and her colleagues recently received from IES that is going to be in the School of Education and Social Policy specifically around a multi-methods framework and then I will open it up for questions.

DR. PETERSON: I will just say briefly that Greg Duncan and Jim Spilant are the co-PIs on this grant and I think that there are, I counted them, there are core course of methods courses that are required. There are 10 methods courses that are required. I was looking at the required list in addition to required courses in learning and cognition and policy.

It is a joint training program with the departments of psychology, sociology, economics and statistics and we will have doctoral students from each of those departments in a cohort. There will be four students a year and the doctoral students will get, from the disciplines will get a PhD in their discipline but then they will get a certificate in educational sciences and it is very exciting and our colleagues are very excited. We are going to hire a joint assistant professor with psychology and we have actually just had two psych colleagues that have asked to transfer part of their budgeted appointment into our school which is unique in my experience at three different universities.

So, we have great relationships. Tom Cook is involved. I think he had to leave to go back to Paris, but he is the point person from sociology, David Utah from psychology, Chris Toiber from economics and anyway it is very exciting and that is an upbeat note. Doctoral students will be coming out in 3 or 4 years. You can hire them.

DR. TOWNE: Even better.