DR. GREENO: I am told that our lunches won't actually be brought in here. I think there was a judgment that we would be a little bit too condensed for that to work. The lunches will be next door. In fact they may be there already.

We are due back at the other room on 50 minutes, 55, and so what I suggest is that we do maybe 15 minutes of the usual kind of back and forth and then we can kind of self-select on that about people who want to quickly get some lunch, bring it back in and continue the conversation. I will try hard to prevail on Greg and George to be in the group that comes back.

So, let us do 15 minutes or so of the usual.

Yes?

PARTICIPANT: Two questions. First is on this issue of the trade-off between random and purpose examples did you stratify your random sample at all?

DR. DUNCAN: We stratified by experimental versus control.

PARTICIPANT: Not in terms of characteristics of the families?

DR. DUNCAN: We didn't. In other extended studies that I have been involved we have done that stratification, but still as you know stratifying is distinct from random selection.

PARTICIPANT: Right, but you can generally increase its power and with a small sample size like that I would think that it would -- why did you decide not to do it?

DR. DUNCAN: We weren't sure about the dimensions of how to stratify it. In other examples we were and we actually over sampled in our qualitative case selection in order to get more of a certain group but we didn't assign a zero selection probability to any of the groups. So, all the elements of survey sampling with stratification and differential sampling proportions you can build into it, but the default I think should be simple random sample.

PARTICIPANT: The second question was one of the things that hasn't been explicitly mentioned as a use of qualitative methods is to test causal hypotheses suggested by quantitative data. Now, that may be what you meant by increasing the opportunity of the quantitative results but that is fuzzier language.

DR. DUNCAN: Right. We could have a very long discussion --

(Laughter.)

DR. DUNCAN: -- about testing causal hypotheses with qualitative data. I think the best way to think about it is with convergence. Is there a comfortable convergence in the causal story that is consistent with the qualitative data and is suggested by the quantitative data? I wouldn't want to rely on qualitative being exclusively to establish causal linkage just like I wouldn't want to rely on quantitative data exclusively to tell me about the process by which impacts are showing up. So, let me leave it at that.

DR. LADD: Helen Ladd from Duke University. I am basically a quantitative researcher, but I have been grappling with this question that George asked. Was the qualitative part of this essential or useful? And where I come out is it was essential but then I go back to Tom Cook's question this morning. Did it have to be the Cadillac ethnographic version or could it have been something cheaper and let me just tell you where I am coming from on this. Though I am a quantitative researcher I have now written two books with my husband who is a journalist, and he is not a sophisticated qualitative researcher, but he knows how to go into schools and talk to people and so I went down the list of the people who think qualitative data can help understand the quantitative data. He has helped me and we have done two books together, one on New Zealand and one on South Africa. He goes into schools and he can help understand the qualitative data.

The second point, qualitative data can be used to generate hypotheses and certainly his going into schools and talking to leaders did that when we worked back and forth.

Third, useful qualitative data can provide useful exemplars or anecdotes. That is certainly the case and our books have anecdotes as well as the quantitative data.

So, then the question is how do you decide whether you need a journalist, a sophisticated journalist who can triangulate and do other things carefully or whether you need the more Cadillac version of the ethnographic research and I think we need to spend some time talking about this because it didn't come out in discussion.

What intrigued me about what you said, Greg was that the ethnographic researchers also had some serious questions that they were interested in and a theory that they started with and that is the difference between perhaps the journalists going out and reporting and the researcher and I think we need to pay attention to both of these streams and it has come up over and over again but didn't come up carefully in Greg's discussion.

DR. DUNCAN: A couple of things. I think in the context of an experimental evaluation if you are very confident that you know the theory of change that the intervention is addressing to the extent that you are very comfortable thinking that you know the process by which impacts occur I could imagine I guess, as Tom Cook could, measuring the implementation process and the process effect process quantitatively and then looking to see whether there were experimental impacts on all the intermediate steps that the theory suggested were important. I think we are rarely at that level of confidence in our theories of process.

I am a recovering sole quantitative researcher. The first 23 years of my life I spent at the Survey Research Center doing only quantitative work and I am a total convert actually. Projects that I take on now I always try to have mixed method components because invariably there are questions about process.

There are surprises that come up that if you just have your survey data already collected you are powerless to address.

If you have the luxury of a follow on survey you have a chance to correct past errors but even then the ideas for what is really going on come I think largely from the qualitative observation.

DR. LADD: The question is does it have to be the Cadillac version of it. That is my question. I am agreeing with you really that none of us should be doing only quantitative research. We need to get out into the schools talking to people or whatever.

DR. DUNCAN: That is hard. What could have been learned from 10 families versus 44 families and 44 was a substantial number.

PARTICIPANT: Or 100.

DR. DUNCAN: Or 100.

DR. LADD: But the Cadillac version of it then generates other benefits related to the theory and the development that qualitative researchers are interested in.

So I am making a case for the Cadillac in some cases but it has to have a reason for being because there are these related questions and you have got to have the right design for the qualitative part as well and then my sense is that the two could interact together and you get more than the sum of the parts.

DR. GREENO: Let us go to another question.

DR. BRADBURN: I will make a comment first and then ask the question. The comment was on the importance of having the investigators trained in both methods or in all methods if possible.

I was trained both as a clinician and as an experimentalist and what being trained as an experimentalist does is make you very much aware of implementation problems, particularly in the laboratory experiment. So, if you’re really trained that way and you go out and do field experiments, you think it is not a real experiment compared to the lab experiment because there are so many things that go on in the implementation of the experiment that you would question its internal validity. So, the training of that sort really sensitizes people how to do research.

The question I wanted to ask you was with this kind of intensive, not as much as your comments but fairly intensive observation, that is a sort of treatment in itself. Did you look at or was there any effect on the people who were in the qualitative since they are a subset of the total? Did it have an effect or were you able to look at that?

DR. DUNCAN: We started 2 years in and most of the kind of economic impacts happened very, very quickly. So, we couldn't have affected that. We have put in a dummy variable for an ethnographic sample, right, and it doesn't show up as being important but you could imagine that it might.

DR. LEWIS: I wanted to ask you to reflect on the order of things. I know there were practical reasons for the way you proceeded and you have talked about how the qualitative data has influenced the subsequent quantitative analysis and even survey design. Could you talk about the reverse? Did your analysis of the baseline contribute to the design of the qualitative? Because this template was set up ahead of time rather than evolved out of the analysis of the qualitative data.

DR. DUNCAN: Right. We wished we had launched the qualitative study much earlier. So, that wasn't at all a planned decision to start 2 years in.

There was an implementation study but it was more done in the offices of New Hope not from the family point of view. There is a lot about how the families view the program that you can get retrospectively and we asked a lot of questions about that and that was very intriguing but I think there was nothing like, where in another study we are trying to do this, interviewing families before they actually get involved in the program and then every 6 months so, you can actually see in the early stages of implementation how it is affecting them, and I think I wish we had had the timing to do that.

DR. LEWIS: But did you use the baseline survey to inform the qualitative design?

DR. DUNCAN: To some extent although it was more of a diversion than a benefit probably. It was this issue of whom to sample and we thought just based on baseline and some very preliminary kinds of anecdotal results that we knew who we wanted to concentrate on and what we thought was actually happening but that turned out not to be the case.

If we had had a full-fledged qualitative study in place as implementation unfolded then I suspect there would be much more of an influence on the quantitative data that we would have been collecting in 2 years.

DR. LEWIS: I didn't introduce myself, sorry. I am Sue Grant Lewis from the Harvard Graduate School of Education.

PARTICIPANT: I wanted to go back to having the same people collect both types of data and I wonder if you could comment on the training because I am perceiving that as a really wonderful learning experience for the graduate students but I am wondering about the quality of what they collected and my bias is that people are stronger one way or the other.

DR. DUNCAN: We pretend that they don't have a choice when they come into the program. Ours is a very small PhD program and it is a PhD program only and I was somewhat selective in the students I recruited on the project, but we offer formal training in both methods and we very much encourage the students to take econometrics classes and really to be quite, in a development social policy school or program where I am, we really want them to be able to understand what economists are saying about policy issues which takes a fairly sophisticated level of training.

So, we had quality control in place with a kind of ongoing series of interviews that enabled Tom Weisner to read notes and feedback and some did field work better than others and some did quantitative work better than others but you know with the random assignment kind of design the quantitative work is unusually straightforward, so I think that helped things as well. By and large there were not substantial problems along those lines. They were very enthusiastic and understood that they had to do both and then once they started to see the value of doing both it was kind of a reinforcing process.

DR. GREENO: We will do one more and then we will get our lunches.

PARTICIPANT: Greg, I was wondering if you could give us sort of access to the conversations you had with the funder as you proposed the Cadillac model. It seems to me the question that they had going in was tell us whether or not one works or the other, but you chose to add some wonderful depth and color to your study. So I just kind of was curious when you talked to the funders about this, how did you sell them?

DR. DUNCAN: Right. That is a good question, and we had two very different experiences with MacArthur where the whole child and family supplement was hatched. It was conceived as a mixed method study and the whole network that Jackie Eccles set up was a mixed method network.

PARTICIPANT: They were already converted then?

DR. DUNCAN: Right. So, that was easy. We have funded New Hope for two rounds from NICHD. I have also tried to get some qualitative supplement to this fragile family study that is a very big study. The review panel funded New Hope and its qualitative supplement only because of the random assignment feature. In the reviews there was no interest whatsoever in the qualitative part. In this fragile families qualitative study which we went through three rounds and I have never worked harder on a proposal. The survey was already out there and it was a way to understand what is going on. It went absolutely nowhere with the review panel.

PARTICIPANT: Just frustrating?

DR. DUNCAN: It was very frustrating. The nice thing about a random assignment kind of component is that you can sneak the good qualitative stuff --

(Laughter.)

DR. DUNCAN: -- past the review panel.

DR. GREENO: The deal is lunches are next door. We will usher Greg and George to the front of the line so that they can come right back and then those of you who want to come back please do.

(Brief recess.)

DR. GREENO: Do people want to have some more back and forth with Greg and George since you came back or --

DR. BRADBURN: I would like to ask Greg a question. How did you get them to implement this in a controlled experiment because it sounded like the origins of this program came from people who would not ordinarily think about doing this.

DR. DUNCAN: Right. Funders required it and an advisory committee strongly urged them to adopt random assignment method as the only way in which the success of this program would be taken seriously at higher levels of policy. It doesn't guarantee it but it certainly strengthens it. But the idea of denying benefits to large numbers of families coming in the door interested in them was anathema to them but they were convinced by the other arguments and by the funding that that is what it took.

DR. BRADBURN: Was the implication of MDRC being involved that they were actually the operational partner?

DR. DUNCAN: No, MDRC was the evaluator and they were chosen in a competitive process. So, New Hope, Incorporated did all of the design work, implementation work. MDRC was just coming in and observing implementation and then doing the impact assessment.

DR. BRADBURN: The reason I asked it is relative to the question that came up this morning about careers. I mean there are so many academics involved in this, how did it tie back to the career development issue.

DR. DUNCAN: Right, I should have made clear the evaluation of New Hope itself involved none of us from the beginning. It was the discussions in this MacArthur network on middle childhood that brought us into New Hope.

Bob Granger was on the network and was also a big principal in MDRC for a number of years and so he laid out for us several different experiments that seemed promising for having a child and family study be piggybacked onto them and so that is how we chose New Hope and began 2 years into it. That was the nature of the timing of the funding.

DR. BRADBURN: Greg, this is really important relative to the training because almost all of the evaluation work in these programs is now done by firms of various sorts whereas it used to be done at universities when I was growing up and so you got a lot of training then by graduate students but now since it is all done by firms there is a separation between what you may be training people for and giving them actual experience except in rare circumstances.

DR. DUNCAN: Right, in the fragile families qualitative study which NSF did fund, I should have pointed out, that NICHD did, the Sociology Division of NSF was quite supportive although they don't get nearly the budget that NICHD does.

So, we were able to do it, not quite on the scale that we had hoped, but the students who did the ethnographic work were actually also trained by Mathematica to do the survey work. So, you recruit from mothers in the hospital shortly after they have had their children. That is the design.

So, they did the Mathematica survey and then set up a conversation that facilitated establishing of the links that enabled them to do the follow on very intensive qualitative interviews that took place afterwards.

So, they did get both kinds of training, but I agree it is a big problem but there is nothing like being part of the process both on the qualitative side and the quantitative side.

Usually we think of quantitative training as grubbing around with coding which is very important, too, but my economics colleagues who never even have a conversation with the people that they are writing articles about do this. I wonder where they get their ideas from.

DR. VALENTINE: Greg, I was interested in your response to the question if you are a quantitative researcher thinking about qualitative methods, do you need to go for the Cadillac or is the Yugo okay and I think I just want to sort of throw out something to the group and see what folks think — that is as a quantitative researcher I often think of qualitative research as being a wisdom behind the quantitative stuff that I measure. It tells me what to measure, what to look for and how to cast a spotlight on interesting findings as you pointed out.

There is a subjective value too — this notion of do I need to go on with both feet or just a toe but setting that aside, the question of how far do you go down the qualitative path might depend on your assessment of the state of the wisdom. If you think that there is a lot of wisdom in the area then you might be able to develop a quick and dirty qualitative way of going in there. If you think that there is relatively little wisdom then you might want to jump in with both feet and Joseph is shaking his head but I would like to hear what other folks have to say about this.

DR. DUNCAN: I would like to hear what other people have to say about it because I have partial and ill-formed thoughts at this point.

DR. GREENO: I have been asked to request that people say who they are.

DR. VALENTINE: I am Jeff Valentine from Duke University.

DR.LEVINE: Felice Levine from AERA. While in the opening or second session where Tom made reference to Cadillac and maybe one should say an Audi or a Camry or a Toyota, it has a sort of a face attractiveness to it, but I think it actually is a contrast that can perpetuate what just happened and that is you talked about Cadillac and quick and dirty and I think that that really distorts how it is all methods should be used. The method or the design has to be appropriate to the problem and it may be, and we were talking in the break, that if you have a better worked out theory it may be that you need fewer strategic observations. That also may be true in terms of how many settings you need when you are doing an experimental manipulation or whether you need to stratify or not or whether you are in a position to stratify. What that ultimately does it sort of perpetuates I think some of the sort of stereotypic distinctions between quantitative and qualitative. That in itself is a dichotomy to be avoided.

When I was at NSF I would only say that we were open to all methods as long as they are appropriate to the questions motivating the research, well designed and rigorously implemented and the combination was often a range of strategies and even in those years which was in the eighties we encouraged multiple methods if appropriate to the problem.

Education research in many respects, because it is inherently interdisciplinary, is both more susceptible to criticism but also more open to some of these strategies I think of change and innovation. I think part of what we need to work on is also getting the language right so that it doesn't look like we have some of the stratification of methodology that has always been complex in social and behavioral sciences.

PARTICIPANT: There is a lot of crappy quantitative work out there.

DR. LEVINE; That is right, exactly.

PARTICIPANT: I think George's distinction was very helpful to think about the difference between design and methods.

DR. LEVINE: Right, absolutely.

PARTICIPANT: Greg had design.

PARTICIPANT: There are data collection methods and analysis methods.

PARTICIPANT: Exactly. I think it is a very helpful way of thinking about that.

DR. GREENO: Let me raise one aspect of it. We talk about the product of the qualitative work in quite a few different ways.

The wisdom behind something is one way but somebody else has talked about the explanatory function type of thing and it reminds me of some things that happened a long time ago in other sciences. They figured out the relation for the gas laws between temperature and pressure and volume and you could have run a randomized trial and discovered that if you increased the temperature the pressure would go up but when they really got somewhere was when they figured out there were all these molecules in there bumping into each other and they went faster if temperature was increased from heat sources.

So, getting inside the system is worth more than just making us feel a little better about it. We really get at mechanisms. I have colleagues who study the relationship between poverty and health and I think I understand what is going on there a little bit. They talk about the importance of protected factors that are a better explanatory proposal for why it is that people with more money get sick less and it sounded to me as though some of the things that you were finding in the references to gangs might fit into that as a general kind of explanatory move that you were able to propose for us because you did that work.

So, I like to think that the main thing we need to try to work on as researchers is getting clearer and more valid explanations of things which seem to come much more from the ethnographic work that you did than they come from the findings. It seems to me NICHD is the equivalent of exactly that.

DR. DUNCAN: At least as skewed. Let us put it that way.

DR. CONOLEY: I am Jane Close Conoley and I am Dean of the College of Education and Human Development at Texas A&M University and perhaps like some others in this room I am trained as a behavioral psychologist, the lowest of low in terms of researchers, but if I would be reading the minds of my qualitative researchers right now, their view it seems would be that they talk about the epistemologies you know and that they have a world view and so the idea of mixing methods is incongruous because they see the world one way with its cause and effects and they see the other world in another way. Now, I may not be presenting it with much elegance but I am wondering if you run into that at another level. There is a certain pragmatism of mixed methods that I mean I am very attracted to but I am wondering as you speak with colleagues who are certainly better informed than I whether or not you come up against those almost philosophical epistemological kind of questions.

DR. DUNCAN: Right. I have very short conversations with such people. I am just too focused on the question and the methods that enable me to get at the questions. I just don't think it is useful to frame these issues in two camps where there is no possible meeting of the minds.

DR. BOHRNSTEDT: When I was in academia I actually ran against that and as a quantitative researcher I felt that in many ways quantitative researchers were more open to multiple methods than some of more pure qualitative researcher friends were even though I think the work they did was very useful.

DR. DUNCAN: Let me tell a story. It was set up almost 50/50 qualitative and quantitative and Barrie Thorne is a wonderful sociologist who was a member of the network and spent the better part of a day talking about how she did qualitative methods and it was a revelation to me because you know she talked about a sampling process and a hypothesis-generating process, a hypothesis-testing process. It is the next kid that you interview or that you not interview but get to know you could try out the tentative hypothesis and it is when the next kid that you approach has nothing new for you to discover that I think you have nailed it down. She was saying how different that all is from quantitative work where everyone starts with a theory, starts with one model, tests the model, and that is it, and I started laughing because she actually believed the way quantitative people wrote up their articles was the way they did the research.

(Laughter.)

DR. DUNCAN: And we do just the same kind of grubbing around in the data and searching and it was just remarkably parallel but we just write it up in a very different way and we are not honest enough to admit that what we do.

PARTICIPANT: I had a question that was remarkably similar to the woman who just spoke because there has been a lot of talk today from the quantitative camp about the benefits and the merits and the usefulness of adding in qualitative data but that presupposes that the researchers and the people involved already know how to do solid quantitative work.

Being one of those people, I am interested from the panelists or anybody else how you work with individuals who may be experts in qualitative work but have no experience in quantitative and how to move them in this direction and what a challenge it might be for purely qualitative researchers in learning to adopt and value quantitative.

DR. DUNCAN: It takes a lot of time and it requires that both sides be willing to spend that time and listen very carefully to what the other is saying.

It is a very frustrating process at first because the jargon is different. The way of thinking about things is different and there is no substitute I think for taking the time to talk things through, but the context in New Hope was the intervention was already given. It was designed by the people who wanted to run it. So, there wasn't a discovery. I mean the discovery process took place over the course of the 20 years before when they got to that point but certainly some research questions are probably best served by a very open-ended kind of qualitative approach to help define what the question is, what kind of measures there ought to be and only then does this quantitative research on that topic begin to make sense.

DR. SCHAFFT: I am Kai Schafft for Penn State and I think getting back to your question the longer term answer really is in training of graduate students.

When I was listening to you talk about the project you are working with and making sure that they were equally immersed in the qualitative and the quantitative aspects of the project it just struck me of what a rich training experience that must be to launch someone's professional career to be a member of this peer group of researchers who are involved in this very rich project working with all different kinds of data and looking at how these data speak to each other.

So, I think that is what it really comes down to ultimately is how graduate students are trained and how they are mentored really to think about that research.

PARTICIPANT: Listening to what you say and what other people say I think I need to go out and buy some gifts for my professors. I am a doctoral candidate. They need some better Christmas gifts because honestly from my perspective at the University of Virginia, Educational Psychology Program, I didn't realize that this was a problem. We were speaking about it briefly before. We are required to take four quantitative courses and three qualitative courses as part of the grad program. That is a given that if you are going to be a researcher you need to have dual expertise to research fully important aspects and that that is essential to being a competent researcher that you are able to use both methods and so this has been sort of an eye-opening experience for me listening to this conversation that this is an issue.

PARTICIPANT: Are graduate students doing mixed methods work or do they tend to sort of shake out one way or the other?

PARTICIPANT: No, we do both.

PARTICIPANT: Because there is a difference between being conversant in different approaches and actually --

PARTICIPANT: Not everybody does both but the professors do encourage it. I mean in the projects that we work on, such as the evaluation and reimbursement evaluation for the State of Virginia, we did both. A lot of the other projects we worked on have both components to them and we do both evaluation and developing.

PARTICIPANT: I started in with a psychology background and not knowing a lot about qualitative research and so I must say that most people enter the program with a predilection but we are strongly encouraged and as we have said you have no choice. You must be able to do both. That is something that if you would like to get your PhD you must be able to do both and show expertise, I think.

DR. HENDRICKSON: Bob Hendrickson, Associate Dean for Research the College of Education at Penn State. You know as I was reflecting on this it is really a tragedy what has happened in social sciences and particularly in education at some institutions around the country where whether it is because of the culture wars or whether it is because of the arguments of which research is better, qualitative or quantitative, we have allowed doctoral programs to digress to a point where we allow students to go through those programs and they pick which research methodology they are going to use. What we have tried to do at Penn State is to make sure that we keep with the integrity of both the education doctorate and the PhD and that we school students in both research methodologies. Unfortunately that is not happening at some institutions and I think where it is not happening I would like to see the faculty in the College of Education get back to that kind of an approach.

PARTICIPANT: I was going to say that I think we have here an interesting example of longitudinal and cohort effects when talking about the 100 cystic fibrosis research centers and the variability in the academy. Clearly there are some, I hate to say it but as I am reflecting on it some intergenerational differences in our not recognizing that we, as more senior people, are really pushing for this in the academy right now. So, it is a lot that we experience but, for example Greg's model or your model is at Penn State and it is a lot of places. It is missing, also, from a lot of places and you heard in Norman's illustration how important experimental training was in informing and clinical training was in informing his life.

I, too, would say the same thing that I always think experimentally even though I rarely continue to do experimental work though certain disciplines still seem to be more not only schools of education, more primarily associated with different methods. So, in the quantitative area if you want to put it in those terms in a psych program you are less likely to get out of the laboratory and so the non-experimental designs need to be incorporated into those fields.

In sociology there tends to be less exposure to experimental methods even though there is an experimental tradition and so I think that is really a challenge in the social and behavioral sciences and I think I come from a long line of Pollyannas, so I think one we are positively engaging in, and I think that is true in graduate schools of education. I tend to be optimistic because I hear much more willingness to talk and think about how to develop a professional training in education research versus the other kinds of missions that graduate schools of education also apply. As someone very familiar with legal education you don't hear that dialogue in terms of what it means to have a trajectory in legal scholarship versus training in the practice of law. You actually hear that very much bemoaned in medical schools.

When I am in my NIH environment on human research protection there is a lot of reflection on the fact that medical schools really don't have as cohesive and coherent training in becoming scientists in medicine as opposed to becoming a solid practitioner.

So, I think in graduate schools of education there is at least hope for some reflection, sometimes too much self-denigration about our capacity to do this well.

DR. LA RUSSO: Maria LaRusso. I am a postdoc at the University of Pennsylvania and I wanted to just push the graduate training issue a little. I am a recent graduate of human development and psychology at Harvard Graduate School of Education where we do have very strong training in both quantitative and qualitative methods, one of the main reasons I chose the program but when I got to the end of my course work and I was designing my thesis study I was wondering where is the course that teaches me how to combine methods.

(Laughter.)

DR. LA RUSSO; I even tried to do some advocacy to try to get one, the problem being that the methodologists practice only quantitative or qualitative. So, there were professors who were doing projects that brought the methods together but there wasn't a methodologist to teach about design that brings together methods and so going to a program like that where it is great because you don't have to choose, a lot of the students end up choosing and doing one or the other. So I know that at some universities they do have some methodologists who are teaching courses but we talked about developing a rubric and I think it needs to go much further beyond the rubric but training in the methodological science about the best ways to bring methods together effectively.

DR. GREENO: It occurs to me that we might want to pause a little bit before we go headlong believing that every program ought to train every student to do everything. The luminaries from sociolinguistics and other places were mentioned earlier. I am not sure that the field would be better off if Irving Goflin had known how to run experiments in --

(Laughter.)

DR. GREENO: Or would be better off if Sam Messick had been required to learn ethnography? Certainly the field needs this diversity and to have it we will certainly have to have some people who are conversant in it themselves but it is not totally clear to me that we want it to be a uniform requirement that everybody has to --

DR. HENDRICKSON: My point is that how can somebody with a PhD who is supposed to have an understanding of research and be able to critique research understand a quantitative study if they have only been schooled in qualitative research? I am not saying that they should be able to go out and do both. What I am saying is they should have knowledge and understanding to be able to at least critique whether the design of the study was adequate.

DR. GREENO: That is a little weaker.

PARTICIPANT: Not to be a competent researcher in the area but to have methodological literacy.

PARTICIPANT: I think that is what I am talking about. The PhD students are empowered to be able to do that and also feel empowered to go to different disciplines, to go to the psychology department because there are grad students in education who do feel intimidated by other programs and that is something that I have spoken to grad students about and they feel like they cannot go to those professors and interact on an equal footing and that is a horrible thing and I think that that is what we have been looking for in a program for graduate students that you empower your graduate students to be able to speak intelligently and go and work with other disciplines and we do need that.

DR. GREENO: I feel as though I need to ask us to close this because it is almost time to start the next one.

I think we have a good start on the three-forty-five session.

(Thereupon, at 1:13 p.m., the breakout session was adjourned.)