DR.BULLOCK: I am Mary Bullock. I am in the Science Directorate of the American Psychological Association and it is my pleasure to moderate the second to the last panel of the day.

We are getting toward the end of the day. So, we are hoping that we are going to be lively and interactive in this discussion. We began with a vision of the promises and the challenges of multiple methods research followed by some really compelling examples and now the last panel and I think this panel and the next one we are going to be looking a little bit proactively at what one ought to do, what one can do to try to further the program of multiple methods research.

Today we have assembled a panel of the research world's gatekeepers and facilitators and funders and I hope that this panel today will provide both responses to today's presentations and also an assessment of how funding priorities can, could or should facilitate multiple methods research.

So, let me first introduce the panel. Bruce is at the National Institutes of Health. Actually before I introduce the panel let me say that the panel will present you with multiple perspectives on multiple methods. The panel collectively will represent perspectives from the Federal Government and from the foundation world and each of the people on the panel, and I am using these very, very short introductions and so let me just say that each of the people on the panel brings a wealth of research and policy expertise as well as their current positions as funders.

So, now let me start. Bruce Fuchs is at the National Institutes of Health and directs the Office of Science Education. His office covers K-12 curricula and also develops resources for teachers and for students and he is a researcher with a background in immunology.

Next is Daniel Berch who directs the program in math, science learning at NICHD and he comes with a broad background in research, research regulation and education policy advice where at least from the biography that I read that you could read he advises the Assistant Secretary of the Department of Education on education research policy.

To my right is Barbara Grombach who comes from the Carnegie Corporation and is in the program association educational division at Carnegie.

Her background is in academic teaching, university school collaborations and she was assistant dean at Columbia University, and next is Barbara Olds from the National Science Foundation who is the Director of the Division of Research Evaluation and Communications and has a background in assessment of student learning. Sorry, I am not reading my own notes and also interestingly a background in engineering education.

So, I think across the panel we have really a wealth of experience in what multiple methods research might mean but much more directly what it might mean to us today.

I would like to begin by looking back over what we have heard, and I think one thing we have heard is that getting real answers to important questions requires multiple levels of analysis, requires multiple methods; that is why we are all here. It requires multiple researchers and to attain this ideal we have also heard is expensive and may require a change in the culture of research. It may require doing research in different ways with different partners.

So, what I would like to do is begin a conversation by asking each of the panelists to do two things, first to discuss what opportunities they currently see in their program that are relevant to further multiple methods research and then after that I would like to turn to them and ask them what message they will take from what they have heard today that will lead them to proactively think about what comes next.

So, let me start with the first question and ask each of you what you currently see in your programs that supports and facilitates multiple methods research. It is very difficult to know where to begin. So, I am going first to my left and I decided we will do this in reverse alphabetical order.

DR.OLDS: Dan Berch got to her. I want to talk just a little bit about some of the current work that we are funding at the National Science Foundation primarily in the Directorate for Education and Human Resources. There are of course other directorates at NSF that fund education research most notably the SBE directorate but the one with which I am most familiar is EHR and there are a number of divisions in EHR, all of which fund research at some level or another. So, you have Brutus Sever over here who has a program of research on women in education and we have Karen King and Janice Earle and others here from the elementary, secondary and informal divisions which funds research in the TPC, Teacher Professional Continuum program in the instructional materials development program and other places. Rita is from the human resources development group. We had some people here earlier today. I don't know whether anyone is still here from the Directorate or Division of Undergraduate Education. That one should be obvious and we also have a division of graduate education and one of the things that I found interesting in the conversation today related to training of graduate students is that that division and the division that I am in, the Research Evaluation and Communication Division have what we call at NSF a Dear Colleague letter where we are jointly funding research on graduate education. So, if you have nifty ideas to follow up on with some of the things that have been talked about today please take a look at that Dear Colleague letter and submit something if you are interested.

We run in the Research Evaluation and Communications Division three programs, two of which are joined very closely, the research on learning and education and the evaluative research and evaluation capacity building and in true NSF fashion of course everything has an acronym where those are known as the ROLE program and the EREC program and they are a single solicitation which has just been posted.

We have revised that solicitation. We have a March deadline for proposals in that solicitation. Research on learning and evaluation is a program that has basically four areas of emphasis we call them. They used to be quadrants for those of you who are familiar with the old program. We have just recently redone that solicitation and those range from neurological and biological elements of learning to cognition through classroom applications and now the fourth of those emphasis areas has to do with policy issues and/or with diffusion of innovation.

So, some of the things that I am hearing today are resonating very nicely with the ways in which we are rethinking the role of solicitation. I would encourage you to take a look at that. EREC, the evaluation research program encourages proposals that look at and don't necessarily do evaluations but do research on evaluations and then there is a strong capacity building component to that.

We are interested in training more people to be evaluators in all types of methodologies.

The third program just briefly is one that we have jointly sponsored with IES and NIH and that is the intergovernmental education research initiative, IERI.

For the last year and this year we are all having separate competitions but I think the spirit of IERI remains the same across all three of the groups and that is really looking at ways in which we can take interventions and scale them up and do research on that scaling and so in at least NSF's portfolio many of the experiments that we fund come in the IERI program although not exclusively.

We are quite agnostic I guess you would say in terms of methodological preferences. The field drives the research that we fund. We look at the appropriateness of the questions and the methodology and we are in the ROLE and EREC programs trying to fund everything from pre-K through graduate. So, we have a very limited number of dollars and very broad constituency both in terms of types of research and research questions and I think I will stop there.

DR. GROMBACH: A very nasty virus felled Dan Fallon over the weekend, the Chair of the Education Division at Carnegie. Otherwise he would have been on the panel as is listed, but I am very pleased to be speaking in his stead.

I thought I would start talking a little bit about Carnegie with a quote that provides a little bit of comic relief but also with some serious intent as you will see.

About a century ago an editor said something about good writers that I think applies to good researchers, a type that has clearly been abundantly evident here today. He said this, "Fine writers should split hairs together and sit side by side like friendly apes to pick the fleas from each other's fur."

Now, this is a metaphor of a community of mammals engaged in activity that is painstaking, pragmatic, conducive to individual and collective health and socially responsible. I think that is a fine metaphor for the work undertaken here today.

Now, I represent a nearly century old philanthropy but the grants budget in education, in domestic education at Carnegie rarely tops $25 million a year. So, Carnegie is rather more like the fleas than the apes in that metaphor but fleas are as significant as apes in the ecosystem. So, I don't mean with the little metaphor to minimize the role of private philanthropy.

Carnegie's president, Vartan Grigorian(?) often says that the corporation should be an incubator not an oxygen tent and that metaphor pretty succinctly conveys Carnegie's general stance these days. In eight words I can say what the education division is funding these days, urban school reform, adolescent literacy and teacher education. Support for research is an explicit although a small aspect of grant making already. So, I have resonated very positively with much of what I have heard today and as you said, Barbara, also, Carnegie is methodologically speaking agnostic about supporting methods. It is the question that drives the research as you all well know.

Instead of talking in any detail about the specific grant making though I wanted to give just one example, an illustrative example of how the flea can nevertheless be potentially influential in the larger ecosystem of education research.

About 5 years ago value added modeling for teacher accountability was looming as a potentially transformative methodology. It was troublesome to many people though because the leading advocates didn't operate within the usual boundaries and canons of the academic enterprise.

On the one hand researchers were understandably skeptical. On the other hand interest was building in public policy quarters, state level and municipal level without regard for the serious reservations of the researchers. Was good policy being constructed on bad science or not in this particular case?

So, here was a rather commonplace and real world dilemma that private philanthropy could make a difference in. So, at that time a grant was made to the RAND Corporation. It enables a team of researchers to critique the methods used and the validity of the claims by that leading advocate whose work as I said was gaining increasing attention in policy circles. The RAND team's critical analysis led to a monograph that I have with me but neglected to bring for show and tell. It is called Evaluating Value Added Models for Teacher Accountability and the authors were Dan McCafree, J. R. Lockwood, Dan Court and Laura Hamilton. Doubtless some of you know of it.

So, as reported in the volume the group of researchers did several things. One of them was this. They used a particularly widely cited paper that had not been peer reviewed, a 1996 study by Bill Sanders and June Rivers at the University of Tennessee. They said the model on which that research was based as false data and their objective was to see if they could force the model to produce false positives.

The found that the model was biased but nevertheless did produce results in the direction that Sanders and Rivers had reported. So, in other words the RAND team among other things as part of this grant showed that one can believe the findings of this Sander and Rivers study that had on the policy level already been influential and gaining traction.

So, this was as I said a small piece of that particular grant but extremely worthwhile for Carnegie because value added modeling for teacher accountability has enormous and direct implications for teacher education, an area that Carnegie is now funding heavily given its own resources along with several other foundations, Ford Foundation and Annenberg and so on.

So, this is just by way of illustration one example about how a relatively small grant from private philanthropy can sometimes provide leverage in the research community without significant dollars behind it.

DR.FUCHS: The office I run at NIH, the Office of Science Education is both a policy and a program office. It was actually formed as a policy office to advise the NIH Director on education policy issues and whether or not the NIH should get involved and to what extent.

We became a program office in the midnineties when our director wanted us to get involved in doing things to enhance the teaching of science. So, while we occasionally fund some research out of my office we are not a major funder of science education research and Dan will talk to you about his portfolio programs, but I have been following this issue,this area of education research for a number of years.

I was appointed as the NIH Director's representative to the National Educational Research Policy and Planning Board which was advisory to OERI before IES.

So, I must admit that honestly I went initially to those meeting kicking and screaming. I had very little interest, but I got hooked and I see myself primarily as a program developer. I want to be a consumer of research and I have been disappointed in the research that is out there and its ability to guide me in designing the programs that I am mandated with creating.

So, I think that the way out of this as far as I am concerned with this community is the education research community. You hold the power and the promise for improving the situation that many of us find ourselves in.

Now, I have watched. As a part of that Policy and Planning Board we funded some of the initial work of the Academy looking at science and education research and so I got a front row seat for some of the paradigm work and it did break my heart because I don't think it is a productive way to spend our time, generating more heat than light I guess, but I hope that we are past much of that and I am confident that 10 years from now, 20 years from now we are going to have a much larger installed base of really valuable research for folks like me to turn to to answer questions.

I will leave you with one question that I asked my colleagues on NERF while I was there when I was still relatively new that I never got a satisfactory answer to and I think it still sort of hangs out there in my mind and I asked my colleagues if one of them would please draw me a diagram on the back of an envelope and describe for me the pathway by which education research works its way into clinical practice and I offered to do the same type of thing in medicine and I am relatively confident that if I were to draw a diagram here and I asked Bob Dehanza to do the same thing that the diagrams that we would draw that would trace the path of a basic research finding on the NIH campus into a drug score, into a surgical suite would be pretty close to one another but I was discouraged by the fact that none of my colleagues on the board took me up on that offer. I was embarrassed initially. I thought I had asked a dumb question but I don't think it is a dumb question and it is something that I believe we need to begin to think about as these wars settle down and we turn to the business of doing research.

DR. BERCH: I am Dan Berch and I will make a few disclaimers. First I am not speaking for all of the NIH which is comprised of 27 separate institutes and centers. Not all 27 fund research but over 20 of them do. I am not even speaking for all of my institute, the National Institute of Child Health and Human Development which is made up of many different centers, programs, branches, etc. There are primarily two branches within the institute that fund educational research, the Demographic and Behavioral Sciences Branch and our branch, the Child Development and Behavior Branch. Even within our branch there are seven different programs, one of which is behavioral pediatrics and they even do some educational work there, but the four programs that are most involved in education research are human learning and learning disabilities, language, bilingualism, biliteracy, early learning and school readiness, a comparatively new program as is my program the fourth one, mathematics and science cognition and learning.

By the way I was looking at my notes which I have been developing during the day and if I trip over my tongue a little bit I don't know if you can see it they are fairly scribbled which someone generated from the, when someone mentioned Robert Rosenthal earlier, a aphorism at the beginning of one of his articles which was Yiddish somehow never got to the second step I guess but I wanted to speak a little bit about funding opportunities but not so much specific ones. I will mention a little bit more about how we go about developing those because it is part of the issue of how do we get influenced; how are some of those decisions made, etc., and a little bit more about some of the grant mechanisms, and I don't mean for this to be a technical systems workshop although I want to say some things about those kinds of workshops.

Part of what I want to say was triggered by something that, Bob Dehahn, twice today was mentioned and something earlier I believe to the effect that both NSF and perhaps NICHD should think about program project grants. We do have such grants and sometimes those mechanisms are mentioned in our solicitations but what people don't understand about the way we operate is that our programs are ongoing programs and even without solicitation you can submit investigator initiated or field initiated applications and there are a number of different funding mechanisms that include things like small grants, the standard RO1 grants and program project grants. The program project grants usually have to have at least three projects and they must produce well, something where the whole must be greater than the sum of its parts. They can be multi-site. They don't necessarily have to be and they can be 3 to 5 years in length with a direct cost up to $750,000 a year. We just funded last year a program project grant in mathematical learning disability and you are talking about multi-disciplinarity and that team has a developmental psychologist, a cognitive psychologist, a neuropsychologist, a special education researcher and a neuroscientist.

While at another institute we funded a project that I oversaw, a 6-year, a $27 million project with 38 investigators at 14 different universities using methods anywhere from an expertise that ranged from neuro-imaging to collection of daily diaries from psychoneuroimmunologic techniques to life histories all packaged into one.

So, there are a number of certainly these kinds of opportunities that already exist. We are a little less agnostic about research methods than NSF and pardon me, I don't think this has to go on the web for those who want to put that on there, that statement. I am always reminded of what Woody Allen said many years ago on the Tonight Show when he was asked if he had ever been married. He said, "No," he got close to it once but he was an agnostic and she was an atheist and they couldn't figure out which religion not to bring their children up in.

(Laughter.)

DR. BERCH: What I am about to read to you is a paragraph I have been writing for another purpose. It is not an official statement of our institute. I synthesized some of the statements that are made in some of our solicitations and it is focused more on the math and science cognition domains of my program and as I read that at least it speaks to some of the issues I think people raised. NICHD accepts grant applications incorporating a wide variety of study designs, research methods and data analytic procedures. The most important consideration is an appropriate fit of the proposed methodology for the research question.

Projects may include experimental quasi-experimental and/or correlational designs among others. Of particular interest is the use of prospective longitudinal designs for examining developmental changes in mathematical and scientific thinking and reasoning.

Epidemiological studies are sought for the purpose of estimating the prevalence of specific learning disability in mathematics. The institute also encourages the use of various neuro-imaging modalities such as FMRI, magnetography, for investigating neurobiological substrates of both normal and atypical mathematical and scientific thinking. They accept quantitative genetic design, for example, twin studies may be used for examining contributions of genetic and environmental factors for the variation in mathematical ability. A variety of data analytic strategies may be employed usually driven in large part by the nature of the study design. A strategy of particular interest to us is the use of growth curve modeling for studying subtype and other individual differences in the developmental trajectories of mathematical learning skills.

Qualitative research processes are welcomed as an adjunct to quantitative approaches insofar as they are methodologically driven the data the generate can be considered trustworthy, credible and transferrable, and they can be used to enrich and illuminate theoretically based interpretations of both quantitative and qualitative data.

Finally, small and large-scale evidence based instructional interventions may be carried out, Forms and methods suitable for testing efficacy and/or effectiveness are employed. Now, that is not to restrict that, for example, if it doesn't favor regression discontinuity design that doesn't mean those can't be used.

The other important thing to really understand here is that at the NIH in general and NICHD in particular there is a fire-wall between the program component and the review component. So, if you are interested in certain kinds of studies you could speak to me or other program directors up front but once the application comes in I have no control over it. There is another entire group of people who handle the review. They pick the peer reviewers who evaluate your application and give the scores and I think of the question I asked earlier of the mini study I think here the one with the highest score wins if you will and the score makes a difference.

Afterwards that score and that application is then taken to a sort of second level review at each of the advisory councils. Each institute has an advisory council. It has to be signed off at various levels.

What I wanted to mention in that regard and I am not going to go into any details about peer review but I guess what I left out is what happens; how do we devise these solicitations, and there are just a few little things to give you a sense of how what you say, what you said today, what people have written may influence us. We often hold a workshop or conference that is sponsored by sometimes several other agencies, federal agencies and sometimes our professional associations as well.

That conference is then written up in some form as a proceedings, sometimes submitted as a special issue to a journal, sometimes in the form of an edited book.

There have been cases where after the scholarly conference we bring in practitioners to speak to the issues that were raised by the researchers to get their take on some of those factors that we then consider when we develop a solicitation. Most program directors including myself have been researchers and we also take some responsibility for leadership in terms of our knowledge of the literature and our role in trying to get the communities to move in certain directions to fill gaps which we believe aren't being filled, but we are constantly operating in conjunction with researchers at formal meetings and informally, etc.

We then may use some of this information to develop a solicitation in part and as I mentioned before the peer review is done in a separate sort of fashion. I must make a little comment and I hope this won't go too far politically or whatever but I thought about this as I attended a session at AERA last spring entitled Is the Federal Government Taking Over Education Research to which I had two reactions. One, I thought if you applied it to my federal agency what sense would it make to say is the NIH taking over biomedical research? My learned answer to that is duh. I can explicate that later.

The other sense of funders and I hope this doesn't sound too defensive is that we are a bunch of no-nothing, pointy-headed bureaucrats huddled together in a dark corner somewhere secretly plotting to take over education and research for our own nefarious purposes. I think this attitude is at best misguided and wrongheaded if not patently ridiculous and at worst potentially damaging to a variety of efforts by associations, researchers and so on to convince the Congress which we can't do because we can't lobby Congress to convince the US Congress to increase funding for educational research.

My last statement is we do have a number of different opportunities for funding of not only educational research in general but multi-methods approaches and approaches to learning more about methods in general. We just came out with an RFA from our early learning and school readiness program to develop theoretically driven outcome measures for school readiness aimed at one or more content areas, reading, math, etc., to be universal in design and potentially it will be taken to scale. That was just announced and the NIH in general has a methodology and measurement program announcement that is going to be revised next year that also calls for multi-methods and multi-disciplinary approaches to behavioral and social science research and even if it doesn't say educational we take it to include that from the standpoint of our institute.

Thank you.

DR. BULLOCK: Thank you. This is actually I think a very good segue into a question that I was thinking as each of you were describing the kinds of things that you fund and it is really a more general question looking at the funding landscape.

You just said that you wouldn't speak for your institute or even for your organization but now I am going to ask each of you step even further back and maybe speak as funders quite generally and continue with, Daniel what you raised to really ask the question to what extent can you or do you as funders serve a proactive role for trying to distil what seems to be happening in the research community and proactively try to bring about a culture change and much of what we have been talking about in multiple methods research has come up over and over again and we need to do things a little bit differently than they have been done in the past.

Yet I hear from funders a sense of we need to hear what the community is doing and then we will fund it and my question I think is to what extent do you as funders take your role to be one of also listening to what is happening in the research role, what problems need to be addressed and proactively think of or design program or calls that to some small degree bring the research community where you think it needs to go.

DR. OLDS: I think that we have sort of a delicate balance between what we call field driven and trying to be to some extent directive at NSF and we have in general erred on the side of letting the field tell us what the important issues are and that works in the sense that the proposals that we get obviously are dealing with things that the people in the field think are important but also when we bring in review panels the panelists as well in terms of making recommendations, we work somewhat differently from NIH as you probably know but the panels at NSF recommend whether a proposal is competitive or not but the final decision is up to the program officers.

So, I think the program officers have in a sense a way of shaping the portfolio and we are moving more and more to thinking about what we fund as a portfolio with a variety of possibilities involved, dualities such as high risk/low risk, how much of your portfolio should be risky; how much of it should be not risky, more mundane things like geographic distribution, diversity of PIs, a whole range of questions that we can in some ways use to shape the portfolio that we end up with even though we do have field-driven programs.

DR. BULLOCK: What would you like to have from the research community? You have in front of you the best among the educational research community. So, perhaps that would be a --

DR. OLD: One of the things that I think has been pretty clear all day today although I am not sure we have heard it too many times is that we I think tend to agree pretty largely on most of these issues. I am not hearing a huge amount of disagreement nor do I think when you really look carefully is there a lot of disagreement among the funding groups about some of these issues.

So, in my opinion this is one of those instances where I think we have to hang together or we are going to hang separately and so I think we have to do a lot more in terms of collaboration about both methodologies and the kinds of questions that we are studying. I think the federal agencies although we do have some history of collaboration and working together can do more. I think the foundations and the federal agencies can work together more closely and I would like to see that kind of effort grow. Certainly it is not all going to happen as a result of a 1-day meeting but I think I can see some ways in which that is possible and desirable.

DR. GROMBACH: Although, Merry from the point of view of private philanthropy I wouldn't much as I would like to be an agreeable panelist I wouldn't pretend to speak for other philanthropies. You all know the old saw, "If you know one foundation, you know one foundation." So, it has emboldened me today to share some of my own perceptions in this to tell you how I see things in one foundation but I think you all know very well that that by no means applies to others, but as far as the question about the push and the pull one of the things I was reflecting on today stimulated by many of the conversations here was that I sense in private philanthropy not only in Carnegie but also in interactions with other colleagues a tendency with regard to education research; this is overstating it but sometimes to both step on the gas and step on the brakes at the same time.

There is a very strong desire and active desire to both support and perceive good ideas and to spread them but at the same time there is considerable caution about what is the base of those good ideas. As I said before, is it good science? I hear this playing out in philanthropy in agonizing and continuous conversations about what constitutes evaluation of success of projects either an individual grant or a collection of grants and so on.

So, I very much perceive this push and pull both. The example that I used of the grant to the RAND Corporation was really stimulated by on the one hand by frankly Dan Fallon himself by his own knowledge of what was current in research, realizing the emergence of value-added modeling but from his viewpoint of looking into within philanthropy looking at public policy impacts of grant making. He then started talking to researchers and saying, "Would you like to get together and take a look at this new development that you as a research community seem to be extremely cautious about and yet it is going great guns?"

So, there is an example of where the idea was based on the stimulus but the grant making was based on a very strong knowledge base within the corporation of particularly one individual that then of necessity turned to the education research community for do you think this is a good idea and do you think it is worth doing.

DR. FUCHS: I am going to give you one more observation. I don't think I have been to one of these meetings in the past 7 years where someone hasn't brought up the medical model in either a positive or negative sense. Sometimes education research ought to be more like medicine and sometimes the claim is that education research is so much more complicated that the simplistic medical model has no applicability.

Let me make one observation of a difference in the cultures in medicine and education that I think is something we will have to address at some point.

If we go back to Dick Murnane's example from this morning of the cystic fibrosis research I think one thing you can assume about that team that we couldn't assume with an analogous team in research is that every member of that team is not agnostic when it comes to research. The PI, the docs on the wards, the nurses, the respiratory therapists all believe at some level that research is going to improve their practice. They are going to depend on research. That is why they are participating to improve their patient outcome.

I see a much larger gulf between education research and clinical practice in the education world. I spend a lot of time each year talking to thousands and thousands of science teachers through our program and I often encounter open derision when I talk about research. The term carries no power there. Every silly notion that anybody wants to foist on a teacher comes with the sticker that it is research based, whatever it is and so I think that is a major impediment that we are going to have to think our way.

I don't think that a family practice doc in my hometown of Springfield, Illinois in the corn fields is any better prepared to pick up a primary research article in PNSA and understand what is going on than is a classroom teacher but I think that the family practice doc has had a culture of knowing that research is the answer. I don't think that is the case for many classroom teachers.

DR. BERCH: Let me turn the question on its head a bit and respond to it more directly. I think one of the issues is just making sure that the community or communities out there know what the various agencies and foundations fund. There is an unbelievable lack of knowledge out there. I can tell you that based on a number of factors including personal experience at giving technical assistance talks at AERA and a variety of other places and I appreciate that that associations permit and even suggest that we hold these kinds of events.

One of the things that is interesting and relates to what I think someone mentioned before is there aren't many people showing up and second, very few ask questions or stick around even though we stick around to answer them afterwards. The people who usually do show up are graduate students because I think the faculty are embarrassed to come and they send their students there which doesn't really help and they usually don't know enough to advise their students on what to do about that or how the various agencies work.

It seems to be more complicated with the NIH especially if you come to our web site and we are immense. The entire NIH reviews 60 or 70 thousand applications a year and millions of dollars just bringing in experts to do peer review, etc.

So, one of the things that you can do is try to attend more of those sessions and in our case we may be able to come out and provide those kinds of technical systems workshops at a university but if we do that we have to invite other universities in the region so that you can level the playing field, but you can certainly explore that and talk to us and we are more than happy to try to provide you information and walk you through various web sites, etc.

So, it is not that we don't want to listen to ideas, but again if you are not aware of what we are already doing then some of that time is wasted.

The second thing is what are the ideas; where are they coming from and how are we to respond to that? Even today there were some suggestions made to me with the recent NRC meeting that we cosponsor with NSF about science learning and some of the researcher on that particular study group talked to me about some of the things that they think should be funded in the future. Again sometimes we meet with heads of associations, both formal and informal settings to hear those kinds of ideas. That said, you have to remember that we are not deciding what research people can do entirely. We are deciding how we will spend the public's money. We are stewards of that money and so we, again, with expertise that we draw on from various communities make those decisions.

At NICHD within our own units that tend to fund more education and learning research, and I want to be cautious how I say this, we aren't necessarily fans of letting 1000 flowers bloom because we are very careful about figuring out what our money buys us and not us per se but when we report to the Congress about how that money was spent and what did we learn from that. So, we often fund, what we do is have our researchers form networks and especially when they respond to RFAs and they are funded when they come in for a principal investigator meeting it isn't like well you come in, give your 2 cents and walk out. We try to encourage them and we don't force them to perhaps use some common measures if they are all working on the same topic and if not collaborate at the very least communicate and cooperate to get more bang for the buck.

So, again, I think we try to be as responsive as possible to the various suggestions and ideas and it takes quite a while to get those vetted. We are not as likely to be able to do it for a particular program independent of the rest of the institute. Those ideas feed up through the hierarchy and through our advisory council which is made up of top-notch researchers in some cases Nobel prize winners. We do have a person who has done educational research, an accomplished person on our advisory council now. So, these things are constantly vetted. It doesn't mean that we can't hear any more or listen to more things but we make a real effort to do that.

DR. BULLOCK: I think from listening to the comments you can see that the door is open for continued dialogue.