Workshop on Understanding and Promoting Knowledge Accumulation in Education:
Tools and Strategies for Education Research

Day 2 – June 30, 2003

Agenda Item: Ways of Taking Stock: Replication, Scaling up, Meta-Analysis,
Professional Consensus-Building - Dr. Daniel Berch

DR. DANIEL BERCH: Thank you. Unlike the other speakers, I’m not planning on changing jobs, unless this talk bombs I guess, but before I start I do want to allude to something that Kenji mentioned yesterday in framing the workshop, namely he spoke about expertise and asked the question about who qualifies as an expert. I don’t know if that was raised again anywhere and I thought I could be of some assistance in that regard since some of you may not know if you operationalize that you can namely define an expert as someone from out of town with slides. I do have slides, I’m not sure if coming from Bethesda qualifies as out of town.

Mostly we’ve been talking about what we know and how we can determine what we know, sort of meta-cognitive activity, but I’ve been asked to talk about how that serves as the basis, or could at least within one federal agency, for setting research agendas. That seemed to me to be kind of a leap and a leap because we haven’t said much about something else that Kenji did mention as well, taking stock of what we don’t know, hence the title of my presentation, Gap Analysis, Bridging the Space Between What we Know and What we Want to Know.

A term actually that is used in a number of fields including land management, but I won’t go into that today, but from information technology gap analysis is the study of the differences between two different information systems or applications often for the purpose of determining how to get from one state to a new state. A gap is sometimes spoken of as the space between where we are and where we want to be. But to some extent that’s the way we frame questions, not only at the NICHD but the NIH in general and I’m sure at many other federal agencies and even foundations when we bring in people who have expertise in various areas and ask their opinions about first what do we know, then what don’t we know, and what should we do to fill in the gaps in our knowledge.

Well, I’d like to suggest that the process of a gap analysis be something very orderly but unfortunately it isn’t, it’s perhaps somewhat akin to the oft quoted Bismark(?) statement about making legislation, that it’s somewhat like making sausage, you don’t want to see the process. But it would be nice if we could have a so-called algorithmic approach to gap analysis, namely if we knew what the state of complete knowledge was like and we’d already figured out what we know we could simply subtract the latter from the former and come out with what we don’t know. But again, it’s not that simple.

So how do we deal with the process, at least at NICHD, in order to figure out how to fill in the gaps? Part of I guess getting into this issue has to do with focusing on the people, the personnel, such as myself, who do the work. And this in fact is something that’s been recently discussed in the NRC report that we’ve been talking about, Scientific Research in Education. In fact you can take directly from one of the principles, the design principles which I believe is in the executive summary, talking about a federal agency, this happens from a federal agency with responsibility in terms of funding research in education, that the agency should be staffed with people skilled in science, leadership, and management. And not only the director but the research staff should hold similar qualifications, that is similar to the director, leadership capabilities, be respected researchers in education, be adept at grant writing, engaging with the field or at least writing grant announcements, engaging with the field to identify research gaps and priorities, and assembling panels of peers to perform various tasks. Fortunately I got in just under the wire here.

There’s actually a similar kid of acknowledgement about staff at the NICHD, and this was written up in a paper actually a number of years ago in the American Psychology by Sarah Friedman and Wendy Baldwin, and they were talking about as we already referred to scientist administrators at the NICHD. The identification or recognition of under investigated but potentially fruitful areas in investigation is important for the success of the institute and for the advancement of science. Scientist administrators are in a unique position that allows them to identify areas in need of special support. They go on to say that responsibilities of scientific administrators is to monitor the progress of grants but also they observed the review of new applications that gives them a birds eye view of their own scientific field. Through their contact with advisors who are well aware of leading scientific issues they learn about potential areas for breakthrough in important problem areas.

Let me unpack that a bit and again, this would apply to many of my colleagues at other federal agencies, this birds eye view, in some of the experiences we have of course are actually right from the get go talking to new investigators and encouraging them as my colleague Peggy McCarl(?) often mentions, that we serve as coaches and cheerleaders if you will. At the NICHD our work is separate from that of people who set up the review panels so we don’t in a sense have a conflict of interest if you will and we can work at the front end and the back end. Some of our work may be almost serving as scouts, it’s not uncommon to be going to meetings and see promising young investigators and trying to attract them to the NICHD, trying to help build their careers through various kinds of post doctoral fellowships and career grants, all of which have to be vetted through the review process, but we sort of do the capacity building and eventually community building as well in addition to trying to support research grants themselves.

The birds eye view also comes from other experiences relating to watching review panels, we attend many of those, I think I’m late for one right now, where we sometimes take copious notes in order to get back to our applicants with details about the nature of the review, and of course we have to have enough expertise to be able to do that and be able to make sense out of it, and that can cover a fairly broad range of areas even within what may seem like a fairly specific domain. Those venues are often very exciting because we’re hearing about research that’s yet to be funded, and many exchanges among top notch researchers that help us in developing our programs further. Some of those exchanges come during the discussions and sometimes during dinner.

Well, what are the ideas, the sources of ideas for developing funding initiatives at the NICHD? I meant to do this sequentially but there might not have been enough time anyway so let me describe some of these to you in a little bit more detail. As we heard about, some of the reports from the National Academy of Sciences are funded by various federal agencies, so we may have some ideas, some areas that we think need further development or we wondered what the consensus is in terms of the state of knowledge, at a particular point in time what do we know, and so we may be involved in commissioning or cosponsoring various study reports. We then may use the recommendations in part, still sticking with the NRC reports, we may build upon the recommendations in the development of our request for applications. In fact one that I oversaw recently in mathematical cognition we did indeed allude to and build upon the reports, adding it up by the NRC and also the RAND report, or a draft of that, on mathematical proficiency for all even though that hadn’t been published fully at the time.

We also receive recommendations from scientific workshops that we sponsor, not unlike some of the things that study directors do at the NRC but we will develop an agenda in our own area, select the speakers, and serve the pastry. And we sometimes cosponsor these workshops with the U.S. Department of Ed or NSF and then private foundations.

Suggestions for research agendas sometimes come from members of our national advisory council, there’s an advisory council for each of the institutes and centers at the NIH, so not counting the Center for Scientific Review that’s 26 other ones, and those are staffed with people who are prominent researchers, it’s not uncommon to have a Nobel Prize winner serving, we have naturally many biologists but also psychologists, some educational psychologists in certain cases for our advisory council. And practitioners, child advocates, etc., who then provide ideas and suggestions for developing research agendas through formal and informal discussions.

We receive directives from the U.S. Congress, most recently we did with respect to developing our new program in early learning and school readiness. Also, although I don’t have it down there, there’s certainly suggestions and directions sometimes coming from the Office of Science and Technology Policy and OMB. Advice and council preferred by scientific societies as well as professional associations, including those comprised of practitioners.

As program directors we also assess the current research literature, and I should say literatures depending upon our particular approaches. So for example my case, my program in mathematical and science cognition learning we take a sort of multifaceted multidisciplinary approach to that topic, and I’ve got to stay abreast of literature in educational psychology, cognitive psychology, cognitive neuroscience, development disorders because we support work in mathematical learning disabilities, in children with some presumed neurobiological basis for their learning difficulties. And so we develop ideas in part based on those reviews. And finally suggestions generated from informal meetings, exchanges, and discussion between program staff and individual researchers.

Then we may combine some of these ideas, modify others, provide some of our own perspectives, for example, in stimulating a new research area or seeding innovative opportunities for collaborative efforts that require combinations of expertise from somewhat different domains, disparate domains, and in certain cases we decided to try to rejuvenate a few by taking state of the art methods from one domain and applying it to another.

Giving an example of the announcement that we recently had in competition in the area that I cover, mathematics cognition specific learning disabilities, you can see the range of topics that we would support all the way from genetic hormone and neurobiological mechanisms underlying math learning disabilities to individual differences to specific learning disabilities, classification, epidemiology, preventive strategies, and instructional interventions. We support work from the study of numerical cognition in infants up through algebraic problem solving in college students.

This is an example from one of our other workshops, adult and family literacy, skip to something that we do. What we often do at these workshops is we’ll pose questions, I’ll just mention two of them here, two of our speakers and participants, here’s one, what do we know about instructional effectiveness? What do we need to know, how might this be approached in terms of research design, methods? Number five, what are the specific needs that must be met in order for rigorous high quality research to be done in these fields? Do we need to help establish research collaborations? Coordination across multiple sites, development of common protocols? Do we need to do some capacity building, again we have various training grants and other mechanisms for doing that so we’re again, building and setting these research agendas by building upon the expertise of a variety of researchers in an area.

As we’re coming close, the final statement, I’ve been thinking again about the meta-cognitive aspects of what we’ve discussing here, and it reminded me of a title from a paper by Anne Browne, probably 1975 I think, I have it down here, called the Development of Memory, Knowing About Knowing and Knowing How to Know, and actually at that time in 1975 I came up with what I considered to be the definitive meta-cognitive study I never carried out, but I thought it might be a fitting closing to a discussion about how we can arrive at what we know. Here’s the title of the study on Knowing How to Know When You Know Enough About What you Know that You Should Have Known That it Wasn’t Even Worth Knowing About. Know what I mean? Hopefully we won’t end up with that.

Thank you.