Committee on the Assessment of Research Doctorate Programs

Summary Minutes of the November 27-28, 2005 Meeting

Washington, D.C.

Attendees: Jeremiah Ostriker, chair; Virginia Hinshaw, vice-chair, Elton Aberle, Norman Bradburn, John Brauman, Jonathan Cole, Eric Kaler, Earl Lewis, Joan Lorden, Carol Lynch, Robert Nerem,,^* Suzanne Ortega, Robert Spinrad, Catherine Stimpson, Richard Wheeler. Staff: Charlotte Kuh, Peter Henderson, James Voytuk, Rae Allen. Also attending: Ralph Cicerone. (11/27 AM)

Following introductions, Mr. Ostriker opened the meeting saying that graduate education is a sort of step-child in higher education, but is very important as the engine that produces new faculty. It doesn’t receive the attention that it should, resulting in a low level of accountability and a lack of transparency in its operations. This study will have three primary audiences: 1) the enterprise itself, which will gain better information about how it operates, 2) administrators, who currently lack objective measurements, and 3) students, who currently lack comparative information about the programs they might consider attending.

The committee will need to take on a number of tasks.
1) The taxonomy of fields. Although largely determined by the Methodology Study, there are unresolved issues concerning the treatment of emerging and cross-disciplinary fields, the organization of biomedical PhD programs and whether the study should address those fields that are a combination of research and professional orientation.
2) Estimation of program quality. Although the idea of using expert panels to place weights on quantitative measures is appealing, this approach is subject to criticism and questions of conflict of interest, whoever the experts are. An approach that samples experts would have greater validity than a pre-selected panel.
3) Study output. The committee will have to work out the details of dissemination. This would include specifying tools that will allow students to enter their preferences and product customized ratings.
4) Updates. The committee will need to decide on how frequently updates should be conducted and what data elements should be updated. It is clear that waiting a decade or more is unsatisfactory.

Mr. Bradburn noted that what is assessed communicates what is important. Different audiences may have different notions of quality. Mr. Brauman observed that the study has an obligation to present data so as to minimize the probability of misuse. Ms. Stimpson added that data should be aggregated in a way that makes clear what matters. She hoped that it would be possible to gather information about teaching post-doctoral positions in the humanities. Mr. Cole hoped the committee’s narrative would be a guide to both the uses and the limits of the data. Mr. Cicerone added that it might be a good idea to involve graduate students with the data, possibly through review of the committee essay or through validation to the data. He is impressed by the experience and commitment to quality of the committee membership. Although U.S. doctoral programs of the envy of the world, the question is, how do we keep improving them? When he discussed the study with government agencies, he had found that they viewed the committee’s task as a university responsibility, a kind of accountability, not a government responsibility. He added that he thought the study could add value and it was a plus that the data would be updateable.

Following a discussion of bias and conflict of interest, Ms. Kuh and Mr. Ostriker discussed the history of the project. Mr. Cicerone then shared his views on the question of graduate program assessment. It was generally agreed among the AAU presidents that there was a need for ranking, but that the methodology for obtaining such a ranking should be transparent. Reputation should not be the only measure used for ranking. Defensible variables should be used that reflect the value of the programs. Although reputation was potentially important, it should not be the dominant consideration

Ms. Kuh briefly explained the funding plan for the project. Funding has been obtained from the Sloan and Mellon Foundations, NSF, and NIH. Mr. Cicerone had sent a letter to universities asking for contributions, the exact amount being dependent on the average number of PhDs produced in 2001-2003. Letters of commitment had been received from over 60% of the AAU institutions and it appears that this method of financing will be successful.

The committee then met in closed session. Measurement of perceived quality and how its relation to quantitative measures could be modeled was the primary topic of this session. The committee also discussed the question of which fields should be chosen for an administration of the trial student questionnaire. The fields chosen were: chemical engineering, physics, microbiology, economics, and English.

On November 28, the committee again met in closed session to discuss the assessment methodology and its approach to taxonomy. It decided to hold an open meeting on taxonomy and sub-fields at which fields that wished to be included could make their case. Dr. Kuh was asked to draft a paragraph specifying the data that would be needed for these presentations. A sub-committee on taxonomy was appointed. Its members are: Elton Aberle, Carol Lynch, Suzanne Ortega, Norman Bradburn, Robert Neerem, and Richard Wheeler

A preliminary discussion of commissioned studies was then held. Ms. Kuh asked for suggestions for the areas to be covered by the commissioned papers. She said that she would send the committee some examples that had been in the proposals for funding. Examples that came to mind were: faculty career paths in the biological sciences, how graduate education has been changing and analysis of the demographics of faculty in different fields, an examination of the characteristics of high quality programs, a contrast of programs by completion and/or attrition rates. The idea is to explain to institutions and scholars how rich the data base is and how it can be used to improve both the understanding of graduate education and practices in doctoral programs. Mr. Ostriker suggested that the analysis of students and what resources they have available would really provide new information and be more useful to students. It also would be a good idea to take into account the work on doctoral education that had been carried out by the Mellon Foundation for the humanities. Ms. Hinshaw suggested that a multidimensional analysis would be very helpful. Can we talk about “types” of graduate programs? Can we say anything about “fit” between student background characteristics and program type? Mr. Ostriker agreed that sort of approach would make the study appear less focused on just one dimension, but that no good way of doing this had been found. He also suggested that an analysis of participation of women and minorities would be informative, but wondered if it were possible to look at that issue over time. Changes in the taxonomy would make this very difficult to do for some fields. Mr. Brauman suggested that the design of the updateable data base should keep in mind the importance the capability to conduct comparisons over time. Mr. Spinrad said that to accomplish this it would be important to set an update frequency and a release date. There followed some discussion of how the updating might be accomplished and how it would be financed. Mr. Bradburn noted that the AAU is already collecting much of the data we wish to collect and would like to enter it into the NRC database as seamlessly as possible. This is an issue the committee will need to address.

Ms. Kuh then raised the issue of communicating to the higher education community. She said that this study is unlike many NRC studies in that the broader community wishes to be informed about the committee’s recommendations as they are arrived at, so they can prepare themselves to provide data. Mr. Ostriker suggested that it would be good to keep the broader community informed, but it would not be advisable to get into too much detail about our procedures. For example, we could say that, as directed in our charge, we are will produce ratings and, quite likely, rankings and if they want to understand how these would be reported, they should be referred to the Methodology Study. Then, on the question of how we will obtain quantitative ratings, we should say we are doing more than one type of study at the present time.

There was a short discussion of where dissemination should occur. It was agreed that the committee should communicate regularly about its progress and conclusions with organizations representing the higher education, such as the Council of Graduate Schools, the National Association of State Universities and Land Grant Colleges, and the Association of American Universities.

Following a short discussion of whether it would be possible to study outcomes for program alumni, it was agreed to refer this issue to the Data Panel.

The meeting was adjourned at noon.