Summary of Clarifications and FAQs

Core faculty are faculty members who are currently (as of 2006-2007 academic year) and formally designated as faculty in the program AND have either served as a principle dissertation advisor or member of a dissertation committee within the past 5 academic years (2001-2002 through 2005-2006) OR who are serving as a member of the graduate admissions or curriculum committee.

• A faculty member may be listed as core even if their salary is not directly paid by the program. For example, a program in Comparative Literature may be entirely made up of faculty from the French and German Departments. Those faculty may be counted as core in Comparative Literature (and also in French or German) if they meet the requirements above (formally designated as faculty and serving on dissertation committees) even though their salary may be formally paid by the French or German Department. This also applies to faculty who are supported on grants or fellowships.

• Outside Readers (who only read the dissertation once it nears completion, but do not contribute to its development) and faculty employed by other universities should NOT be listed as core.

• Endowed chairs should not be listed unless they satisfy the above criteria for core faculty (formally designated as faculty and serving on dissertation committees).

• Emeritus faculty should not be included unless they have, within the last 3 years, either chaired a dissertation committee OR have been the primary instructor for a doctoral course listed in the catalog for credit.

• Adjunct faculty should generally not be listed as core; however, they may be listed as Associate faculty if they have served on or chaired a dissertation committee within the past 5 academic years (2001-2002 through 2005-2006).

New faculty are those faculty who do not satisfy the dissertation service requirements for being listed as core faculty (above), were hired in the past 3 academic years (2003-2004 through 2005-2006) and are expected to become involved in doctoral education through the process of sitting on or chairing dissertation committees.

• The expectation that a new faculty member will become involved in doctoral education is usually evidenced by tenured or tenure-track appointments. However some programs do not have a traditional tenure process; in these programs, non-tenure track faculty may be listed as new if they are expected to supervise doctoral students.

• If new faculty members are already serving on a dissertation committee, they should be listed as core faculty.

Associate faculty are those who have chaired or served on dissertation committees in the program within the past 5 academic years (2001-2002 through 2005-2006) but are not core or new faculty in the program. They must hold some type of university appointment; they may be core in other programs or employed by other research institutions (such as national labs, medical centers, professional schools, etc.) Do NOT include outside readers or faculty currently employed at other universities.

• Associate faculty must be actively involved in advising students (regularly serving on dissertation committees) and should be available to advise future students in the program. “Outside” committee members that are required by the program but only review the dissertation once it nears completion should not be considered Associate faculty.

• Only count dissertation committees on record with the Graduate Division or program director (usually when the student advances to candidacy or files for the degree). A mentor or advisor assigned to a new student does NOT count as dissertation committee service.

• Chair versus Principal Advisor – Institutions vary in what they call the person who works most closely with a student on his or her dissertation. The allocation formula gives five points for that activity. In many places, it is called the “chair” of the dissertation committee. In some places, the chair is only nominally involved; in that case, we want the points to go to the person who serves as the student’s principal advisor. In the case of co-chairs, where two people work closely with the student in developing the dissertation, both may get credit for serving as principal advisor.

The allocation of faculty productivity will be based on the Faculty member’s status in the program (core, new, or associate) or programs and their dissertation service.

• The formula posted in the FAQ (see # 35 below) has been generalized for those Faculty who are Core in one or more of the programs that are included in the study.

• The formula for Faculty who are associated with one or more programs in the taxonomy but do not appear as core in any of the included programs, was modified to reduce the known over-allocation of these faculty members. It has been noted that if an associate faculty either chairs or serves on many dissertation committees outside their core program, the formula would allocate an increasingly large fraction of the faculty member’s productivity to the associate program. This is due to the lack of information about dissertation service in their core programs. To reduce this effect, a factor of 2 was multiplied to the denominator (and the + 5 dropped). With this modification, the allocation to an associate program for those faculty whose core program information is unavailable will never be greater than 50%. Please see FAQ # 35 below for the revised formula.

• The allocations will be calculated directly by Mathematica from the faculty lists. The ICs will have the opportunity to review and, if necessary, revise the allocations. A faculty member may have a zero allocation to a particular program if it is not appropriate for a portion of their productivity to be allocated to the program; however, their names will still appear on the faculty lists for rating exercise.

1. What is the NRC Assessment of Research Doctorate Programs?

The NRC Assessment of Research Doctorate Programs is an effort to help universities improve the quality of their doctoral programs through benchmarking; providing potential students and the public with accessible, readily available information on doctoral programs nationwide; and enhance the nation's overall research capacity. The National Research Council has conducted other assessments that were published in 1982 and 1995.

The previous studies gathered data that described doctoral programs—size, university resources, program faculty productivity, and student characteristics. The 2006 study will expand these data to include data relating to Ph.D. student financing, teaching, and other aspects of student resources. The data that will be collected for the study are found on the Questionnaires section of this site. Institutional coordinators will be notified when the questionnaires have been finalized.

Following data collection, the Committee to Assess Research-Doctorate Programs will publish a summary analysis of the data and will release the data on the web. The web database will also contain software that will permit users to conduct their own analyses of the data.

2. Should the questionnaires be filled out on the ResDoc website?
No. These are prototypes which are presented so that you can prepare for the study’s data requests. The actual questionnaires will be interactive. Prior to filling them out, you will be given a password and directed to a secure website.

3. Will the Assessment produce ratings of doctoral programs?

Yes. For more details of the approach that the Committee is taking, see Ratings and Rankings. For FAQs on ratings, please see the “Ratings” section below.

4. What is the taxonomy and what is its use?
A comparison or assessment of doctoral programs requires a framework within which that assessment may occur. The Committee has chosen to keep the general framework used in past studies in which doctoral programs are arranged by field. An institution may name more than one program in a field. Faculty and students are associated with programs. For interdisciplinary programs, see further FAQs below.

To be included in the study, a field must have granted 500 doctoral degrees during the period 2001/2 to 2005/6 and there must be similar programs offered in at least 25 universities.

5. Why is a list of sub-fields included along with the fields in the taxonomy?
The sub-fields are included for two reasons: 1) To help respondents place their programs in the taxonomy, and 2) To indicate to students using the study the areas of specialization of program faculty. The list of sub-fields is not comprehensive and respondents should not be concerned if their sub-field is not included. They may just leave that question blank. Respondents may indicate more than one sub-field of research specialization.

6. What is the difference between the “Additional” fields and the subfields?
The “Additional fields” should be used by programs that are interdisciplinary or encompass more than one field in the NRC taxonomy. Programs that select additional fields will appear in the database under the additional fields with an asterisk(*) that will indicate all fields listed for the program.

The subfields in the taxonomy have two purposes. First, the subfields may act as a guide to help respondents place their programs in the taxonomy. Second, the sub-fields may be used to indicate a “field of research specialization.” These data will be useful to students who later use the collected data to determine the areas of specialization of faculty in programs of interest.

Please note that the subfield lists are limited in scope and not all subfields within a field are included.

7. May doctoral degrees other than Ph.D.s be included in the study?
Programs that offer doctoral degrees other than Ph.D.s may be included in the study as long as the degree is in a field that falls within the taxonomy.  Please note that Education and Music Performance are not included in the NRC taxonomy and programs in these fields should not be part of the NRC study. For more information on how to list joint MD/PhD programs, please see FAQ # 17 below.

8. Can the Completion tables from the Institutional survey be submitted in a format other than the web?

Yes, Mathematica has a specified file format. Please ask your Mathematica liaison for more information.

9. Which students should be included in the completion tables on the Institutional Questionnaire? All students, including those in programs not included the study, or only students in the programs included in the study?

Only include students in programs included in the study.

10. What do you mean by “full-time doctoral students” in section E? Only those students who are currently enrolled as full-time (as of Fall 2006)? As of when for those who have graduated? What about those on leave of absence, or those who switched from part-time to full-time or vice versa?

By “full-time”, we mean a student who is enrolled full-time throughout his or her time in the doctoral program or, when the question applies to a single year, students who are full-time during that year. If students have formally filed for a leave of absence, they may be counted, but they are not full-time. If a student is not enrolled and has not filed for leave of absence, he or she is considered to have left the program and should not be counted. Students who have switched between full-time and part-time status should be counted as part-time. We are trying to get a measure of time to degree and financial support that would give a potential student an idea of how long it would take to finish if he or she went straight through the program and what kind of funding would be available if enrolled as a full-time student.

Revised 11/20/06

11. Will an “Unknown Ethnicity” or an “International Student” table be added to the Institutional survey?

No. When this question was developed, NRC was primarily interested in comparing the progress of domestic students from different racial/ethnic groups. This is not an overall diversity question. That information is addressed elsewhere.

12. How should Institutions involved in multi-university programs complete their institutional surveys?

All participating institutions will be asked to complete an Institutional Survey, regardless of whether they are involved in a joint or multi-university program or not. The responses to the Institutional Survey should reflect the practices and characteristics of the institution responding only. The numbers in the “completion tables” should include the students enrolled through the institution responding, regardless of whether they are in a joint program or not.

If your institution has a joint doctoral program with another institution, please email NRC-Assessment@mathematica-mpr.com with the names of the institutions involved, the program name, information about how the program is administered (i.e. which of the three “situations” your program falls into), and the name of the individual who will be handling data submission for the joint program.

Situation #1: The degree is always conferred by one institution

If the degree is always conferred by only one of the two institutions, then list the program under the institution which confers the degree. The institution which grants the degree will be responsible for submitting data for the joint program. When filling out the program spreadsheet, please make sure to name the program such that the joint nature of the program and the names of the institutions involved are easily determined. For example, the program could be titled “The joint program in Biomedical Engineering at University X and University Y.” The program will then be listed with an asterisk to indicate the participation of the institution which does not confer the degree.

Situation #2: Only one of the institutions involved in the joint program is participating in the NRC study

The institution which is involved in the joint program and participating in the study will be responsible for submitting data for the joint program. When filling out the program spreadsheet, please make sure to name the program such that the joint nature of the program and the names of the institutions involved are easily determined. For example, the program could be titled “The joint program in Biomedical Engineering at University X and University Y.” The program will then be listed with an asterisk to indicate the participation of the institution which is not involved in the study.

Situation #3: The degree is conferred jointly by both institutions or may be conferred by either institution

If the degree is conferred jointly by both institutions or if the degree may be conferred by either institution, than a new entity will be created consisting of both institution names. For example, if University X and University Y jointly confer a degree, the program will be listed under University X-University Y. In order to prevent duplication of data, the two institutions should work together to make sure that one set of complete data for the joint program is submitted to the NRC. The institutions should decide amongst themselves which institutional coordinator (IC) will be responsible for submitting data on the joint program. This IC should then download an additional program spreadsheet from https://www.NRC-Assessment.com/upload to contain only program names of joint programs between the two institutions. If multiple joint programs exist between the same two institutions, then they may all be entered on the same spreadsheet. When filling out the program spreadsheet for joint programs where both institutions confer the degree, please make sure to include the names of both institutions in the “Institution Name” column and name the program itself such that the joint nature of the program and the names of the institutions involved are easily determined. Prior to submitting the spreadsheet, please make sure to rename it using both institution names.

13. What is a program and what is its use?
A program is a unit of graduate study that is defined by its performance of at least three of the following four activities:

1. Enrolls students in doctoral study

2. Designates its own faculty

3. Develops its own curriculum

4. Recommends students for doctoral degrees.

Most programs will be units to which students wishing to pursue a doctorate may apply. However, in the case of large “umbrella” programs, such as General Biology, the program should be listed under the fields in which it makes recommendations for degrees. In that case, a program questionnaire should be completed for each degree field.

To be included in the study, a program must have produced at least 5 Ph.D.s in the 2001/2 to 2005/6 period.

Not all doctoral programs are included in the study, although institutions are encouraged to name all doctoral programs that fall within the taxonomy.

14. Should multiple programs be combined into one NRC field?
Separate programs should not be combined into one NRC field. The study aims to evaluate programs that grant doctoral degrees to which a student could potentially apply. Instead of combining programs, an institution may name more than one program in a field. For example, if an institution has two Biochemistry programs with one located in the Medical School and one located in the College of Arts and Sciences, both programs should be listed separately in the study. As another example, if an institution has separate History and History of Science programs, both programs may be listed under the History field. The only exception is that under certain conditions some engineering specializations may be aggregated under “General Engineering,” please see FAQ # 25 below for more information.

15. Should one program be split into multiple fields?
One program should not be split into multiple fields within the taxonomy. Instead, additional fields should be selected to reflect the nature of the program. The program will appear in the database under additional fields with an asterisk(*) that will indicate all fields listed for that program.

16. How do I assign a field to an interdisciplinary program?
There is no perfect way to assign interdisciplinary programs to fields. Institutional coordinators must choose a field into which to place the interdisciplinary program. If the program spans more than one field within the taxonomy, the program can be listed under additional fields and will appear in the database under those fields with an asterisk (*) that will indicate all fields listed for the program.

17. How should MD/PhD programs be reported?
Due to the diversity of programs that enroll MD/PhD students, there are two options for reporting them in the time-to-degree numbers and completion tables, either (A) include them in the time-to-degree and completion tables with the PhD students or (B) report their information separately (in the same program questionnaire). For those programs that CAN separate their MD/PhD students, we would prefer that you report their time-to-degree and completion information separately.

We have added additional questions to the program questionnaire (these new questions will only be asked for programs in life science fields) which ask (1) if the program enrolls MD/PhD students, (2) how many were enrolled in Fall 2005, (3) whether the program has ONLY MD/PhD students (in which case separating their time to degree is unnecessary), and (4) whether the program is able to report their information separately from the PhD students. If a program is able to separate the MD/PhD student information, you will get another set of completion tables (C18a/b) for MD/PhD students, in this case, please DO NOT include the MD/PhD students in the Male/Female completion tables (C16-C17). NOTE: We are no longer asking for an entire program survey from the MD/PhD portion of existing PhD programs. If you have already submitted your program lists with the MD/PhD portions listed as a separate program, please notify your MPR liaison so you do not receive two program surveys.

When calculating median time to degree and when filling out the completion tables for MD/PhD students, please do NOT separate out the MD years. This will have the effect of extending the median time to degree for students in the MD/PhD program; that’s what we want to measure. In the completion table, do NOT include MD/PhD students who left prior to beginning the PhD portion (they are only MD students).

Revised 11/20/06.

18. When should the taxonomy field “Integrated Biology/Integrated Biomedical Sciences/Biology” be used?
The taxonomy field “Integrated Biology/Integrated Biomedical Sciences/Biology” should only be used if the program is a broad multidisciplinary program that grants degrees in the undifferentiated field. Programs that are interdisciplinary, but have some degree of specialization should be fit into the taxonomy using additional fields. For example a Microbiology and Immunology program should select both “Microbiology” and “Immunology and Infectious Disease” as fields instead of using the “Integrated Biology” taxonomy option. Since one aim of the study is to compare like programs, the “Integrated Biology” taxonomy should only be used if that is the true nature of the graduate program.

19. What is the purpose and use of data from “emerging fields”?
The emerging fields in most cases do not currently meet the standards of degree production established for the study but are perceived to likely be of interest or importance in the future. They may be free-standing programs or included as research specializations within existing programs. The data collected from free-standing emerging fields will be limited to the number of faculty members associated with the program and the number of doctoral students enrolled in the program. Such data will allow the NRC to obtain some measure of the prevalence and growth of the emerging fields and will assist prospective graduate students in locating such programs. Emerging fields will not be included in the ratings process. If a program in an emerging field is included in an already existing field, it will appear as a “research specialization” for faculty. These programs will have the numbers of faculty with this research specialization reported under the emerging field, but not the number of students who study in the specialization.

20. Can the database behind the web survey be shared with the schools?

The SQL database behind the web survey is not portable.

21. Can a question asking for the URL of the program website be added to the Program survey? This information will be helpful to raters during the rating exercise.

Yes. This will be added to the Program Survey.

22. Can a decimal point be added to the time to degree question to allow for mid-year graduates?

Yes, the web-based questionnaire will accept decimals. For this question, please report full calendar years (or fractions thereof).

23. Can we submit aggregated Race/Ethnicity data on Faculty if we are unable to submit this information on individual faculty members?

Yes. The web-based program survey will ask for demographic information if it is missing from more than 10% of individual faculty members on the Faculty spreadsheet.

24. Are programs in an emerging field exempt from the requirement of having graduated at least 5 PhDs?

No. All programs included in the study need to meet the minimum 5 PhDs requirement.

25. How should specializations within a program in General Engineering be classified?
Research specializations within a general program in Engineering should be classified in the following manner:

- List an engineering specialization as a separate program if it meets all of the criteria that define a doctoral program, that is---it has graduated 5 PhDs within the past 5 years AND it satisfies 3 out of the following 4 rules: 1) Enrolls doctoral students, 2) Has a designated faculty, 3) Develops a curriculum for doctoral study, or 4) Makes recommendations for the award of degrees.

- Engineering specializations that either have not graduated 5 Ph.D.s in the past 5 years or that do not satisfy 3 of the 4 requirements that define a program should be aggregated and reported as one program. The specializations together must have generated 5 Ph.D.s in 5 years. Under the "Program Name" column in the IC spreadsheet please label the program as "Engineering" or "General Engineering" so that it will be clear that this is a broad and multidisciplinary program. The specializations within the broad Engineering program (Chemical Engineering, Mechanical Engineering etc.) should then be identified in the “Field” and “Other Program Fields” columns of the spreadsheet. All of the information in these columns will be listed as specializations within a general Engineering field in the NRC database, regardless of which specialization is listed first. Aggregated General Engineering programs will not be rated.

26. What is the definition of Teaching Assistant (for questions D5-D7 of the program questionnaire)?

There are two separate definitions, depending on the question. In D5-D6, the implicit definition of a teaching assistant is a person who engages in teaching duties, regardless of whether they are compensated for it. In D7, we are only asking about TA's who teach in return for a stipend/remuneration.

27. For the median time to degree calculation (and completion tables), what is the start date: date of first enrollment in PhD program, date of first enrollment in the program (including those first admitted as Master’s students) or date of first enrollment at the university (regardless of program)?

For the median time to degree, please calculate elapsed time (including quarters or semesters not enrolled) since first enrollment in the program in question to completion of PhD requirements. This includes those students first admitted as Master’s students who later switched into the PhD program, regardless of whether they completed the Master’s degree or not. Do not include the years the student was enrolled in any other program (whether at the same or a different university -- the program the student left would report the student as a non-completer).

Similarly for the completion tables, for students who are currently in the PhD program, list them with the cohort when they first enrolled in the program, regardless of whether they first enrolled as master’s students or doctoral students. Do not include those students currently enrolled in the master’s program.

For time to degree for students who were “full-time during their entire time in the program,” please only include those students who were enrolled full-time every semester/quarter from first enrolling through completing. Do not include students who were part-time or on a leave of absence or otherwise not enrolled at any time between first enrolling and completing the PhD requirements.

Revised 11/20/06

28. For the completion tables (C16-C18), should the “Number of students who left without a Master’s Degree” or “after completing a master’s degree” include those students who have master’s degrees from other programs or other universities?

Report only those students who left without completing a master’s degree or completed a master’s IN THE PROGRAM IN QUESTION, regardless if they have a prior master’s degree from another university or from another program at the same university.

Revised 11/20/06

29. Do the questions in Part C of the program questionnaire pertain to the same students as the Institutional Questionnaire (i.e. only cohorts from 1996-2006) or ALL students regardless of when they entered? For example, if a student entered in 1995, and earned a degree in 2001, should they be included in C1?

Yes, include all students who graduated in 2001-2002 in C1 regardless of when they entered the program.

Revised 11/20/06

30. For C9b, can the number of US Citizens and Perm. Residents be reported as one number, if our university doesn’t distinguish between citizens and perm. Residents?

Report them all as US citizens if you are unable to distinguish between US citizens and permanent residents.

Revised 11/20/06

31. When counting the number of postdocs in a program, how do we count a postdoc working for a faculty member who is core in more than one program? Which program should the postdoc be counted in, should they be associated with both programs?

They should be counted in both programs.

Revised 11/20/06

32. When reporting the top four “countries of origin” for postdocs in F3, what do you mean by “country of origin”? Birth country or citizenship country?

In general, we mean birth country. If that information is not available in your data system, use the country from which he or she applied for his or her visa.

Revised 11/20/06

33. If the Modal Value for financial support of first year students requested in questions E2 and E3 is not a single number, what value should be given?

The modal value for a distribution of numbers is defined as the most frequent value. If there is a bimodal distribution or a range of values that is most common, then the response should be the average value that most closely represents the level of support. For example if most of the students receive amounts near $18,000 or $26,000, then the average would be taken over these two groupings and might be a number near $22,000, or if there is a range from $21,000 to $25,000 with a somewhat even distribution of values, then the average would be taken over that range.

Revised 12/05/06

34. Why is it required that the "new faculty" must by tenured or tenure-track, but this is not a criterion for "core faculty"?
One aim of the study is to evaluate faculty who are likely to be involved in graduate education. The dissertation committee restriction accomplishes this goal for faculty established within an institution. The "tenured or tenure-track" restriction is designed to identify newly hired faculty who are likely to be involved in graduate education at a later date. For programs that do not have a formal tenure system, non-tenure track faculty may be listed as new if they are expected to become involved in graduate education in that program (through serving on dissertation committees). Please see the information on New Faculty under “General Clarifications” (above) for more information.

35. What about faculty members with joint appointments?
A faculty member may be listed as a core faculty member in those programs in which he or she is formally designated as faculty. For example, if a faculty member holds a joint appointment in History and Classics and serves on dissertation committees in both fields, she may be listed as a member of the core faculty in both the History program and the Classics program.

36. How should clinical faculty be categorized?
Clinical faculty will usually not be involved in the supervision of dissertations and so, in this case, will not be listed. In the event that they meet the faculty definition, clinical faculty can be considered either core or associated faculty depending on how the individual is appointed. If the faculty member serves on dissertation committees in a program covered by the taxonomy, but is formally designated as faculty in a clinical department (e.g. Internal Medicine) only, then that faculty member would be considered associated faculty. If a clinical faculty member is formally designated as faculty in a program covered by the taxonomy and serves on dissertation committees in that same program, than that person would be considered core faculty.

37. Should emeritus faculty be included in the study?
For the most part, emeritus faculty should not be included in the study. The study aims to collect data from faculty actively involved in graduate education as a means of evaluating and benchmarking program quality. Since the degree to which emeritus faculty are involved in doctoral education varies among institutions, inclusion of emeritus faculty may be appropriate in some cases. Only emeritus faculty who have, within the last three years, either chaired a dissertation committee (i.e. been the PhD supervisor) or served as the primary instructor for a doctoral course listed in the catalog for credit should be included in the study.

38. What is the difference between an “outside reader” and an Associate Faculty member?
An "outside reader" is someone who serves on the dissertation committee but who does not contribute substantively to the development of the dissertation. Some institutions require that an individual from outside the department, or even outside the institution, serve on the dissertation committee primarily to make sure that the dissertation is comprehensible to those outside of the field. Individuals who have not had a significant impact on graduate education in the program should not be included as program faculty.

In contrast, faculty members often serve on dissertation committees outside of their programs because they have specific knowledge that is needed by the student to properly develop the dissertation. This situation may be particularly common in fields where there is more overlap among programs (as in the Life Sciences) or when a faculty member has expert knowledge of a particular technique or research area which can not be found within the program. Faculty who serve a particular role or need on a dissertation committee despite being outside of the program and contribute significantly to the development of the dissertation should be included as associated faculty. Please see the information on Associate Faculty under “General Clarifications” (above) for more information.

39. May faculty who are not employed by the university be considered program faculty?
Faculty who are not employed by the university may still be considered associated faculty for a program as long as they hold some type of university appointment. For example, a faculty member with an adjunct appointment who works in industry or at a national lab may be considered associated faculty if she has served on a dissertation committee in the past 5 years. Adjunct faculty may not be classified as core faculty. Please see “Faculty Classification” under “General Clarifications” (above) for more information.

40. Where should Faculty who have either recently moved or were recently hired be listed?

The Core and New Faculty lists should include only those faculty who are currently (2006-2007) and formally designated as faculty in the program. A faculty member hired in the 2006-2007 academic year should NOT be included. If a faculty member is on sabbatical at another institution or is otherwise formally on leave, they may be listed with their home institution.

41. Which students should be listed on the student lists in the five fields that will receive student questionnaires?

Only those students who are currently enrolled (or have formally filed for a leave of absence) who have advanced to candidacy (i.e. passed the qualifying exams) during or prior to academic year 2005-2006 should be listed. Please note: summer 2006 is NOT included in the 2005-2006 academic year. If the program does not have qualifying exams, please list advanced doctoral students only.

42. How will faculty productivity be allocated for faculty serving in more than one graduate program?
The Committee held a lengthy discussion of how to allocate faculty publications and citations in the case where a faculty member participates in more than one doctoral program. It recognized that, for most cases, faculty members tend to publish in the field of the program(s) in which they are core faculty—and it is in this field (or fields) that they also serve as chair or principal adviser of dissertation committees. The allocation, however, also needs to recognize the efforts of faculty members in fields beyond their own. The allocation formula should have the quality that it encourages the development of faculty lists that accurately include faculty who are truly active in a program and supports listing of truly multidisciplinary faculty. With this in mind, the Committee designed a formula.

a. For faculty members who are core in one or more programs that fall within the NRC taxonomy (regardless of the number programs they may be associated with):

where

A_i is the share of publications and citations allocated to the faculty member in program i;

P_i is the number of committees in program i for which the faculty member serves as chair or principal adviser;

n_i is the number of committees in program i on which the faculty member serves in a capacity other than chair or principal adviser;

d_i is a variable that takes on the value 1 if the faculty member is a core faculty member in program i and is 0 otherwise;

m is the total number of programs where the faculty member is a core faculty member.

b. For faculty members who are core in a program in a non-included field but are listed as associate faculty in an included one:

where P_i and n_i are defined as above.

The factor of 2 in the denominator was included to reduce the over-allocation of associate faculty members when we do not have information on their core programs. The + 5 that was there previously would have become proportionally smaller as these faculty sit on more and more committees outside their core program making the allocation closer to 100%. To remedy this situation, we have multiplied the denominator by 2 to effectively reduce the allocation to a reasonable fraction. With this modification, the allocation for associate faculty members (who are core in a non-included field or program) will never be greater than 50%.

c. New faculty members will have all their publications and citations allocated to their core program(s) since they will not yet have a record of dissertation committee service. New faculty who are listed in more than one program (such as a joint appointment) will have their allocations split evenly among their programs.

These allocations will be calculated directly by Mathematica from the faculty lists. The ICs will have a chance to review and, if necessary, revise these allocations during the data validation period.

Revised 10/16/06. The formula in a. has been clarified only; it has not been changed. The formula in b. has been modified to reduce the over-allocation of associate faculty.

43. Can more information about the purpose of the questions be added to the cover page or section introductions on the Faculty and Student Surveys?

Yes. The Faculty and Student questionnaires have not yet been finalized. Once they are final the rationale for the questions will be updated and added to the survey. We will also include additional information about how the data collected in each question will be used in the ratings.

44. Can the cover pages of the Faculty and Student questionnaires be revised to say that the address information will be destroyed at the end of the study?

Language is currently being drafted to assure respondents of the confidentiality of this information.

45. How will it be decided what faculty information is “publicly available”? Can these items be marked on the Faculty Survey?

The public database posted on the web at the end of the study will not include the names of Faculty associated with the programs. Only aggregate program data will be listed. We will indicate which questions we do not consider to be public and will take care to ensure the confidentiality of that information.

46. Can programmed versions of the Faculty and Student Questionnaires be made available for IRB review?

Mathematica will set up a test version of the Faculty and Student Questionnaires for each of the schools, and send the user information to the IC.

47. Can the cover page be customized to satisfy institutional IRB requirements?

Yes. Mathematica can customize the cover page, if required by your institutional IRB. For more information, please contact your Mathematica liaison.

48. Will respondents be able to edit the information pre-loaded for A2 (faculty programs and committee service) of the Faculty Survey?

Due to the delays related to the Program Questionnaire, this question will be revised as follows: the information pre-loaded into the Program Name field will include a drop-down box of the names of all participating programs from a participating institution. No other information will be pre-loaded. Respondents will be asked to select the Names of the Programs they are involved with, and then indicate the number of dissertation committees chaired, and the number of dissertation committees served on in each of these programs (additional information about which committees should be counted will also be included – i.e. to not include committees where they served as an outside reader). The faculty will not specify whether they are core, new or associated in the programs - that information will be taken from Program Questionnaire. The introduction to this question will also be revised to make it clear that the faculty may not see all the programs they are affiliated with in the drop-down box; only programs participating in the study will be listed.

49. Committee service means different things at different Institutions.

The institutions should use their own judgment when counting dissertation committee service. However, the guidelines on dissertation committee service listed above under “General Clarifications” should be helpful in clarifying what we’re looking for.

50. Does the student’s status matter when counting dissertation committee service? In other words, should we also count dissertation committees for current students and students who have dropped out the program?

The student’s status in the program does not matter as long as a formal dissertation committee has been formed. Mentoring or advising a new graduate student does not count toward dissertation service. Committees of current students and those who have left the program without a degree may be included.

51. If a faculty member has changed institutions, should all of their dissertations committees be counted or only those at the current institution?

Only count dissertation committees for students at the current institution.

52. Will Faculty who are core in more than one program receive multiple questionnaires?

No. Each Faculty member who is listed as core or new in at least one program will receive only one questionnaire.

53. Should the explicit program quality measures on the Faculty Questionnaire be reworded to imply direction?

These items will be kept non-directional. Faculty are only requested to rate the importance of each item.

54. Can additional items be added to the explicit program quality measures?

The NRC accepted comments from the institutions on these measures until October 3^rd. Items were only accepted if they could be related to variables that could be measured in the course of the study,

55. Some of the 21 measures of program quality vary in importance depending on field. How should faculty respond to this section if they are core in more than one program?

These measures should be rated in importance for a general program. It is assumed that Faculty who are affiliated with more than one program will not cross broad fields. For instance, it is very rare for a faculty member to be associated with both a humanities program and a science program. They may respond based on the overall importance in their broad field.

56. If Associate Faculty will not be sent questionnaires, how will information on their publications and citations not available through ISI (particularly in the humanities fields) be collected?

Associate Faculty who also appear as core in another included program will receive a questionnaire. Associate faculty who are not core in any included program will NOT receive a questionnaire. Information on their publications and citations, and honors and awards will still be compiled from other sources to be included in the program data.

The difficulty of collecting publication and citation data on Faculty in the humanities is a much broader problem and is equally difficult for Core and New faculty, as well as Associated. The faculty survey asks for information on articles published in foreign languages as well as books to address this issue. However, we are unable to compile that information on Core and new faculty who choose not to respond to the Faculty questionnaire, as well as associate faculty who may not receive a questionnaire. The NRC is asking for assistance from ICs in communicating the importance of filling out the surveys to their faculty, particularly those in fields underrepresented by ISI.

57. Will you provide more information about how the data needed to evaluate the 21 measures of program quality will be gathered?

Yes. Key variables will be indicated on the Program Questionnaire.

58. How will information on student publications and presentations be collected?

This information will only be collected and reported for students in the five fields receiving the student questionnaire.

59. Will faculty publications be weighted based on quality of the journal?

Publications will not be weighted based on journal quality. There are specialized journals that don’t get on A lists but are important in the field. Also, the choice of what represents an A versus a B journal is sometimes somewhat arbitrary. Weighting for journal articles and citations will be determined through the regression analysis of the rating exercise.

60. How many years of publications and citations will be collected?

In general, publications and citations from the last five years will be collected. Ten years will be collected in humanities fields.

61. How will the publications and citations of new faculty with a joint appointment in two programs be allocated?

They will be split evenly among their programs.

62. Will Faculty grants be allocated to programs according the allocation formula along with publications and citations?

The amount of individual faculty grants will not be collected and allocated. Grant information will be reported by Institution only to give a sense of the resources available at that institution. The percent of Faculty receiving grants and number of students supported with grants will be reported, but without dollar amounts.

63. The Faculty questionnaire asks for the number of grants he or she has served as PI for – do Co-PIs count?

Yes, this question has been modified on the Faculty survey.

64. For the question about the number of students supported on grants – should only students receiving full support be included, or can students receiving only partial support also be included? What does “currently supported” mean – supported this quarter/semester or within the last year, such as over the summer?

Students receiving any amount of support within the last year may be counted.

65. Why are you asking for the CV? How will it be collected?

The Faculty survey asks participants to attach their CV to assist the NRC in verifying the information on publications and citations obtained from ISI. The Faculty will be able to upload their CV as part of the Faculty Survey.

66. Will the questions on the Faculty survey that duplicate information provided on the Program survey be removed?

No. Faculty must independently verify the data that are collected centrally.

67. How will NRC deal with inconsistencies relating to dissertation service as reported on the program questionnaire versus that reported on the faculty questionnaire?

If there are major conflicts, we will contact the IC for assistance in resolving the inconsistencies.

68. Will faculty be allowed to indicate more than two specializations?

The Faculty Survey will allow the respondents to indicate up to six specializations. This question will be modified to have a fill-in part and also include a drop-down box of the fields and subfields in the taxonomy. Faculty will be asked to fill in their specialization and select the field or subfield that best describes their specialization. In this way, we will have an accurate description of the specializations of the Faculty in the program that will also be grouped according to the taxonomy.

Since the Faculty spreadsheet is already live, we cannot change the number of specializations or format of responses. There are two different interpretations of this question on the Faculty lists we have already received. Some programs are listing fields or subfields within the taxonomy, and others are filling in specific specializations. If you have not yet completed this section of the Faculty list, we prefer that you try to list fields or subfields within the taxonomy, if appropriate; if not, you may fill in specific specializations. This question is primarily meant to identify the range of research interests within a program and also to identify niche programs.

69. Can the NRC share the list of awards which will be used on the Faculty Survey?

This question has been removed from the faculty survey and all data on honors and awards will now be compiled independently. A sub-committee is being assembled which will discuss which honors and awards will be gathered in the data collection, and a list of those awards will be posted on the NRC website.

70. Does the NRC have FERPA exemption?

The NRC is in the process of obtaining a blanket exemption form the Department of Education. In the meantime, it is our understanding the institutions, if they choose, may release directory information for those students who have not instructed them otherwise.

70. What grants should be counted in order to answer Question E2?

The grants to be counted should be those that support the faculty member’s own research. Grants for postdocs and traineeships should not be counted.

71. Must the sum of grants in E2a add up to the number of grants in E2?

No. A faculty member may, for example, be partially funded for work in a center in which he or she is neither the PI nor Co-PI on the grant that funds the center.

72. Do the questionnaires have Institutional Review Board (IRB) approval? Where can I obtain information related to IRB approval?
All information related to the IRB approval of the questionnaires is on Questionnaires page of the NRC website. The Faculty and Student Questionnaires for this study have been approved by the Institutional Review Board (IRB) at the National Research Council (NRC). Copies of the introductory material describing the confidential, voluntary, and secure nature of the questionnaires are available in the introduction to the faculty and student questionnaires. All questionnaires are now final.

The NRC has also received IRB approval for the wording that will be used to ensure confidentiality when collecting faculty gender, citizenship, and race/ethnicity information from the institutions. Please click here to view the NRC IRB Approval and the IRB Review Form.

73. What information will the ICs be able to verify during the validation stage?

The ICs will be able to verify all the data collected in the program web questionnaires for their institution that will be included in the public database. For the institutions that are providing the program data by batch process, we assume the data will be verified before it is given to MPR. 

74. Can the ICs review and change the allocation factor used to allocate Faculty productivity to programs?

Yes. The ICs will be able to revise the allocation factor during the validation stage, including reducing a particular Faculty member’s allocation to zero, if it is not appropriate that a portion of their publications and citations should be allocated to that program. Faculty with a zero allocation factor will still appear on the lists of Faculty used in the rating exercise.

75. How will you follow up with Faculty and Students to encourage a high response rate?

MPR will send reminder emails and phone calls. We will report to the ICs the percent of non-respondents in each program and ask for their assistance with encouraging participation. We will also alert the ICs when reminder emails are sent, just in case the emails are tagged by spam filters. It is especially important that we communicate the importance of this study with the students in the five selected fields who will receive student questionnaires, since this is the first and only time that students have been asked to participate in a national study on this subject.

76. Are there any changes to the timeline?

The deadline for submitting faculty spreadsheets has been extended to November 15, but we would greatly appreciate earlier submissions when possible, as this will facilitate their processing.

We hope that this new deadline will give all institutions sufficient time to collect the information requested, including information about committee service, and that this will eliminate the need for processing spreadsheets twice.

77. How will the rating/ranking process be conducted?
Unlike previous NRC studies, this study will not use reputational rating to determine program rankings. Instead, the ranking process will use the quantitative data collected during the study to derive program ratings and rankings.

A two-step process will be used to determine which quantitative measures best estimate program quality and the relative contribution or weight of each variable to program quality. First, an “explicit” questionnaire will ask faculty to rate the importance of a number of quantitative variables in determining overall program quality. This explicit questionnaire will be part of the faculty questionnaire and will be administered across all fields to all faculty. The second step of the rating process involves an “implicit” or anchoring study. In the anchoring study a sample of faculty will be provided with data including faculty lists and other information about a sample of programs in their field. Regression analysis will then be used to determine which quantitative variables most closely predict program ratings. The regression analysis will generate a function with coefficients for each significant quantitative variable which can then be used to provide a predicted rating for a program. The technique is described in greater detail in Appendix G of the Methodology Study. The anchoring study will be performed for each field, since the function which best predicts program ratings may differ by field. Additional analyses may be developed to combine the “explicit” and “implicit” measures.

For more information, please see Ratings and Rankings.

78. Will programs be rated in fields or subfields?

In general, programs will only be ranked in fields and not in subfields or research specializations. Since there over 380 subfields included in the taxonomy, computing separate rankings for each subfield is simply not feasible. We realize that this may have the adverse effect of ranking dissimilar programs. However, please keep in mind that data from the study will be available as a database and users will be able to sort programs by whatever parameters they choose, including research specialty (or subfield). For example, if a potential student is interested specifically in studying the History of Science, she could perform a search among the programs in the field of History for programs where faculty specialize in the History of Science.

The only exception is that programs in the field of “Language, Societies, and Cultures” will be ranked according to research specialty. The reason for this exception is that the field of “Language, Societies, and Cultures” is extremely multidisciplinary and will include very diverse programs, such as Islamic Studies and Italian. For this field only, separate “anchoring” studies will be performed for each research specialty where a sufficient number (≥ 25) of programs is available for comparison and separate rankings for such specialties will be generated. If there are too few programs, quantitative data about the field will be available, but not ratings.

79. Will interdisciplinary programs associated with more than one taxonomy field be rated in more than one field?
Programs will only be rated in the primary field in which they are listed.  "Other fields" that are listed will appear with an asterisk that will refer the user to the primary field.  Thus, multidisciplinary programs will appear with an asterisk in a number of fields, but they will only be rated in the primary field in which they are listed.

80. Will Non-tenure track faculty be excluded or given a lesser weight in the analysis?

No.

81. How will the number of dissertation committees Faculty serve on affect the ratings?

The dissertation committee service will not be used directly in the ratings. The number of dissertation committees will only affect the allocation of faculty productivity (publications and citations), and thus will affect the rating in the way that publications and citations do. It will not be known to what extent publications and citations affect the ratings until after the rating exercise.

82. Will raters be matched to rate programs in their own specialization? How will raters be chosen to rate highly interdisciplinary programs?

The Committee will make a decision on this in January.

83. What are the implicit program quality measures?

The implicit program measures will be determined by a regression analysis on the range of rating. Please see the FAQ on the rating process for more information.