DR. SEBBA: Okay. We do have a short time and I am going to pack in quite a lot which means I am going to speak rather quickly. So, if you have trouble with my accent it is going to make things worse. I understand that these presentations will become available somehow or other electronically to people and so you will be able to follow it up and I will certainly be happy to receive e-mails.

This was prepared with my colleague, Diana Elbourne who is a professor both in the medical statistics area, in fact of health care evaluation as well as working in education in the epicenter which I will tell you more about.

I have been working in government for 6 years I suppose as the custodian of educational research funding which puts me in rather an interesting position given some of the discussion of the last session because I have had to justify to ministers why we spend the money we do on producing results which they don't seem to be always terribly impressed with.

I have now got the position of going into, back into university and expecting a fairly cool reception from my colleagues --

(Laughter.)

DR. SEBBA: -- from my colleagues on some of the issues that I am going to be facing in terms of trying to get them to be more realistic about what government will and won't fund and what other research councils will and won't fund and what the expectations of reporting are.

So, I do feel a bit kind of trapped in between the, if you like, the camps but that also makes me feel very determined to be constructive about how we can all move forward.

The context for what I want to cover is that in 1998, I came in straight after the 1997 election as an adviser to government to construct a research strategy in a department that had been described as a knowledge-free zone.

(Laughter.)

DR. SEBBA: And the confusion that was raised this morning about the relationship between evaluation and research was one which was illustrated by every possible different view on that issue from people who thought that all research was evaluation and anything else was a waste of time right through to people who saw research as being a larger field of which evaluation is just a part.

So, we had every kind of complexity there.

DR. SCHNEIDER: Is that educational research for just England or for everybody?

DR. SEBBA: Right, I need to make it clear. Reference has been made to Britain today but I can only claim to represent England on this because the governments are now devolved and therefore the research strategies in Scotland and Wales are different and is the educational provision. So, this is just England. It is not neither UK nor Britain. I should make that clear and they don't agree with the other countries about everything that is going on but they are very interested in quite a lot of it and there is quite a lot of cooperation between the countries on various initiatives.

The context when I came in then meant that we needed to undertake a review of educational research to see what was happening as a basis for future action and commissioned economists to undertake that review which as you can imagine was very unpopular at the time, but quite deliberately so because of the problem we had about people bringing baggage into that situation and they concluded although it was unpopular at the time I don't think there is too much disagreement now with the general conclusions at the time that the research was having little impact on policy and practice. We have already made reference to that today, that it was characterized by small-scale non-cumulative which has also been made reference to today fragmentation with a lack of coherence across funders and the OECD review in 2002 which was led by Mike Smith whom many of you will know from here and consult was from Switzerland and Norway concluded that some progress had been made but that there were further developments needed, particularly on this issue we looked at today about how the research gets into practice and they did highlight the need for further training in reviewing.

So, a number of initiatives were developed to address these shortcomings and the ones I am going to talk about now are really, there are really three that I am going to talk about.

One if the second bullet point, evidence for policy and practice center to develop systematic reviews and I am only going to be able to cover it in a fairly brief and superficial way, and the other two refer to initiatives in the last bracket which are aimed at better communication of research and one was to, under the auspices of the National Education Research Forum, the top one which is a body of researchers, funders and practitioners.

We ran two journal editors' conferences, national conferences about 18 months apart to look at many of the issues that have been discussed in this seminar and to try to take some action. I have to say that there hasn't been as much action as we would have liked, but there we are.

I think there has been some development and the third one is a web site which certainly doesn't answer the problems but which attempts to recommunicate journal articles to practitioners and policy makers in a more accessible form and I am going to give you some examples of those because it is these initiatives, the rewriting of research journal articles and the evidence center, the epicenter as it is known which have given rise to identifying particularly issues about abstracts and research reporting more generally.

So, the initiatives were aimed at really better use of current evidence and investment in a better future knowledge base and the epicenter was set up in the end of 1999, early 2000 to set up systematic review groups to develop a methodology for undertaking systematic reviews in education and for supporting the groups.

So, the center itself does not mainly carry out reviews although it does do some reviewing. It mainly supports the carrying out of reviews by groups.

In that sense it is a bit like I suppose the UK Cochrane(?) Center or some of the other Cochrane Centers that are around the world which are doing a supportive and developmental role, and since that time I should say that it has set out specifically to develop a way of doing systematic reviews that would take in the full range of methodologies in educational research and this is probably the most controversial issue about it both because the educational researchers are still convinced it is only interested in RCTs and because the people who are working predominantly in RCTs in England, I won't necessarily say internationally, there is some range of opinion on this think it should be working mainly on RCTs. So, there is quite a lot of tension in that system.

There are 16 established review groups, and I have a list here with me of the review groups and the reviews they have undertaken if you would like to see it. You can also get that from the web site. There is some information here about the center. I am not sure there is quite enough for the number of people here because I wasn't sure how many how people were going to be here and also about some of its training initiatives.

There are in addition to the 16 established review groups, there have been on average about four to five a year developed since it has been started. There are nine new groups just starting up which have been funded by the Teacher Training Agency which is kind of related to government but not a government department as such which are specifically looking at aspects of teacher education and there are, also, two review groups collaboratively working between epi and other organizations, one of them with the Colorado State University Education Department on transition. So, there are some interesting developments there.

Ongoing reviews are undertaken directly by the epicenter but not very many of them. They are doing one for example at the moment directly commissioned by the Department for Education and the Treasury on school size, the effects of school size, mainly because thanks to the American literature our ministers had seen ERIC abstracts on school size and gotten terribly heated about the outcomes.

When I went into the ERIC abstracts I discovered they weren't as representative as they might be which was a little bit alarming and some of the reporting is a bit strange, for some of the reasons that Hannah has already outlined, and we therefore felt the need to commission our own review.

By December there are likely to be about 25 published reviews on the web site. In order to undertake this work and particularly because the epi system is taking in all kinds of educational research studies, studies using all types of methodology, it has been necessary to develop a different data extraction system to the existing systems and again I have a copy here. I am sorry it is quite a scruffy copy, but if you are interested and it shows the criteria that are used on the data extraction.

I am sorry, I am making a big assumption here that many of you know about systematic reviewing and know what kinds of stages we go through on the systematic reviewing and there isn't time to go into that. It is not really the focus of this work that I want to talk about here, but there it shows you on this slide some of the areas that are covered in the data extraction, obvious things like study age, research questions, methods, etc., sampling and so on, and each data extraction, each article is reviewed by two reviewers and then a third if there are substantial areas of disagreement.

Again, very controversially a weighting is allocated to each study that is data extracted. Obviously when they do the search there is a huge number of studies that might come up and then the inclusion and exclusion criteria reduce that number of studies quite considerably usually because of research focus rather than because of any other factors because from the abstract and title they are not likely to be able to make these more detailed judgments.

So, then a lot of studies are obtained in full and a smaller number are then data extracted according to their fitness for purpose in terms of the research question. Fitness for purpose in terms of methodology is then a later judgment.

So, judgments will then be made based on completeness of the data use of methods appropriate to the study question, whether the conclusions go beyond the findings. Quite a number of the first drafts of reviews that Diana and I and others looked at, the review groups had strayed from the findings at various points.

I thought that this was just a problem again of educational research until I looked at some of the articles that have been published about the Cochrane reviews and I found that a couple of the issues identified like going beyond findings and also a tendency towards bias towards positive outcomes or problems in other kinds of review as well.

An overall assessment then of the weight of evidence is then given very crudely as high, medium and low to the studies. The other initiative I mentioned involves the rewriting of journal articles by teams of teachers, journalists and then sending these shortened versions of the articles back to the researchers for verification and this web site now has about 25 such articles and it hasn't been running very long and unfortunately the actual web site development, the technical side of it took rather longer than we had hoped, but basically the idea here is to make journal articles more accessible to policy makers and practitioners and I can't emphasize enough the point that was made in the questioning on the last session that policy makers in particular very rarely read anything that is more than half a page and in order to get some of this information across quickly and easily we need to look very much more at these kinds of schemes but still trying to remain true to the research and that is why these articles are verified back with the researchers.

The recent report by the British Commission on Social Sciences in England recommended that researchers be trained in journalism. I am afraid our view was that that was a bit of a long haul as a way of addressing these issues.

We feel that some researchers, some excellent researchers would never make good journalists, and why should they? They haven't chosen to be journalists in the first place and secondly, that there are different skills involved and that in the end it is probably not a good use of the researcher's time to be spending and somebody has already mentioned that today about whose responsibility it is for I think it was Gary that raised the issue in the last issue about whose responsibility it is to communicate the findings and implications of the research and we feel that although we welcome the kind of example that Lynn gave in child development and I believe the same kind of thing is going on in sociological abstracts although somebody else probably would know better than me with respect to writing lay people's abstracts. We do think that it is important that researchers can get on with the job that they are employed to do and that other people can help them and support them to get the stuff out in a more accessible form.

Now, this example, which I doubt -- can you read that in the back? Probably not. If I just read you some of the criteria sections down the left hand side you will see how this relates to the topic. These are the areas. This is an article about the effects of thinking skills program, cognitive acceleration program on year one pupils. So, it would be 5 to 6 year olds and it says down the left hand side, "What is cognitive intervention all about? How was the research designed? What were the main components of the program? What were the characteristic features of the activity used in the project? What happened in the classroom activity? How did the pupils benefit? What happened? How were the teachers prepared for the intervention? Which tests were used on the pupils, implications; where can I find out more?

It is possible for people coming into these web sites to search on any of those aspects. So, they can search just on methods, on studies that have been done in a particular way or they can search just on the content of programs or they can search just on something else

Now, we believe that for busy teachers coming to a situation where they might have a staff meeting this afternoon about a science, the science curriculum and they need to know; they need some undated information, this combined with what they can get on the web epi site might just contribute, and I am not saying it will answer the problems.

So, what have we learned? You need to remember that in order to undertake these summaries the same processes have to be gone through as with the epi reviews. They have to look, search electronically, do hand searching and use titles and abstracts in order to get the information and both systems have suffered from the very poor in particular quality of abstracts, in some cases very misleading titles.

Okay, but first of all from the reviews we have learned that something about the quality of the research is being judged. Very few were judged to provide high overall weight of evidence for the review. Now, it might be because they don't address the question of the review. It is important to say that is not only to do with the quality of research.

Few provided high quality based on the quality of the study itself but of course we don't know whether that is to do with the research design or the reporting and we feel that the process itself, and we have discussed this a lot this morning about how to train people, we think that the researchers and the, I should say the review groups are made up researchers, practitioners and policy makers, but the researchers on the review groups have reported back that despite the pain and they feel that there wouldn't have been the gain without the pain they have gained an enormous amount of personal development in research through undertaking reviews. They describe it as the best staff development they have ever had and they said that will never approach another research design themselves, a piece of research or report research again without thinking about what they have learned from this painful data extraction process and using the abstracts that have been woefully inadequate.

From the work of the research summaries on the web and this article describes that work, there has been a concern that electronic abstracts are less useful than the actual hard copies. That meant, the systems known as TRIPS(?) that many of the journals used don't provide the abstract electronically. We have already talked about ERIC. So, I am not going to into this now, but of course, in ERIC the author's abstract is reduced and therefore some of the crucial information is missing.

We have, also, talked already about inadequacies of databases. So, I will leave that out but there is an example here of how titles can be misleading and in an effort to be appealing authors sometimes make up titles which are very catchy and interesting and don't actually tell you a great deal about what it is about which if you are doing reviews of this type is very problematic, but the abstracts are often indicative rather than informative. They frequently do not include information about samples, characteristics of samples or the context and this all makes reviewing extremely expensive and laborious because people are having to read probably three or four times as many articles as they would have to read if they had proper abstracts.

I am going to skip this because now is a good time to just finish up. I picked out some examples and I color coded here just very roughly which unfortunately if you get a hard copy won't help you much, but if you go on to the web site you will be able to see this. The blue is supposed to be background information because one of the complaints was that the abstract contained sometimes too much background information and it is more of an introduction to the area than a summary of the study. Red was supposed to be the purpose. Green was the sample, pink methods and analysis, brown findings and implications. This particular article not only didn't give any information about the numbers of schools or the numbers of learners in this study and this is a journal of which I am on the editorial board; so, I can't blame anybody else. It didn't give that information in the actual article which is even more worrying.

So, when I wrote to my editor and pointed this out, they said, "It is time you rewrote our guidelines for us."

Okay, I then went to education evaluation and policy analysis, two quite different studies here, different examples, one of which I think has a lot more information in it than the other, this second one. This one doesn't give you very much detail either about the sample, again, but this one you can see immediately much more information specifically about the sample characteristics, low income, racial minority children and so on which I think a lot of people doing searches do need that information. Okay, we don't have time for this. I am going to just finish with the recommendations. There is just 1 minute's worth. Okay?

The first thing we talked about today concerns training and reviewing and we feel that the systematic review work is just one of several channels but a very key one for helping people to train in writing and abstracting because we think people who go through it are never the same again and that is what they themselves say.

There is a whole issue about giving editors and peer reviewers which has already come up today time for their own development. There is the need to better recognize the time and efforts that go into editing and reviewing in the systems.

Now, your system analysis is different and ours is not so much, you have talked a lot today about promotion. Ours is more about the way in which external assessment of research occurs and how the subsequent funding operates. I think there is good evidence now in the literature for the use of structured abstracts. There are lots of articles by James Hartley who is actually a professor of psychology at Q(?) University and showing preference for structured abstracts, better information and the fact that they provide more standardized information which helps search and retrieval systems, and I don't really understand why we haven't got cracking on these in education, and I would argue that this seminar is an opportunity to think about developing international standards rather than just US standards and here are some examples of systems which we could develop further in order to do that. So, the HAYNES one is the medical one actually but it is not that we don't have the criteria, it is that the criteria are not very well developed.

So, it seems to me if we are going to produce abstracts which are readable, well-organized, brief and self-contained but still have the information in them that we need, we need to get cracking on how we can succeed in this balance. I am sorry, I thought you were going to get a copy of this. So, I put all the references in it, but you will get it soon.