« October 2007 | Main | February 2008 »
The Lithuanian Ministry of Education who participated in the recent P2V meeting in Barcelona has published a webpage with access to key documents regarding the previous policy visit which took place in Vilnius, Lithuania on 18-20 June 2007.
The link is available here: http://korys.emokykla.lt/projects/peer-to-peer-networking-for-valorisation and include links to documents, pictures and additionnaly information about the visit.
Schools are complex organisations. That means that to do them justice through evaluation (be it external-, self- or peer-evaluation) there are many things to take into account. Many inspectorates have adopted a formalised procedure and methodologies to ensure reliable and fair evaluations of schools. Such procedures by the way cannot only be of help when carrying out external evaluations, but could mutatis mutandis also be used when schools carry out self- and peer-evalution (material for a future post maybe). When carrying out external evaluations with inspectors from different countries with a diversity of responsibilities, experience and mandates, having such formalised methologies in place is even more important.
P2V WP6 is a cooperation of 6 inspectorates, each of them evaluating 4 schools in their own country. They use the same evaluation framework during those visits, but 'simply' having agreement on the indicators and criteria is not enough to ensure consistency and reliabilty of observations - when agreeing with 6 European inspectorates on evaluation indicators there was a feeling of shared achievement but also of anticipation: will our understanding of things be as uniform once we enter a school? Earlier experience showed that it is necesarry to develop a number of support materials, not only to help experienced inspectors in getting consistent evaluations but also with the aim of possibly extending the use of the framework to people whose everyday activity is not observing and evaluation. To an untrained eye the original P2P framework in itself leaves too many things open, although some schools and inspectors who used the framework 'stand-alone' reported back with enthusiasm. As part of P2V we decided to enhance the framework by adding a number of supports:
1. an evaluator guidance
2. a self-evaluation questionnaire
3. interview guidance
4. a lesson observation form
5. a template report
Besides these materials, we also improved the framework by adding the possibility to add notes and a column in which scores can be assigned to each indicator. Also the framework now contains clues about the sources that can be used to find evidence about a specific indicator. In another post I will elaborate more in the framework itself.
The evaluator guidance contains all kinds of practical information and helps the local evaluator in setting up the visit and in making appropriate chices before, during and after the visit. This document gives tips about making observations, about documenting evidence and about writing and referencing the school report.
Prior to each visit a self-evaluation questionnaire should be filled in by the school that is going to be visited. This questionnaire should be made available to the evaluators before the visit. It contains questions about hardware, vision and policies. The schools are also requested to send policy briefs etc to the evaluators. This information helps making the visit more efficient (as no time is lost during the visit itself to gather this info) and effective (as the visit is more focussed on themes and hypotheses that were identified based on the materials).
The interview guidances help the evaluators during the school visit. There are three guidances: one for teachers, one for management and one for pupils. As part of the methodology, interviews should be scheduled with each of these groups. The questions are directly derived from the quality indicators and evidence pointers in the framework, thus helping the evalutor in making informed decisions.
An essential part of each school visist are the lesson observations. Each evaluator makes notes on a lesson observation form, on which also appropriate boxes can be checked about the subject, type of ICT used, type of classroom etc.
After the visit the local inspector writes a report. In this report he/she elaborates on the scores, providing strengths and weaknesses for each Quality Area. A template report is provided, with a clear format that is linked to the framework.
To an inspector or someone familiar with inspections, most of the above is not radically new. What is new however that we try to develop this toolkit specifically for ICT, aimed at transnational use and building on the body of knowledge and expertise of inspectors in several countries. During our first trials it proved that sometimes a small adjustment in wording, in the order of criteria or in the guidance was needed to prevent confusion on part of the school or the evaluators. The iterative process of the school visits will mean that by the summer of 2008 we will have a toolkit that has passed its practical test.
The evaluation framework used in the policy strand of P2V is based on a method called RUFDATA described in a paper by Murray Saunders of Lancaster University (1). RUFDATA provides a convenient general framework to plan and develop evaluation within a collaborative project. Two evaluation sweeps have taken place so far.
The evaluation framework focuses on a number of activities to be evaluated and the evaluation in P2V is structured around the following focus points:
1. The shared understanding of the project and the underlying concepts;
2. The understanding and development of the methodology;
3. The process of identifying and formulating the issue in collaboration with other actors in the host country and communicating it to the project partners;
4. Identifying (on the basis of the formulated issue) and formulating (in various forms and from various sources) input on the part of project participants and members of the wider networks;
5. Bringing together the issue and the input through organised peer exchange to design tentative solutions to the host country issue;
6. Transforming the tentative solutions developed in the exchange visit into applicable policy strategies;
7. Interaction between the three strands (policy, schools, inspection) particularly with respect to the methodology, the evaluation and horizontal communication between strands about the work done; and
8. Understanding, developing and using the evaluation framework.
The first sweep of evaluation took place in June 2007 during the Lithuanian peer-exchange visit and was completed by an online questionnaire about key concepts. The evaluation involved only participants in the policy strand and covered focus points 1, 2, and 3 (see above). Amongst the recommendations resulting from the evaluation were a number of improvements to the methodology used and the workings of the project, in particular about reaching a shared understanding of the underlying concept of the project, about the formulation and communication of the issue chosen by the host country to be discussed during the visit and the provision of input from R&D about the issue.
The second evaluation sweep took place in December 2007 during the EMINENT conference and extended coverage to all participants in P2V. It addressed focus point 7: horizontal communication between the policy, schools and inspection strands. It also raised the question of possible synergies between the methods used in the three strands. The evaluation report should be available by the end of January 2008.
(1) Saunders M., Beginning an evaluation with RUFDATA: theorising a practical approach to evaluation planning, In Evaluation Volume 6 no. 1 pp 7-21
The second P2V Project Partners Meeting took place on 7 December 2007 in Brussels. Participants reviewed the first year of the project and discussions covered reports on peer learning visits for policy-makers (to Lithuania), schools (to Austria) and inspectorates (to the Netherlands and Lithuania), future plans and administrative issues.
One of the main issues for policy peer learning was the methodology used in P2V. Alan McCluskey from the University of Fribourg, Switzerland, presented the ‘deep dive’ approach as a methodology which could be applied in the second visit, to Barcelona in February 2008. The deep dive approach builds on small intensive workshops for one and a half day where the participants develop valuable policy proposals. At the end of the meeting policy makers are invited to listen to the propositions. This approach is seen as a good way of bridging the gap between policy shapers and policy makers. Inviting key policy actors will help to communicate across the policy proposals developed during the meeting.
Other items discussed during the meeting were the continuing formation and modification of school triads in the P2V project. An important aspect is the coherence of these triads. There has to be a balance in the e-maturity of schools as this project must be beneficial to all of them. For instance, a school with high level of e-maturity will not gain much if it is in a triad with two low e-maturity level schools. Also partners decided to work on the development of P2V Wiki (http://p2v.wikispaces.com/) and P2V Blog (http://blog.eun.org/p2v/).
The next P2V Project Partners meeting will take place in Brussels during March 2008.
Can you ask inspectors of education from different countries in Europe to walk into a school and evaluate the use of ICT in that school and then expect them to agree on the conclusions? This is the experiment that is carried out in workpackage 6 of the P2V project. Inspectors from Sweden, Scotland, France, Catalonia, Lithuania and The Netherlands will visit some 25 schools and see whether a common evaluation methodology can be adopted to assess the use of ICT in schools.
In the week of 17 September 2007, 11 inspectors flew into Utrecht, The Netherlands to experience using a common ICT evaluation methodology for the first time. This methodolgy was built around the transnational evaluation framework that was developed as part of teh P2P project. In P2V we added to this framework by developing support materials that would help inspectors from different backgrounds to make reliable observations and evalutions.
In this first meeting we visited two schools: de Triangel primary school in Gouda and the Helen Parkhurst Secondary school in Almere.
De Triangel is a multidenominational school in the west of The Netherlands. The school has about 250 pupils in the age range of 4-12. ICT is well integrated and the school has taken steps in the recent months to extend the number of available interactive whiteboards. Although some of the iwb's were installed only a day before the visit of the inspectors and despite the delegation of six inspectors, the teachers and the pupils were quite at ease with the technology. We witnessed a music lesson where internet tools were integrated effectively into the teaching. The visitors were also impressed by the use of the adminsitartion systems to monitor and record pupils progress and to analyse test scores. Pupils had plenty of opportunity to use pcs with a 1:4 computer:pupil ratio and pcs placed not only in the back of all the classrooms but also in the hallways. The school had a very pragmatic approach to the integration of ICT in the school, with an important role for the ICT coordinator who together with the school leader managed to enthusiasmise every team member.
Helen Parkhurst is a school for general secondary education for students aged 12-18. The objectives for 6 (young students) to 12 weeks (older students) are fixed through so called curriculum lines by the teachers in the different subjects. But the order of activities, pace and to some extent the way how to reach the objectives can be determined by the students, in close agreement with the teacher. Many activities are based on collaboration with other students. Reflection on the different activities and results of the learning process are very important. The school has some 1900 students. The ICT curriculum is divided over different subjects. Mediaplus Office is used to train students in the lower forms for the use of a word processor, presentation tools and spreadsheets. In the higher forms the subject Computer Science can be chosen, and also modules from the Cisco Academy. At Helen Parkhurst ICT was especially used by the pupils for independent and group work - which reflects the value the school puts on self-directed learning. We witnesse some very confident and reflective students who were eager to see ICT used more by teachers. HP has about 1 computer for every 5 students, almost all of them connected to broadband. Other facilities are also in place (digcams etc) to support studnets in their learning.
So, did we manage to agree on our verdict at the end of the day? The answer is yes, although in some cases there was disagreement on details. We also agreed that some elements of the methodology needed improvement, but that overall the toolbox was very helpful in making informed decisions that did justice to the school. The schools found the visit useful and sometimes tiring. They recognised the findings. For each school visit, a comprehensive evaluation report was drafted, highlighting both strengths and weaknesses of the schools. Base on the visits a new revised evaluation toolbox wwas developed to be used during the next school visist in Luthuania.