Internal Evaluation Report of the Accreditation Comission for the Year 2009

Last Updated on Saturday, 30 November 2013 17:37
Written by Dana Mosazná

Internal Evaluation Report of the Accreditation Commission, 2009


 

Report on the Implementation of Recommendations Formulated in Internal Evaluation Reports of the Accreditation Commission for the Years 2007 and 2008

 

 

 

November 2009

Introduction

Standards and Guidelines for Quality Assurance in the European Higher Education Area (in the following Standards and Guidelines) have instituted as an integral part of the system for quality assurance a cyclical evaluation of accreditation agencies (evaluating the evaluators). The evaluation is twofold – external evaluation and internal evaluation. The importance of internal evaluation lies in the fact the accreditation agency develops its own internal mechanisms for tracking and evaluating quality for its own activities. Internal evaluation should however at the same time act as basis for external evaluation; therefore within a cycle of 3-5 years a complex self-evaluation report should be created. The data presented therein is then evaluated by an independent panel formed to perform external evaluation. As opposed to the above, internal evaluation takes place annually for the purposes of conducting regular analyses of current issues and reflecting on recommendations from previous internal evaluation reports by the Accreditation Commission.

This year is of specific importance in terms of evaluation, since it is the year of the first external evaluation procedure; the Accreditation Commission prepared a detailed self-evaluation report as background particulars for this evaluation. The cyclical internal evaluation concentrates above all on how and to what extent the previously determined faults and risks have been remedied and how the recommendations stated in the previous internal evaluation reports of the Accreditation Commission have been reflected (follow-up report).

The preparation of the internal evaluation report was influenced by the retirement of prof. Milan Sojka, one of the creators of the internal evaluation system of the Accreditation Commission and an author of the internal evaluation reports for 2007 and 2008.

 

Basis for evaluation of the current state of affairs

The internal evaluation reports of 2008 and especially 2007 have determined positive and negative aspects of the Accreditation Commission activities. The positive aspects include the following: 1) All procedures and results of activities of the Accreditation Commission are in accordance with its mission and goals in terms of quality assurance; 2) The Accreditation Commission has in place a well functioning mechanism for defence against conflicts of interest; its decisions and recommendations are consistent and independent of interest group and government institution pressures; 3) The Accreditation Commission has in place a well functioning internal system for quality assurance for its activities. An integral part of this system is internal feedback (mediation of opinions of Accreditation Commission members and permanent working groups as well as workers of the secretariat and the academia).

The evaluation also identified the problematic areas for the Accreditation Commission in terms of the implementation of Standards and Guidelines and best practice examples from higher education quality assurance agencies abroad. In connection with determining these lacks, recommendations for removal of problems and quality improvement were made. Already in 2008 a majority of these problems were removed. This fact was then stated in the internal evaluation report for the following year. Many of the described problems and risks could however not be removed operationally because they either represent conceptual issues and as such cannot be altered without the consent and good will of institutions determining Czech higher education policies (for instance many changes can only be implemented via legislative procedure), or their removal requires a longer period of time.

The internal evaluation reports from the previous years have put forward six major recommendations to improve the quality of Accreditation Commission activities:

  1. The Accreditation Commission should strive to improve its own effectiveness and create room for discussion of conceptual issues.
  2. There is need for discussions to further specify and refine the criteria applied above all for master and doctoral study programmes and for applications to grant accreditation for habilitation procedures and procedures for the appointment of professors.
  3. It is imperative to improve mutual communication between the Accreditation Commission and working groups. There is a need to further consolidate methodologies for evaluating applications and institutions in the individual working groups and special working groups. In this respect, greater use of websites is recommended as well as more direct contact with the working groups.
  4. General awareness about the activities of the Accreditation Commission needs to be raised among higher education institutions and the general public.
  5. The Accreditation Commission should aim for greater inclusion of students and external experts in working groups and special evaluation working groups.
  6. At the Ministry of Education, Youth and Sports a request for Accreditation Commission budget increase needs to be issued in connection to i) new tasks arising from the fact the Czech Republic participates in the Bologna Process; and ii) better provisioning of the secretariat of the Accreditation Commission (in terms of staff and materials).

For the purposes of compliance with the above cited recommendations the Accreditation Commission has adopted the following measures in 2009:

 

Improving effectiveness and creating room for discussion of conceptual issues

a) basic effectiveness improvements: moving from accrediting study programmes to accrediting institutions

The internal evaluation report of 2007 stressed that the Accreditation Commission had been so far mainly concentrating on accreditations and evaluation of accreditation activities. As a result of the accreditation and accreditation validity prolongation agenda overload, no space remained for strategic outlooks and discussions of conceptual facts. Since 2002 (by which time the transitional period set down by Czech higher education legislation had ended and all study programmes needed to have obtained accreditations), the Accreditation Commission has been placing more stress on evaluation; although due to the high number of accreditation applications and accreditation prolongation applications the Accreditation Commission members as well as members of a majority of the working groups are constantly overloaded. In this respect the report states that the accreditation and evaluation outputs are mostly of good quality and correspond to the set criteria. The accreditation process is thoroughly based on evaluation for which the principal criterion is the application of minimum standards.

The transition from programme accreditations to institutional accreditations was initiated in 2008 when study programme prolongation decisions began to be bound with evaluation of institutions. An example of best practices is the evaluation of the Jan Evangelista Purkyně University in Ústí nad Labem (the evaluation report was discussed at the Accreditation Commission meeting no. 5/2008, November 2008). Based on a thorough evaluation of the activities at the university the Accreditation Commission expressed its consent with a prolongation of accreditation for all bachelor and master study programmes. The accreditation term of validity for all the study programmes was thus unified and in the future the university will apply for prolongation of its accreditations for all programmes at once. Therefore the Accreditation Commission will be able to evaluate the to-be-accredited activities in the overall context of the institution; at the same time there will be a substantial decrease in the administrative workload.

The example of the evaluation of the Jan Evangelista Purkyně University in Ústí nad Labem presents a possibility for improving the effectiveness of the Accreditation Commission. Currently, similar links between evaluation and accreditation are being implemented for almost all evaluated institutions. However due to the high number of higher education institutions, the number of accredited study programmes and the current legislation, this new model currently cannot fully replace the older model of prolonging accreditation validity for individual study programmes.

A systematic transition from accrediting study programmes to accrediting institutions is impossible without a change in the Czech legislation on higher education institutions. The Accreditation Commission therefore expressed its support for a change in the accreditation system as a part of a tertiary education reform. However unlike some authors of the working version of the law on tertiary education the Accreditation Commission believes that programme accreditation should be partially preserved even after the transition to institutional accreditations is made, mainly for the purposes of accrediting new study programmes. Accreditation validity prolongations would then take the form of a cyclical evaluation of the given institution. This system however would require the higher education institutions to carry out internal evaluation based on specific external criteria.

The possible ways of constructing a functional system for internal quality evaluation at Czech higher education institutions could be outlined in a project of the Ministry of Education, Youth and Sports (within the operational programme “Education for Competitiveness”). However this programme has not yet been initiated.

 

b) applications are to be submitted and processed electronically

A solution incorporating a revised version of the public notice regarding the contents of an application for accreditation of study programmes has proven impossible already in 2008. We can therefore expect new legislation only once the new act on tertiary education is ready.

During the course of 2009 the Accreditation Commission has asked the Ministry of Education, Youth and Sports to create modern software applications to provide the Accreditation Commission with all the necessary administrative functionalities. Such applications should replace the current out-of-date database system (Paradox system) and the system for distribution of accreditation applications and other electronic documents (via FTP). It is desirable for the new application to be able to fill-in the necessary documents directly in the interface and submit electronic applications. This should modernize and streamline application receipt, recording and distribution as well as improve effectiveness of the entire Accreditation Commission administration.

 

Summary of adopted measures:

 

Discussing the criteria used by the Accreditation Commission

It can be stated that in 2009, the Accreditation Commission has made big improvements in this area. Already in the previous periods, the following documents were created with the aim to provide guidelines for evaluators of applications for accreditation of study programmes as well as for higher education institutions when preparing applications for accreditation: “Příručka pro posuzovatele žádosti o akreditaci” and “Formulář pro posuzovatele žádosti o akreditaci studijního programu” (“Handbook for accreditation application evaluation”, “Form for evaluating an application for accreditation of study programmes”). Due to requirements as to the transparency of the accreditation process, both these documents are available on the website.

In all meetings of the Accreditation Commission there was enough room for discussing the criteria for accrediting bachelor, master and doctoral study programmes, for accrediting habilitation procedures and procedures for the appointment of professors and discussing the criteria applicable to the evaluation of higher education institutions and their accredited activities. The outcome of these discussions (which were also very important for the purposes of comparing and clarifying criteria applicable across the individual scientific and subject-specific areas) was a new definition for the study programme standards of the Accreditation Commission (ratified at the Accreditation Commission meeting no. 4/2009, September 2009). These standards underline the general requirements on the study programme content, requirements as to staff, information and material provisions and requirements pertaining to the specific research or artistic activities. By defining and publishing these requirements the transparency of evaluation of accredited programmes was improved.

Due to higher education quality being a continuous process, the criteria cannot be at any point considered final and definitive; they must rather be subject to constant discussions.

 

Summary of adopted measures:

 

Improving mutual communication between the Accreditation Commission and the working groups

The aspect of communication between the Accreditation Commission and the working groups is closely tied with the previously discussed issue. The report of 2007 has stated that working groups in their activity strive for maximum transparency of the criteria and procedures applied. Working groups apply the standards set down by the Accreditation Commission to individual study programmes (the question arises as to the level of application of the criteria with respect to the nature of the study programme or scientific field) and Accreditation Commission criteria are mostly upheld in the activities of working groups. However the main problem remains in the fact that the criteria were not interpreted identically at the working group level in the past. Since its discovery in 2007 this problem has been among the top problems on the Accreditation Commission agenda. Another important point under discussion is the working group activity in general and the role of the working groups in the process of evaluation of applications for accreditation and in the argumentation of (i.e. the reasoning process behind) negative standpoints. Here the need is stressed for the argumentation to be sufficiently convincing, to build on specific facts and to eschew misunderstandings and false interpretation by the applicant. At the same time the role of working groups as Accreditation Commission advisory bodies is underlined (i.e. the working groups are not independent decision-making bodies).

In 2008 further tasks were set out for closer cooperation with working groups chaired by persons from outside the Accreditation Commission, for setting rules for the term of office of working group Chairs, for revising and as necessary amending the rules of procedure for working groups, for unification of administrative procedures for all working groups (procedures for handing-off meeting minutes) and for determining a procedure for providing working groups with better administrative support.

A very positive outcome was attained especially with respect to improving effectiveness of working groups chaired by persons from outside the Accreditation Commission. Two working groups with partial workload were united; one of these groups showed problems in communication with the Accreditation Commission. Apart from that the Chairs of working groups regularly attend Accreditation Commission meetings and are involved in discussions as to criteria and standards. New administrative rules were set for processing and handing-off minutes and other materials from working group meetings. For the future, the task remains to revise and amend the rules of procedure for working groups and determine a procedure for providing better administrative support.

Discussions on criteria taking place between the Accreditation Commission and working groups and among the working groups themselves must be a continuous process.

 

Summary of adopted measures:

 

Improving awareness of higher education institutions and the general public regarding Accreditation Commission activities

The report of 2007 pointed out the problems in communication between the Accreditation Commission and the general public and determined there was a need to look for channels to better inform the general public about the mission and the activities of the Accreditation Commission and its working groups. In this respect more use needs to be put to the Accreditation Commission website and the quality of the English version of the website needs to improve.

During the subsequent year the website was improved (however the improvements were not exhaustive, as the report of 2008 points out). It has become apparent the current version of the website has its limitations set by, among other things, the fact the Accreditation Commission website is a part of the website of the Ministry of Education, Youth and Sports. This fact complicates user access; the Accreditation Commission must also adhere to the Ministry guidelines for releasing documents for public viewing. In the given situation the only feasible solutions is a separation of the Accreditation Commission website into a unique domain. That should help the Accreditation Commission to be perceived by the general public as an independent institution. A solution for the separate website for the Accreditation Commission should be a top-priority part of the above mentioned new software application request.

Generally it can be said that throughout 2009 the general awareness of the public about the role of the Accreditation Commission has been greatly improved. This occurred mainly due to the media coverage of the several problems in Czech higher education that came to light throughout the year and that the Accreditation Commission was resolving. In the future it is however equally important for the Accreditation Commission to provide more information about its activity itself, for instance by creating written documents introducing the activities of the Accreditation Commission in detail as well as describing the Commission intentions in terms of upholding quality in Czech higher education. In the context of introducing the Accreditation Commission abroad, these documents will need to be translated. Special attention must be given to the English version of the website.

 

Summary of adopted measures:

 

Inclusion of students and external experts in working groups

The report of 2007 describes the composition of the Accreditation Commission as relatively compliant in terms of expert presence and programme representation. Improvements are required in terms of representation of external and international experts specifically from the Slovak Republic and countries with modern systems for assuring quality in higher education. This problem is being resolved by a cyclical exchange of Accreditation Commission members as per legal requirements. The first changes in this respect occurred in September 2008 when the Accreditation Commission acquired a new expert member who is at the same time a member of the Accreditation Commission of the Slovak Republic.

A positive turn can be seen in a greater inclusion of students into the activities of the Accreditation Commission via their participation in evaluation of higher education institutions and the accredited programmes. In this respect the Accreditation Commission has initiated cooperation with the student chamber of the Council of Higher Education Institutions of the Czech Republic which nominates its student representatives into special working groups. Inclusion of external experts has not been very successful so far due to the lack of motivation on their part. It is rather a question for discussion to determine the specific study programmes or areas where the aid of external experts would be an asset. It would definitely be an improvement to increase the representation of foreign experts in working groups (specifically experts experienced in quality assurance processes outside of the Czech Republic). Currently such experts are only represented in certain working groups. In addition to that they could also be present in the process of evaluating higher education institutions (i.e. they could participate in special working groups). There are several obstacles to such participation: the language barrier (with a majority of foreign experts there is the need to interpret everything into English or to conduct the entire evaluation process in English), and insufficient funding to be able to adequately remunerate the work and the time the experts dedicate to any such activity.

 

Summary of adopted measures:

 

Issuing a request for Accreditation Commission budget increase and for better provisioning of the Accreditation Commission secretariat

The 2007 report stated the secretariat of the Accreditation Commission was working very well given the demanding work it carries out. However a strong lack in terms of financing and staff was reported. The report supported this claim with reference to the increase in requirements for processing of evaluator documentation, higher demands for language competences due to greater cooperation in international panels (ENQA, CEEN, ECA, etc.) and higher demands on the internationalization of Accreditation Commission activities. For these reasons the report expressed the requirement to further support the secretariat in terms of financing and staff.

Currently there is a clear and urgent need to hire a new worker to the secretariat (ideally a lawyer) to handle the evaluation activities of the higher education institution administration and the quality of internal guidelines at higher education institutions for those institutions not subject to registration at the ministry in connection with fulfilling the educational and creative activities (for instance the links to the recognition of lifelong learning study programmes when lifelong learning certificate holder is accepted into an accredited study programme). This worker would also be in charge of handling the increasing amount of accreditation agenda fully subject to the Code of Administrative Procedures.

The Accreditation Commission also appealed to the Ministry of Education, Youth and Sports to determine a method of providing for the external administrative workers who run the Accreditation Commission working groups. Due to the high number of permanent working groups of the Accreditation Commission, their workload and distant places of residence of the Chairs of some of those working groups (i.e. those working groups conduct their meetings outside of Prague) the secretariat of the Accreditation Commission cannot handle all of this agenda. In the past such activities were provided for by using administrative workers from higher education institutions (usually an administrative worker from the university department of the working group Chair) or doctoral study programme students. However this system of provisioning is not sustainable for on a longer time scale (without the possibility to remunerate the workers concerned). At the same time strengthening the secretariat of the Accreditation Commission with workers in charge of the working group agenda would be ineffective because this agenda is dependant on the volume of applications under discussion. Therefore, a system must be found for supporting external administration for the working groups.

Similarly, an effective process must be implemented to enable the Accreditation Commission to maximize inclusion of foreign experts especially in the processes of evaluating higher education institutions and accredited activities (including a proper system for covering travel expenditures and remuneration).

In the recent times (autumn 2009), the Accreditation Commission has started to feel strong support on behalf of the Ministry of Education, Youth and Sports, and the Board for Science, Education, Culture, Youth and Sports of the Parliament of the Czech Republic (Výbor pro vědu, vzdělání, kulturu, mládež a tělovýchovu Poslanecké sněmovny Parlamentu ČR) which in a decision at the 36th meeting, October 22nd 2009, asked the Ministry of Education, Youth and Sports to provide the Accreditation Commission with financial funding to cover administration costs and external experts costs (section 4 of the decision).

The report of 2008 stated that the independence and professionalization of the Accreditation Commission could be greatly strengthened by setting its budget to be a percentage of the annual budget for public higher education institutions. Passing such a budget rule would help protect the Accreditation Commission from political pressures and pressures from various interest groups. It would also create suitable conditions for the professionalization of the Accreditation Commission. However this request is at the present time rather a mere consideration which might be taken into account when setting the future system for Accreditation Commission funding. It is apparent that any greater changes will be implementable only once the new higher education legislation is in effect.

 

Summary of adopted measures:

Přílohy:
Download this file (CZ_vnitrni_hodnoceni_AK_2009.pdf)CZ - Vnitrni hodnoceni AK 2009[ ]174 kB
Download this file (EN_Internal_ Evaluation_Report_AC_2009.pdf)EN - Internal Evaluation Report AC 2009[ ]128 kB