Feedback on the draft summary report

Feedback on the draft summary report

• 1.
  Feedback on thedraft summary report Bea Mahieu - Technopolis Group www.metodika.reformy-msmt.cz
• 2.
  Many topics covered •The risk of peer review • FTE researchers • The self-assessment • Attention for applied research • The use of Research Units • The periodicity of the evaluation • Co-publications • Not only English • An RO-type funding pot for Academy and HEI together • The basis for the funding budget • Publication of the PRFS weights • Etc … 1
• 3.
  Opinions are divided…. • The overall quality of the reports • „Reports are well-written [...]; the suggested changes and evaluation strategies are also generally well-defined and justified.“ • „On the whole, the final report as well as the two previous reports, is very general. Large parts of the report can be universally used for any costomer.“ • Attention for applied research • „We are very glad that the authors from Technopolis Group accepted some of our comments regarding the applied research and applied outputs. We agree with the basic principles of the new methodology, i.e. [...] and also more balance between outputs from basic and applied research“ • „In spite of many criticism from various research organizations, the IPN Methodika leaves out such types of scientific results that are crucial for applied research and its industrial exploitation “ 2
• 4.
  At times contrasting •The self-assessment / SWOT • “Strengths of the methodology: assessment of future developments planned by the RO based on the self-evaluation report” • “Too much weight will be given to self-assessment which is a totally wrong approach. Request of SWOT analysis is not justified and is clearly unnecessary.” • The importance of the ‘research environment’ • “Strengths of the methodology: the focus on the evaluation of the faculty (EvU), a very important element for the assessment of performance of the relevant units” • “Again, we ask for the cancellation of the assessment criterion ‘Research Environment’. The subcriteria and indicators such as HR management and the management in general [….] make no sense.” • PhD students • Ph.D. students should not be counted as researchers if the evaluation methodology is to be used, as it will be, across the divide Universities / Academy of Sciences and other ROs. • You propose to exclude not ony PhD students but also temporary workers. Why? They also cost money. 3
• 5.
  And at timesincorrect • In some cases, SCOPUS database has a higher explanatory value than WoS. There is no reason to prefer WoS to SCOPUS and ERIH databases. We don’t • Enhancing the RD&I IS as suggested will bring additional costs that are not counted. They are counted • The amount of money obtained by research activities, such as contract and collaboration research and various research projects, is also an important indicator of research quality and MUST be used in the evaluation of ROs. It is used • The number of paper citations and patent licences is an important indicator of their scientific merit. These indicators are completely missing in the proposed Methodology. See the bibliometric indicators • Technopolis Group tries to implement the R&D evaluation system working in the United Kingdom in the Czech Republic. We didn’t 4
• 6.
  Panel review “An independentand quality international peer review is unrealistic in the Czech Republic” Risk mitigation measures: •Informed peer review •Conflict of interest statements scrutinised by Main panel Chair & Evaluation Management Team •Ongoing ‘auditing’ function by the Main panel •100% international experts •High requirements for expertise •Realistic day rates for the expertise required •Work load at an acceptable level (max. 5 days in the CR / 20 days in total) •Main panel members and Specialist advisors for information on the national context 5
• 7.
  Applied research “Many appliedresearch results are missing as eligible results. Why?” •A broader range of research outputs for the assessment of Research Productivity would be appropriate • Condition: improve definitions for applied research outputs (especially software) •No direct link between output data and the quality level score for research performance • Research output = research productivity + publication profile + value for the advancement of R&D • Competitiveness in research = external funding + reputation (attractiveness for PhD students) + national and international competitive positioning •Quality of applied research outputs = illustrated through take up for further R&D or by the users 6
• 8.
  FTE researchers “It ishard to divide the duties related to teaching and to research” “The rules will never be exact as in many cases these duties are very closely mixed with the teaching duties.” Is it really that impossible? Two approaches in international practice: •Contracts with the university researchers that specify the share of teaching versus research – e.g. UK, Belgium •Consensus in the community – e.g. Norway, Italy: 50% teaching, 50% research 7
• 9.
  Evaluated Unit &Research Unit Assessment of the research environment Information asked at Research Unit level The more pronounced use of self-assessments and the SWOT analysis •The institutional environment is a key dimension for the assessment of research performance •The evaluation has a double function: acts as a PRFS and provides strategic information – for national policy makers and the institutional management •EvUs can include fields with very different field characteristics and dynamics  the field dimension is important for the assessment of and feedback to the institutional management •The value is in the self-assessment  not a sterile reproduction of official papers 8
• 10.
  A big burden?A big bang? Not really … if you start the learning process Now 9
• 11.
  Thank you foryour attention! www.metodika.reformy-msmt.cz