Draft of the Second Interim Report

Draft version for public consultation, 150120
R&D Evaluation Methodology and
Funding Principles
Second Interim Report: The Funding Principles

R&D Evaluation Methodology and Funding Principles
Second Interim Report: The Funding Principles
January, 2015
Erik Arnold, Barbara Good, Bea Mahieu, Fritz Ohler, Brigitte Tiefenthaler – Technopolis Group
Vladislav Čadil, Michal Pazour, Ondřej Pecha, Ondřej Pokorný, Tomáš Ratinger – Technology Centre
ASCR

Second Interim Report – Draft version for public consultation
R&D Evaluation Methodology and Funding Principles i
Summary
Introduction
This report is the draft version of the Second Interim Report of a study developing an
evaluation methodology and institutional funding principles for the R&D system in the
Czech Republic. The report describes the new principles for the institutional funding
of research organisations (RO) in the Czech Republic.
International context
Internationally, performance-based research funding systems (PRFS) operate within a
hierarchical system of governance, in which there are layers of authority from
government through ministries, their agencies and down to the research-performing
organisations. Policymaking tends to be quite strictly separated from policy
implementation. Most countries have difficulty in coordinating aspects of research
and innovation policy across ministries but this is to a degree countered by having
analytical resources decentralised across various organisations. The Czech Republic
has an opportunity to coordinate well, through the R&D&I Council.
Research organisations tend to be of five types: universities; scientific research
institutes; research and technology organisations (RTOs) supporting industrial
innovation; public service research organisations (or ‘government labs’); and national
resources or infrastructures such as libraries and museums. These are steered and
funded through various combinations of unconditional block funding and
performance-based funding, which may be based upon a performance contract and/or
a system that counts or assess results of research. The Czech Republic is unique in
attempting to use a single performance-based system to address all the different types
of research organisation. It is also unusual in being among the minority of countries
with a high ratio of project-based to institutional funding of research.
Internationally, the use of PRFS is believed to improve both the amount and the
quality of the output from research organisations. Quite small performance-based
adjustments to institutional funding lead to large changes in the behaviour of
individual researchers and of the research system as a whole. PRFS can be tuned to
reach different policy objectives, for example to concentrate research in a small
number of institutions or to boost overall research capacity across the system.
However, they also promote ‘gaming’ and there is evidence that they favour traditional
and mainstream research approaches over unorthodox and interdisciplinary ones.
The Current Czech Institutional Funding System
The current Czech institutional funding system is rather decentralised with no official
central co-ordination body. In fact, the RD&I Council, which is an advisory body of the
Government without real executive power, plays this role. Currently, seven ministries
and the Academy of Sciences provide institutional funding. Institutional funding
distribution among providers is based rather on political negotiations and decisions
than the research organisations evaluation results or to previous commitments for the
funding of on-going RD&I activities.
Laws guarantee research organisations a relatively high degree of autonomy in terms
of institutional funding allocation and distribution. The research organisations are not
able fully to utilise their autonomy and except for the Academy of Sciences have not
developed a specific evaluation methodology and strategies for institutional funding
distribution.
The system for allocation of institutional funding among research organisations based
on the rule of three according to evaluation results (the number of RIV-points) does

ii R&D Evaluation Methodology and Funding Principles
not reflect differences in the role (mission) of research organisations and types of
research activities.
The recipients of institutional funding for research form a large and heterogeneous
group. The increasing number and wide spectrum of the research organisations
supported risks posing a problem for further fragmentation of institutional funding in
the future.
Based on the comparison of institutional funding systems in international practice
with the Czech institutional funding system, we can draw the following key
conclusions that should be taken into account while drafting revised funding
principles for the Czech Republic:
• Admitting research organisations to the institutional funding system is a policy
decision and as such should be taken by policy makers at the highest level in the
R&D system. An evaluation methodology or institutional funding principles
cannot substitute for or replace the policy decision.
• The Evaluation Methodology and the funding principles should be designed taking
into account the missions of the research organisations in the RD&I system and
avoid competition between research organisations with different missions.
• Changes in institutional funding principles should be introduced incrementally in
order to provide sufficient time for adjustment on both sides – principals (funding
providers) as well as agents (research organisations).
• Introduction of several components of institutional funding allows combining
elements of continuity, stability and incentives for a desirable change in
behaviour. It also enables differentiation of institutional funding schemes for
different types of research organisations.
• Performance contracts are suitable instruments for activation of a dialogue
between ministries (providers), and research organisations.
The funding principles
Distribution of institutional funding for RO among research funders
A pre-requisite for the proposed funding model is that there are separate ‘pots’
(budgets or budget lines) for different types of research organisations: scientific
research organisations, RTOs, public service research organisations, national
resources/infrastructures. The thinking behind this pre-requisite is that different
types of research organisations fulfil different missions and functions in society.
Hence, they should not be made to compete for the same budget pot as this may lead
to some research organisations not being able to fulfil their roles any more.
This pre-requisite is in line with international practice where different budget pots for
different types of research organisations are normal. The distribution is generally
contingent on: laws (responsibilities), history (e.g. size of budgets), politics, policies,
(entrepreneurial) individuals (or the opposite).
In the end, how much money to allocate to the different pots is a policy decision. For
the sake of transparency and stability, we suggest the following procedure for the
transition from one single budget for institutional funding for RO of all kinds to
different budgets for different types of research organisations: We suggest taking the
current expenditure on institutional funding for RO as a starting point (mean values or
weighted mean values over several years). This would provide continuity and stability
even if it is a starting point that some may consider unfair given the re-allocations of
institutional R&D funding entailed by the application of the earlier ‘coffee mill’ version
of the Metodika. Nonetheless, the starting point suggested is transparent and
predictable, and this is important given that the definition of pots for different types of
research organisations ultimately involves a policy decision.

R&D Evaluation Methodology and Funding Principles iii
For the future, ideally the responsible ministries should base their funding decisions
on a thorough understanding of the research organisations’ ‘funding mix’
(institutional/competitive/contract funding). On this basis it will be possible to
determine better how large the pot for ‘their’ type of research organisations should
ideally be.
Components of the funding system
Based on our mandate, we propose a funding system that encompasses three
components: a block grant, a performance agreement and a PRFS (performance-based
research funding system).
In order to ensure trust and stability, part of the institutional funding for RO will be
allocated in the form of a block grant. While the block-grant ensures trust and
continuity, the performance-based funding component reflects both (i) past
performance as well as (ii) future developments and plans. Therefore, the
performance-based part of institutional funding for RO will encompass:
• A performance-based research funding system (PRFS) which allocates funding
on the basis of the new Evaluation Methodology, and is mainly based on the
judgement of external peers in their respective fields of expertise and types of
research organisations assessing mostly past performance but also future
strategies and
• A (negotiated) performance agreement, mainly based on the negotiation between
the government (a ministry) and the respective research organisation.
Figure 1 Allocation of funding by type of instrument
Source: Technopolis Group
Figure 13 shows on what basis institutional R&D funding is determined and to whom
the funding is allocated. Block grants and performance agreements are allocated to
research organisations as legal entities and are determined on the basis of the budget
allocated to research organisations in the preceding funding period. The thinking for
PRFS money is different. The money is determined on the basis of scores achieved by
research units (RUs) and then aggregated to the level of the research organisation as
the legal entity which receives the PRFS money.
Block grant Performance-based components
Performance
agreement
Performance-based
research funding
system (PRFS)
Funding determined at
the level of the legal
entity
entity
the level of individual
RUs
Funding allocated to the
legal entity
legal entity
legal entity

iv R&D Evaluation Methodology and Funding Principles
Shares of the components of the funding
We suggest that, similar to the current system, in the first funding period after the first
evaluation based on the new methodology, the block grant should make up 80% of the
pot (or pots for the different research organisations) and 80% of what each research
organisation received in the previous funding period. The previous funding period
should refer to an average of what research organisations received over a period of 3-5
years (a ‘reference funding period’), in order to take into account trends. Moreover,
these 3-5 years should cover the years when institutional R&D funding was based on
the evaluation methodology valid for the years 2013-2015 (or later), encompassing an
80% block grant and an 20% performance based part. Using the years before the
Evaluation Methodology 2013-2015, when a higher share of institutional R&D funding
was allocated by the ‘coffee mill’ would not make sense because fluctuations were too
large.
The starting point we are suggesting for the block grant safeguards continuity with the
present funding situation for each research organisation involved. We have developed
a special treatment for RUs that achieve low scores in the new evaluation, which
should somewhat mitigate the troubles associated with the starting point.
With regard to the performance-based research funding system (PRFS), we suggest
basing 15% of total institutional funding for RO on it. Moving 15% of institutional
funding for RO through a PRFS is in line with international good practice, e.g. in
Norway or Denmark. The PRFS, such as it is proposed at the moment, would be run
every five or six years. Experience with performance based research funding systems
in other countries shows that even small shares of budgets allocated on the basis of
performance indicators can have dramatic effects on the research system, not least
through a psychological effect. Contrary to what might conventionally be expected,
researchers’ behaviour tends to be more strongly impacted by measures of esteem.
Getting a bad score in the PRFS generally is an enormous incentive to perform better
next time.
As to the performance agreement, we suggest linking 5% of funding to a ‘light touch’
performance agreement to promote institutional development and capacity building.
The performance agreements should be negotiated between a funding provider
(ministry) and research organisation for a period of two and half to three years, to be
compatible with the PRFS periodicity. Those ministries that in the past lost their R&D
budget to the Ministry of Education need also be involved in the negotiation of the
performance agreement to ensure the relevance of the performance agreement. In the
long term, the possibility to transfer R&D funding responsibility back to the founding
ministries should also be investigated.
‘Light touch’ in this context means that the performance agreement should not cover
the entire research organisation with all its activities but focus on one or several
strategic projects focusing on well defined issues of institutional development. The
rationale for this ‘light touch’ performance agreement with a 5% share in the funding
system is that principals (providers/ministries) and agents (research organisations)
are new to this instrument. This low share will give research organisations and
ministries an opportunity to learn and introduce a cultural change, while not risking
too much.
Together, the performance-based components will make up 20% of total institutional
R&D funding. This may appear to be a low share although at the same level as the
current system. However, given the high percentage of competitive funding
(approximately 50%) in the Czech R&D system, we think that the competitive element
in the institutional R&D funding system should not be too pronounced, in order to
guarantee stability and allow capacity building.
In the present system, a formula-based evaluation takes place every year, based on
achievements in the prior period, i.e. the preceding five years. In the future, the new
peer-based evaluation is scheduled to take place every five or six years, which implies

R&D Evaluation Methodology and Funding Principles v
that the overall amount of funds allocated through the corresponding mechanism for
institutional funding for RO is also set for the same period of five or six years
Shares of components in future funding periods
We suggest an 80+15+5 allocation of institutional R&D funding to all types of research
organisations for the first funding period because they all are in need of stability.
However, shares of the block grant component and the two elements of the
performance based component can be changed in future funding periods. This means
that the share of the funding components can be increased or decreased depending on
policy requirements. Of course, this requires a policy decision that cannot be delegated
to a PRFS or some other ‘mechanical’ system.
We suggest the following principles should be considered when thinking about
increasing or decreasing the shares of the funding components.
• One prerequisite is that the PRFS is well established among stakeholders and
needs to work well before its share can be increased. The same is true for the
performance agreements.
• Experience in other countries shows that in countries where there is an increase in
institutional funding, this increase tends to be allocated through the performance-
based part, which implies that the share of the block grant decreases – while not
decreasing in absolute terms, thus ensuring stability. In countries where there is
little increase in institutional funding, the shares of the block grant and the
performance-based part typically remain fairly stable.
• Because there are different budget pots for different types of research
organisations, the shares of the funding components can be changed separately for
each type of research organisation, depending on their needs. For example if a
certain type of research organisation displays a particular need for institutional
development, the performance agreement component should be given a higher
weight. Similarly, if a certain group of research organisations displays a need for
more quality and is homogeneous enough to benefit from more direct
competition, the PRFS part of the funding could be increased. The guiding
principle would be that changes in funding component shares need to be in line
with the needs and requirements of particular types of research organisations, as
articulated both by the research organisations themselves and their owners.
• Another important guiding principle is that the funding mix
(institutional/competitive/contract funding) ought to be part of the decision-
making on the shares of the different funding components for different types of
research organisations. If there already is a high share of competitive funding in a
group of research organisations, then the performance-based components, in
particular the PRFS, should not be over-emphasised in order to ensure stability
and opportunities for capacity building. Similarly, if the share of competitive
funding is fairly low, then the performance-based components can be increased
more. The share of competitive funding is typically higher in more applied
research organisations compared to basic research organisations.
Performance agreements
The performance agreement component resembles an existing funding stream among
HEIs called Rozvojové programy (Development Programme) administered by the
Ministry of Education, Youth and Sports (MEYS), which also promotes institutional
development projects. However, while the current programme for HEIs uses a funding
formula based on quality indicators, the performance agreements we are proposing
here will not because in the funding system the quality aspect is covered by the PRFS.
The performance agreement component, however, foresees sanctions in case the
research organisation does not fulfil the agreement.
The performance agreements are set up individually, taking into account the specifics
of each research organisation (e.g. role and mission, research profiles, institutional

vi R&D Evaluation Methodology and Funding Principles
setup). They are light touch and promote institutional development and capacity
building. The main idea behind this approach is that a leverage effect is expected: if
the strategic projects succeed, they will have a positive influence on the whole or at
least essential parts of the research organisation.
The performance agreement is an agreement between the ministry and the research
organisation on a small number of strategically relevant undertakings that the
research organisation agrees to implement in the performance agreement period in
order to increase R&D capacity and to improve working conditions for research as well
as for support staff.
Performance-based research funding system (PRFS)
The new Evaluation Methodology provides us with scores from 1 – 5 against five
categories of indicators: research excellence, research performance, societal relevance,
institutional management and development potential, membership in the national and
global research community 1.
In order to translate the scores into funding, our proposal is to use the five categories
and define “sub-pots” for them. This implies a weighting of the categories in line with
the organisations’ mission and in line with the PRFS’s objectives. This in turn implies
that the weights have to be different for the different types of organisations because
they have different missions and roles in the Czech R&D system. While the weighting
itself is essentially a policy decision, we can provide guiding principles on which to
base the decision on how to weigh the different categories for the different type of
organisations.
• The main guiding principle will be that the weights of the different evaluation
criteria need to be in line with research organisations’ missions. For example,
research excellence is of highest relevance for scientific research organisations.
Hence, this category should have a higher weight for scientific research
organisations than for all the other types of research organisations. In contrast,
societal relevance is more important for research organisations that conduct
applied R&D and serve user communities such as industry sectors (RTOs), the
public sector (public service research organisations) or other researchers (national
resources/infrastructures such as archives and libraries) than for scientific
research organisations. Consequently, societal relevance should have more weight
for these types of research organisations.
• Weights will also need to be in line with ministries’ strategies for ‘their’ research
organisations. While these strategies should be in line with research organisations’
missions, strategies will typically go beyond the mission, emphasising certain
policy objectives. For example, if it is a strategy to increase excellence in scientific
research organisations, then research excellence should be given a high weight. If
the strategy is to improve management, then management needs to emphasised.
1 See R&D Evaluation Methodology and Funding Principles, First Interim Report: The Evaluation
Methodology EM2015, p. 62-63.

R&D Evaluation Methodology and Funding Principles vii
Table of Contents
1. Introduction 10

2. The background 11

2.1 International organisation and funding practice 11

2.1.1 The R&D governance system 11

2.1.2 Governance of public research performers 16

2.1.3 PRFS and Performance Contracts 21

2.1.4 Institutional funding budgets 22

2.1.5 Effects of funding systems 26

2.1.6 Some lessons 29

2.2 The current institutional funding system in the Czech Republic 30

2.2.1 Governance of the institutional funding for research system 30

2.2.2 Methods of institutional funding allocation 30

2.2.3 The amount, distribution and utilisation of institutional funding 31

2.2.4 Evaluation and control mechanism 32

2.2.5 Impact of the transition from the research intentions system to the current
institutional funding system 33

2.2.6 SWOT analysis 33

2.2.7 The Czech institutional funding system in international comparison – main
lessons 39

3. The revised funding principles for the Czech Republic 41

3.1 Context to the principles for institutional funding for RO 42

3.1.1 General concepts for the institutional funding for RO 42

3.1.2 The specific starting point in the Czech Republic 43

3.1.3 Cross-subsidies between different sources of institutional funding 44

3.1.4 Ensuring adequate field coverage 45

3.1.5 Efficient sharing and use of national and international research
infrastructure 46

3.1.6 Changing national priorities and the need for stability of institutional R&D
funding 47

3.2 The core funding principles 48

3.2.1 The distribution of institutional funding for RO among research funders 48

3.2.2 Components of the funding system 49

3.2.3 Shares of the components of the funding 50

viii R&D Evaluation Methodology and Funding Principles
3.2.4 Shares of components in future funding periods 52

3.3 Performance agreements in more detail 54

3.3.1 Scope and content of a light touch performance agreement 55

3.3.2 Reporting and implementation 58

3.4 Performance-based research funding system (PRFS) in more detail 59

3.4.1 Weighting the evaluation criteria for different types of research
organisations 59

3.4.2 Options for translating scores into funding 67

3.4.3 The issue of RUs with low scores 73

3.4.4 The issue of small RUs 75

3.5 How can new research organisations enter the funding system 77

Appendix A - List of Abbreviations 80

Table of Figures
Figure 1 Allocation of funding by type of instrument.................................................... iii

Figure 2 Generic Organisational Structure for Research and Innovation Policy ..........12

Figure 3 Government Expenditure on R&D (GOVERD), 2011 ......................................17

Figure 4 Research-performing organisations in the comparator countries that receive
institutional funding (not exhaustive)............................................................................18

Figure 5 ‘Competition arenas’ in the Norwegian research institute sector................... 22

Figure 6 Implications of the economics of knowledge for state research and innovation
funding ........................................................................................................................... 23

Figure 7 Government budget appropriations or outlays for research and development
(2013; million EUR)....................................................................................................... 24

Figure 8 Share of GUF versus direct government funding of R&D expenditure in the
Higher Education Sector, 2009 (in millions of €)......................................................... 25

Figure 9 Current university institutional funding systems........................................... 25

Figure 10 R&D Governance ........................................................................................... 34

Figure 11 Recipients ........................................................................................................37

Figure 12 Research questions and location of answers..................................................41

Figure 13 Allocation of funding by type of instrument................................................. 50

Figure 14 Evaluation criteria and their weights for scientific research organisations
(default scenario) ........................................................................................................... 60

Figure 15 The two dimensions of PRFS funding system (default scenario) ................ 62

Figure 16 Weights in the default scenario .................................................................... 62

Figure 17 Translation of weights into shares of total institutional funding for RO, by
type of research organisation......................................................................................... 63

R&D Evaluation Methodology and Funding Principles ix
Figure 18 Weights in the medium scenario...................................................................65

Figure 19 Translation of weights into shares of total institutional funding for, by type
of research organisation .................................................................................................66

Figure 20 Weights in the radical scenario.....................................................................66

type of research organisation .........................................................................................67

Figure 22 FTE parameter at sector level .......................................................................68

Figure 23 Translation of scores into points...................................................................70

Figure 24 Examples of scores of RUs, by evaluation criteria........................................70

Figure 25 Examples of average scores of research organisation, by evaluation criteria71

Figure 26 Example of aggregating scores at national level, by evaluation criteria ....... 71

Figure 27 Example of funding implications, by evaluation criteria .............................72

Figure 28 Example of funding implications for research organisations ......................72

10 R&D Evaluation Methodology and Funding Principles
1. Introduction
This report is the draft version of the Second Interim report. It sets out our thoughts
and a first draft of the new institutional funding principles for research organisations
in the Czech Republic.
The Second Interim report responds to the requirements in the Terms of Reference
(ToR) for the Work Packages 5 to 8 in this study.
In the next chapter of this report, we give an overview of the background for the
development of new funding principles, i.e. a summary of international R&D
governance and institutional funding practices (Section 2.1), and the current
institutional funding system in the Czech Republic, its weaknesses and strengths
(Section 2.2). We conclude this chapter setting the Czech system in the international
context and identifying some ‘lessons learned’ for the design of the new funding
system.
In Chapter 3 we propose the new principles for institutional funding of research
organisations in the Czech Republic.
This report builds upon a set of analyses that are reported in the following background
reports (separate documents).
• The R&D governance and funding systems for research in international practice
• The institutional funding system in the Czech Republic
• The ex-ante assessment of the proposed funding system. The analyses reported
there have fed - and will continue to feed - into the design of the funding
principles. Currently, the use of the simulation model is therefore predominantly
internal

Report title
R&D Evaluation Methodology and Funding Principles 11
2. The background
2.1 International organisation and funding practice
This section aims to put the Czech funding system into an international context,
focusing on the five countries identified for this purpose. A fuller discussion of these
countries appears as Appendix B to out First Interim Report.
2.1.1 The R&D governance system
The key parameters of the new Evaluation Method, especially those that determine
funding, are components of a wider ‘policy mix’ for R&D. Many policy decisions such
as the policy mix ‘emerge’ from the interplay of a complex set of drivers that are
specific to the national research and innovation system. For example, the optimal
balance of ‘basic’ and applied research funding may be influenced by the structure of
industrial and social needs as much as the characteristics of science and determined
by the sum of decisions made by a variety of actors, including ministries, funding
agencies and Research Organisations (ROs). In this section we summarise some key
aspects of R&D governance systems that need to function well if they are to produce
good policy.
Figure 2 offers a framework for comparing the way the state organises, governs and
coordinates the R&D system in different countries. It shows the organisational
hierarchy that is usual in West European countries. Organisations at each level
generally act as ‘principals’ to those immediately below them. The UK has long
followed the ‘Haldane principle’ that a government ministry can direct its research
funding agency in broad terms but may not interfere in specific project funding
decisions. In Sweden, such micro-management (ministerstyre) is formally illegal.
The tradition of ‘new public management’ increasingly means that principals
everywhere set objectives for those below them, rather than micro-managing them.
Policy coordination matters because it provides a way to ensure the overall coherence
of research and innovation policy – an aim is to make sure that one part of the system
does not rely on another part to deliver something, which it turns out is not delivered.
It is increasingly important as research and innovation policies have to confront the
‘grand’ or societal challenges such as climate change, ageing, HIV/AIDS and so forth,
which cut across the needs and abilities of individual parts of the system such as
ministries to deal with them.
The analytic framework in Figure 3 involves four levels. Each has a distinct set of
functions and it is possible to try to organise policy coordination at each level.
• Level 1 is the highest level. This involves setting overall directions and priorities
across the whole National Innovation System. Coordination be achieved through
advice to government or by more binding means, such as decisions of a cabinet
sub-committee
• Level 2 is the level of ministries, where policies are normally set. There is a need
for co-ordination among ministries, whose sectoral responsibilities otherwise
encourage them to pursue independent policies. In practice this level of co-
ordination may involve administrative aspects, policy issues or both.
• Level 3 is the level of funding agencies, whose task is to translate policies into
funding concrete programmes, projects and other activities. This level can involve
administrative co-ordination as well as more substantive co-ordination of funding
activities, such as co-programming
• Level 4 is the level of research and innovation performers. They execute the
projects and programmes funded by Level 3. Co-ordination at this level tends to be
achieved through self-organisation rather than using formal mechanisms

Internationally, this four-level hierarchy tends to be reflected in four organisational
levels: governments; ministries; funding agencies; and research performers. There are
still examples of organisations that span more than one vertical level. For example,
some of the UK research councils operate research institutes, so their work spans
Levels 3 and 4. By this kind of vertical integration is becoming less and less normal.
A key difference between this archetypal organisation structure and the Czech
Republic is that the Czech Republic retains the old Soviet structure where the
Academy of Science had the same status as a ministry, internalising the distinctions
between policymaking, research funding and research performance that in Western
systems tend to be looked after by different organisations. The Technology Agency
also follows this structure, effectively having ministry status.
Figure 2 Generic Organisational Structure for Research and Innovation Policy
Source: Modified from Martin Bell, Knowledge Resources, Innovation Capabilities and
Sustained Competitiveness in Thailand: Transforming the Policy Process, report to the
National Science and Technology Development Agency, Bangkok, Brighton: SPRU, 2002

Report title
The way in which national R&D activities are governed or ‘steered’ by the state is
complex in all countries. The private sector, of course, steers itself – though the state
can use incentives such as subsidies, tax breaks and regulation to encourage certain
types of behaviour. Within the state, there are multiple stakeholders – policymakers,
funders and performers – in relation to R&D, all of whose decisions affect the actual
pattern of R&D activity. In such a complex system the overall characteristics of the
national effort are ‘emergent’: they emerge from the way the different actors in the
system behave.
The organisation and governance structure shown in Figure 2 is effectively the way in
which the state connects R&D activities to social needs. Government is a major
influence. But the individual ministries also have a strong say, with each representing
a particular ‘sector’ of society. In principle, each ministry has an understanding not
only of the overall needs of its sector but the kind of research needed to advance
knowledge and develop policy. In many cases, some of this need will be expressed
through a PSRO ‘owned’ by the ministry. In research policy as in policy more
generally, therefore, the spending ministries make competing claims about their needs
– and have an annual battle with the finance ministry about how many of these claims
can be afforded. The relative power of government centrally and the individual
ministries varies among systems but the outcome is rarely the result of a simple top-
down decision. It emerges from the competition among claims – a competition that
can in many systems benefit from being expressed in an ‘arena’ such as a policy
council, where it is possible to coordinate and negotiate. Such an arena may also be
helpful to focus the national effort in pursuit of a strategy, make sure all the needed
parts of the system function and make it possible for the national system to change
direction when circumstances change.
One of the biggest of these changes currently is the emergence of ‘grand challenges’
(climate change, ageing, HIV-AIDS and so on) as policy priorities. To a greater extent
than earlier policy foci such as industrial development and growth, the grand
challenges require joint and consistent activities across many disciplines and many
sectors of society. They therefore appear to imply a need to change the way research
and innovation governance is structured and the need to tackle them needs to be built
into any such governance system.
For a comparatively decentralised governance system to operate well, the individual
actors need to be able to analyse, express and lobby for the satisfaction of their own
sector needs. This means that people with relevant skills and capabilities and the
independence not only to undertake the needed analysis but also to be able to present
its results to the wider policy community must populate them. ‘Distributed strategic
intelligence’ is needed – in the sense of a wide availability of data and sources of
information. Inherently, such a system will not only foster disagreements among
actors, but also the transparency of information that allows wide participation in
policy debate.
In summary, key characteristics of the funding and governance systems in our
comparator countries are as follows. (More detailed descriptions are in the Appendix
to our Second Interim Report).
Austria tends to have coalition governments, which means that there is often not a
single government view on research and innovation policy. Rather, different political
parties treat individual ministries as their own fiefdoms, making policy coordination
harder than in some other countries. With separate policy advisory committees in the
industry (BMVIT) and science (BMWFW) ministry spheres and no overall arena or
‘referee’ at the level of government, it is difficult to maintain a coherent and holistic
policy. The ministries – especially BMVIT – also micro-manage some of the work of
the agencies, for example by giving final approvals to project grants, so the space for
cooperation at the agency level is also limited. The Balkansation of policy continues
down to the performing level, with the two key ministries maintaining separate
institute systems. Other sector ministries maintain their own labs and – in the case of
agriculture – university (BOKU). There is no over-arching thematic research and

innovation strategy. Thus, to the extent that the grand challenges are addressed this
tends to be by single-ministry programmes rather then through more coordinated use
of the state R&D system as a whole.
The Netherlands maintains separate, powerful ministries of economics (EZ) on the
one hand and education and research (OCW) on the other. In the past there has been
a proliferation of research and innovation policy advisory committees but now there is
a single one (AWT) operating at the level of government as a whole. It advises the
government on individual research and innovation issues rather than on overall
strategy. Separately, the Royal Dutch Academy of Arts and Sciences (KNAW) offers
research policy advice to government. EZ maintains a research and innovation policy
focused on funding ‘top sectors’ defined in industrial terms, while OCW operates a
fairly traditional set of education and research policies focusing on the higher
education sector. The NL Agency implements EZ programmes in the business sector
but has little responsibility for strategy or policy. The national research council NWO
focuses largely on academic research, though it has a small sub-council for innovation
(STW), which it funds together with EZ. At the level of research performance, EZ
supervises TNO (the national RTO) but also two other chains of institutes: the Large
Technological Institutes (GTIs) and the DLO institutes, both of which can been seen as
spanning the PSRO and RTO functions. OCW and NWO maintain their own scientific
research institutes. Other ministries – notably agriculture and defence – have large
institutes of their own. As in Austria, however, the mechanisms that would enable
horizontal coordination in areas such as the grand challenges appear weak.
Norway has separate industry and education ministries, with responsibilities
respectively for innovation and research. It maintains a large number of PSROs
serving individual sector ministries. It has at various times had advisory councils for
research and innovation combined and separately, both at the level of the government
as a whole and latterly at the level of the education and industry ministers. Generally,
these have proved ineffective and the current government has abandoned any attempt
to use such structures. The Research Council of Norway – a combined research
council and innovation agency – was set up in 1993 in response to the inability of the
existing R&D funding organisations to implement the government’s cross-cutting R&D
priorities. Its ability to coordinate across the strong boundaries between the sector
ministries has been limited but in recent years ministries have been forming small
clusters in order to generate cross cutting strategies in areas like climate and
nanotechnology that are implemented chiefly through RCN programmes. The
education ministry takes the lead in coordinating other ministries’ research policies
and in the last decade or more has succeeded in establishing national thematic
priorities through a series of research bills.
Sweden took a policy decision in 1942 to avoid fragmenting its scarce national R&D
resources and therefore to use the universities not only for basic research and higher
education but also to cover many of the roles played by PSROs and RTOs in other
countries. This would therefore appear to be a natural place for coordination to take
place, were the university system not at the same time so fragmented. The industry
ministry runs the major innovation agency (VINNOVA) and owns most of the small
RTO system. In recent years, it has launched an innovation policy, separate from the
national efforts elsewhere in research. The education ministry operates the Swedish
Research council, focusing on basic research, and shares ownership of two more
specialised councils (FORMAS and FORTE) with other sector ministries. Boards
containing a majority of people elected from the research community run all the
research councils, so they are difficult to steer any wider government strategy. There
has been for many years a group of academics advising the education minister on
research policy, but this has not generated any thematic strategy. The education
minister chairs all discussion of research in the cabinet, with the intention in that way
of creating a holistic policy but this appears to have had few results. The lack of
strategy and coordinated effort is exacerbated by the presence of the Wage Earner
Foundations (Strategic Research, MISTRA, Knowledge and a number of smaller ones),
which are organisationally separated from the state and lie outside the normal

Report title
policymaking mechanisms. Recent governments have tried to impose overall strategy
by allocating research funds to the universities, earmarked to specific areas of research
and innovation. However, with over 2o priorities in each case, it is hard to see these as
either effective or strategic – though they do in a number of instances address the
grand challenges.
The UK has over the years centralised its higher education, research and innovation
policies into a single ministry, currently called the Department for Business,
Innovation and Skills (BIS). BIS has published a national research strategy for
innovation and growth and recent BIS policy announcements point to the
prioritisation of certain key and enabling technologies – although the research
councils and universities are expected to set their own priorities. The Council for
Science and Technology advises the government on research and innovation policy –
but like the Dutch AWT this provides advice on individual issues rather than strategy
or overall policy. In parallel, the Chief Scientist provides the Prime Minister with
science-based advice on policy – but this essentially is ‘science for policy’ rather than
‘policy for science’. In recent years, all the departments of state have taken up this
idea of having a Chief Scientist to provide science-based policy advice. Outside BIS,
four departments of state (defence, health, international development, and
environment, food and rural affairs) spend large amounts of money on research.
Several departments fund so-called public sector research organisations (ie PSROs),
some of which are operated by private sector companies. While the work of the seven
research councils is coordinated by their umbrella organisation RCUK, there is no
clear coordination mechanism or ‘arena’ for defining cross cutting strategies in the
UK. To some extent this may be because the funding system is so big compared with
the other countries discussed here, that issues like climate change can be handled
within the research council system.
In the Czech Republic, the main national policy document is the R&D&I Policy
2009 – 2015. The Council for Research, Development and Innovation (CRDI) is the
first level body that advises the Government on the priorities, budget and the overall
organisation of the R&D&I System in the Czech Republic. It is the key ‘arena’ in which
overall research and innovation policy can be coordinated. A member of the
Government - currently the Deputy Prime Minister - acts as the Chairman of the
CRDI, thus enforcing its legitimacy. Besides the CRDI, there is a set of ministries and
three agencies (the Science Foundation, the Technology Agency and the Health
Agency2), responsible for the implementation of the R&D&I policy, that constitute the
second ‘intermediary’ level. Apart from the Ministry of Education, Youth and Sports,
there is the Ministry of Industry and Trade, Ministry of Health, Ministry of Defence,
Ministry of Culture, Ministry of Agriculture and the Ministry of Interior. These
ministries act as providers of institutional support. Three Ministries, i.e. the Ministry
of Defence, the Ministry of Health, and the Ministry of Agriculture were assigned
responsibility for sector-specific R&D&I. All of these Ministries manage the national
institutional funding for the research organisations – public or private non-profit – in
their area of competence; most of them also develop and manage competitive R&D
programmes. Other sector ministries lack (or have been deprived of) budgets to fund
institutional R&D costs – and therefore to maintain a body of research capabilities and
evidence relevant to developing their policies.
Among the countries discussed, the Czech Republic might appear to be the best placed
organisationally to coordinate research and innovation policy because it has an over-
arching R&D&I council closely linked to government. However, a key weakness of the
CRDI is the fact that it comprises almost entirely members of the research community,
as opposed to incorporating wider stakeholder interests inside and outside
2 established only in mid 2014

government. The apparent lack of coordination in most of the other systems, however,
seems to be compensated in the short term by the ability to take comparatively
decentralised decisions based on decentralised absorptive capacity and strategic
intelligence about policy in research and innovation systems that are somewhat
mature. In Norway, this ability to self-organise extends to creating new constellations
of ministries to make strategy. But most of these systems are poorly adapted to taking
major changes in direction – in particular to reorganising the national effort in a way
that addresses the grand challenges. In general, overall policy or the policy mix
‘emerges’ as the sum of many decentralised decisions rather than being decided.
Those places that lack an ‘arena’ like the Czech R&DI Council also lack a place in
which it is possible to address systemic issues, like the balance between institutional
and competitive funding, the relative investments in basic and applied research or the
degree to which it makes sense to reorient parts of research and innovation policy to
the grand challenges.
2.1.2 Governance of public research performers
Research-performing organisations have become increasingly autonomous from
government and the state over time. This means in practice that principals micro-
manage them less than before and to a greater extent use incentives to encourage them
to implement policy, while recognising that there are also many aspects of their
behaviour that the state simply does not need to regulate. Perhaps the most important
limitation on universities’ autonomy is that most countries use a central agency to
accredit degree courses. In the UK, the Quality Assessment Authority uses peer review
to monitor degree standards and advises on which organisations should be allowed to
grant degrees.
In all the countries considered, the universities now effectively set their own strategies
and control their internal budgets. In particular, they are free to use their institutional
research funding as they themselves choose. The academies and research councils
that control most of the scientific research institutes set their own policies while the
RTOs have always needed a high degree of autonomy in order to be flexible enough to
serve their largely industrial markets. The spread of the new public management
means that PSROs are increasingly managed by objectives, leaving them to decide for
themselves how to reach these objectives. In many cases, governments also want to
see the labs winning industrial income in addition to fulfilling their tasks for the state,
so this is another factor encouraging autonomy.
For the purpose of this comparison, we distinguish among five different types of
research-performing organisation. At some points in this report we refer to the first
two collectively as ‘scientific research organisations’.
• Universities – or, strictly, research universities (since some universities only teach
and do not do research)
• Scientific research institutes – which, like research universities, conduct
fundamental or applied research but either have no teaching responsibilities or
only provide education at PhD level (alone or in combination with a university)
• Research and technology organisations (RTOs) – which conduct applied research
and experimental development and provide technical services to support
industrial innovation. These can be distinguished from technical consultancies in
that they receive institutional funding from the state
• Government labs – known as ‘public service research organisations’ in the Czech
Republic, these do research and provide technical services on behalf of
government. They produce knowledge the government needs in order to legislate
or regulate or they produce ‘public goods’ such as standards, certification or
weather forecasts that society needs but that private companies lack the incentives
to make. Some PSROs are run by private companies on behalf of government

Report title
• National resources or infrastructure – such as libraries and museums, which
enable others to do research and which may otherwise be needed for social,
educational or cultural reasons. Normally, these need to do some research of their
own in order to support their infrastructural function
Some of these organisations may do quite a limited amount of research, so they are not
all always considered in discussions of research policy. For example, the Austrian
PSROs tend to be ignored in literature on Austrian research policy. Few research
policy documents take account of libraries and museums.
While there are important overall similarities in the types of research-performing
organisations present in different countries, especially if they are considered in terms
of their roles and processes, their relative importance, governance, funding and quality
control mechanisms vary.
Figure 3 shows that higher education dominates government expenditure on R&D in
all the countries compared. It is especially high in Sweden because of the national
focus on using universities as all-purpose research institutions and in The
Netherlands. The government sector is especially important in the Czech Republic
because the ASCR is in this category, while the other countries considered do the
equivalent research in the universities and do not maintain large academy systems. In
The Netherlands and Norway the national RTOs fall into the government sector,
together with the public service laboratories. In the Czech Republic, Austria and
Sweden, the RTOs are organised as limited liability companies so they are classified as
belonging to the ‘business sector’, despite the fact that their mission is essentially
public. The high proportion of Austria’s GOVERD spent in the business enterprise
sector is also influenced by that country’s unusually high rate of R&D and innovation
subsidies to industry. The UK figures are influenced by the fact that many of the
public service laboratories have been privatised or are operated by private companies
on behalf of the state. There is probably also a legacy from the UK’s historical high
level of expenditure on defence R&D in both the public and the private sectors.
Figure 3 Government Expenditure on R&D (GOVERD), 2011
Source: Eurostat
Figure 4 gives some basic information about research-performing organisations in the
comparator countries. Where individual organisations are shown these are examples
– the lists are not always exhaustive.
0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Private
non-‐pro8it
sector

Higher
education
sector

Government
sector

Business
enterprise

sector

Figure 4 Research-performing organisations in the comparator countries that receive institutional funding (not exhaustive)
Universities Scientific research
institutes
RTOs PSROs
Austria 22 public universities Austrian Academy of Sciences
Ludwig Boltzman Institutes
Austrian Institute of Technology
Joanneum Research
Salzburg Research
Upper Austrian Research
Institute for Advanced Studies
(HIS)
WIFO
BEV (metrology)
AGES (health and food safety)
ZAMG (meteorology)
Netherlands 14 public research universities KNAW – 18 research institutes
NWO – 9 research institutes
TNO
4 Large Technological Institutes
(GTI)
9 DLO institutes (agriculture)
12 PSROs attached to 5 ‘sector’
ministries
Norway 39 public universities See PSROs box and the main text
for explanation
13 ‘technical-industrial’ research
institutes
38 mixed-purpose institutes
• 23 social science
• 8 environment
• 7 ‘primary industry’
Sweden 46 public universities None RISE – comprising 16 RTOs
grouped into 4 technological
divisions
FoI Swedish Defence Research
Agency
SSMHI Swedish Meteorological
and Hydrological Research
Institute
SMI Swedish Institute for
Infections Disease Control
SSI Swedish Radiological
Protection Authority
UK 154 public universities About 50 institutes belonging to
various research councils
7 Catapult Centres About 100 public research
establishments (including
museums and libraries)
Czech Republic 29 public universities 54 institutes 33 RTOs 34 PSROs
Note: Grey shading indicates the use of a PRFS. Yellow shading denotes performance contracting. Blue shading denotes a mixture of the two.

Draft of the Second Interim Report

We have not included centres of excellence or competence centres in Figure 4. These
are usually organised as funding programmes by research councils or innovation
agencies – even in cases where (as in Austria) the individual centres are legal persons.
Austria is unusual too in operating some small competence-centre like activities
through the Christian Doppler Society, though the main effort in this area is a
programme funded through the innovation agency, FFG.
Naturally, all the countries considered have research universities. The numbers of
universities shown in Figure 4 are inflated in the Swedish and Norwegian cases by the
treatment of ‘universities and colleges’ as a single category. Only in the Czech
Republic are private universities routinely eligible for state institutional research
funding. In recent years, Sweden has been experimenting with university legal form
by allowing Chalmers to become a foundation rather than a state agency. The
education ministry contracts with Chalmers to buy certain numbers of degrees and to
provide institutional funding for research so that it effectively competes on an equal
basis with the state universities.
In most countries public universities are agencies of the Education Ministry or can
otherwise be instructed by the government. Each public university in the UK has a
‘charter’ granted by the monarch, under which it is self-governing and can essentially
pursue the purposes for which it was established as it sees fit.
All but one of the cells in Figure 4 is populated: Sweden does not have any scientific
research institutes, since as a matter of policy scientific research is the job of the
universities. The other comparator countries’ scientific research institutes are almost
all associated with academies or academy-like structures such as research councils.
But unlike in the Czech Republic where the ASCR is a major block in the research
performing system, these are mostly small and are often historical legacies rather than
being a mainstream way to organise research. Some of the UK research council
institutes were set up to establish new fields, in the expectation that their role will later
be transferred to the university system. Some other countries not considered here do
have large scientific institute sectors – notably Germany, Spain and France, though in
the latter case almost all the institutes now ‘cohabit’ with university departments.
With the exception of the UK, the comparator countries all have at least some large
RTOs – even though (as in Sweden and Austria) RTOs may constitute a modest part of
the total research-performing system. The government withdrew institutional funding
from the British RTOs in the 1980s, so the former generation of RTOs (made up
mostly of branch-based research associations) either collapsed or became technical
consultancies. In 2010 the UK government decided to re-establish an RTO sector and
started to build a network of ‘Catapult Centres’ based loosely on the Fraunhofer model.
So the small size of the individual Czech RTOs is distinctive.
RTOs and PSROs may generally receive institutional research funding irrespective of
their legal form – but those that are organised as private companies must in practice
operate as not-for-profit organisations (typically in the sense that they do not
distribute dividends to their shareholders) and must recognisably be serving a pubic
rather than a private purpose. Thus, while the large Austrian AIT institute is organised
as a private company (and is 49% owned by industry), it receives institutional research
funding. However, Austria also has a number of research associations organised
under the umbrella of Austrian Cooperative Research. The pharmaceutical industry
owns the Institute of molecular Pathology. AVL is an internationally very successful
contract research organisation working for the vehicles industry. All of these are
privately owned and serve private purposes. None receives institutional funding from
the state.
All the countries considered have dedicated public service research organisations
(PSROs). (In fact, the only country of which we are aware that does not have some
such organisations is Denmark, which integrated its labs into the university system in
2007 but left the ‘GTS’ RTO system outside the university sector.) Austria has a

considerable number of PSROs, though the sector is poorly documented. Sweden has
very few, on the local principle that the universities should do research.
2.1.3 PRFS and Performance Contracts
The comparator countries have rather stable systems of research performing
organisations. Changes are introduced by government policy – for example, as part of
a decision to set up a new university or to privatise a PSRO – though in practice these
are rare events. Entitlement to institutional funding generally follows automatically as
a consequence of the policy decision. Dutch, Norwegian and Swedish universities have
to be accredited in order to qualify. In the Norwegian institutes’ PRFS, the Ministry of
Education and Research decides case by case whether to place institutes under the
stewardship of RCN and therefore to make them eligible for institutional research
funding via RCN.
The Czech Republic’s institutional structure is less stable, in the sense that there have
been many changes since the change of political system and there is still a process of
readjustment in progress between the Academy and the universities. Like Latvia and
Lithuania, the Czech Republic permits a bottom-up process where research
organisations can apply to become eligible for institutional funding. The experience in
all three countries is that this fragments the research performing system.
Figure 4 shows where PRFS and performance contracts are used in institutional
research funding.
Since the 2002 university reforms, Austrian universities, scientific research institutes
and RTOs have generally been funded via performance contracts. Austrian PSROs and
infrastructure tend to be funded directly, with few performance requirements.
Swedish PSROs are steered using the local form of a performance contract, namely a
‘steering letter’, which is an annual letter from the responsible ministry that sets out
the specific tasks required of the agent in the short term and allocates the overall
budget. Dutch universities received funding based on performance agreements from
2012 on. The large Dutch RTOs (TNO and the GTIs) have a different kind of
performance contract. This provides some basic funding more or less without
conditions but then provides a second tranche of institutional funding that has to be
devoted to work in specific fields or themes, specified by the responsible ministry as a
representative for society. This is intended to influence the developmental trajectory
of the RTO, so it is exposed to social needs through the institutional funding and to
industrial requirements through its external contracting. UK PSROs are steered using
‘service level agreements’, the local form of performance contract.
The status of the UK universities (based on a royal charters) means the government
cannot tell them what to do, so they are steered using incentives in a PRFS rather than
through contracts or letters of instruction. The Czech Republic is unique – not only
among the comparator countries but also more widely – in using a single PRFS to
allocate research funding across all kinds of research performing organisations.
The Norwegian and Swedish university systems effectively use both performance
contracting and PRFS. The bulk of their institutional funding is allocated through
formulae and specific requirements made of them by their parent ministries.
However, a small proportion is allocated through PRFS. Both elements are under the
control of the education ministry. The Norwegian institute system is altogether more
complex. The Norwegians make little conceptual distinction between scientific
institutes, RTOs and PSROs. Any ministry can provide institutional funding to any
institute through an annual letter of instruction. The Research Council of Norway
(RCN) is responsible for the stewardship of the research institutes through evaluation,
special funding instruments and a PRFS. RCN recommends to the Ministry of
Education and Research which specific institutes perform sufficient research of
sufficient seriousness to be admitted to this stewardship. (About 50, mostly small,
institutes are excluded in this way.) The criteria are no more precise than this, so the
Ministry decides case by case.

Not all PSROs are part of the RCN system, which contains examples of scientific
research institutes (especially in the ‘soft’ sciences), RTOs and PSROs. The institutes
can each address scientific, industrial and government markets so they can in practice
configure themselves as multifunctional organisations. RCN organises the institutes
into four ‘competition arenas’ and operates an indicators-based PRFS within each
arena. While the rules of the PRFS are uniform, separating the institutes into different
arenas is intended to minimise unreasonable competition caused by field differences.
Many institutes are highly dependent upon project funding in competitive markets, so
the PRFS may provide the bulk of their institutional funding. The greater the extent to
which institutes operate as PSROs, however, the more institutional funding they
receive directly from the parent ministry. As Figure 5 suggests, the PSROs are
concentrated in the ‘primary industries’ (agriculture, fisheries, etc) category.
Figure 5 ‘Competition arenas’ in the Norwegian research institute sector
Arena Core/PRFS
Funding
Direct ministry
funding
No of Institutes
Social science 14% 6% 23
Environment 14% 4% 8
Primary industries 15% 34% 7
Technical-industrial 6% 3% 13
Source: RCN, Instituttsektoren: Forskningsrådets Strategi, Oslo: RCN, 2014
2.1.4 Institutional funding budgets
A key part of the logic of institutional research funding is based on the economics of
knowledge, which in turn determine why the state needs to be involved in research
funding and knowledge production. In economic theory, knowledge is ‘non-rival’ in
the sense that it can be used by many people without being consumed. This is quite
unlike the normal case of physical goods. Once a cake is eaten it is gone. A machine
gradually wears out as it is used. But knowledge once made does not disappear.
Knowledge is also ‘non-excludable’ in the sense that it is difficult to prevent people
from getting hold of it. Non-rival, non-excludable goods are known as ‘public goods’.
They are unattractive for private organisations to produce because they cannot get
money for them. This is often referred to as ‘market failure’. The market can therefore
not produce them. Since they nonetheless are socially useful, the state tends to fund
their production.
A direct consequence is the pattern of funding illustrated in Figure 6. The closer
knowledge gets to market application, the more companies are able to monopolise
aspects of the knowledge and secure economic returns. So the state lays a big role in
funding basic research (usually paying all the costs) but invests a far smaller
proportion in those cases where it intervenes in more applied areas that are closer to
market. The logic maps in a simple way across to different types of RO. Scientific ROs
tend to produce rather fundamental knowledge where there is a high level or most
failure, so the state pays all or most of the cost. PSROs largely produce knowledge
needed only by the state or that are public goods, so they are normally also heavily
subsidised as are infrastructure, for which private funders are unlikely to be found. In
contrast, RTOs work close to market with industrial problems whose solution will
enable one or more companies to make money. The need for subsidy is therefore
smaller, so RTOs internationally tend to get 10-33% of their income as core funding
whereas the other types of organisation considered here get a great deal more.

Figure 6 Implications of the economics of knowledge for state research and innovation
funding
The general way to fund state universities in the post-War period was through a single
block grant – what the OECD called the General University Fund. Historically, the
trend has been for the ratio of competitive project funding to institutional funding to
rise, increasing the external competitive pressure on quality. Research councils
initially dominated the external research-funding stream, responding to investigator-
initiated (‘bottom up’) proposals. However, most changes seem to have been
incremental and uncoordinated. Typically, decisions to increase external funding are
made and implemented separately from decisions about institutional funding, so the
funding mix can change significantly without there being an explicit decision to
change it.
Explicit decision-making about the use of institutional funding – or, indeed, about the
overall funding of specific research-performing sectors – is rare in the comparator
countries. In general, the countries rely on external, competitive funding as the main
way to influence the thematic priorities of the research-performing system. Some
countries incorporate feedback mechanisms from the amount of external funding to
the amount of institutional funding in their PRFS.
Sweden and Norway have a practice of publishing Research Bills every four years.
This means that there is automatically a periodic process of reviewing research and
innovation policy. While these two countries have only weak coordination structures,
the Research Bill discussions force a process of considering priorities. Norway has
periodically specified national thematic priorities since the late 1990s while Sweden
has done so since the Research Bill of 2008. However, these are not coupled to
specific budgets or instruments. Austria’s new RTI strategy (2011) underlines the
need to develop national thematic research and innovation strategies that cut across
ministry boundaries, not least to tackle the Grand Challenges, but does not yet appear
to have devised specific strategies. The Netherlands and the UK have not set thematic
priorities. The Czech Republic has set a number of priorities, which it is the job of the
CRDI to implement.
A debate has raged in Sweden for many years about the balance between institutional
and project funding in the universities. The 2008 Research Bill finally took an explicit
position on this, providing an increase in institutional research funding. The same Bill
accepted that the RTOs’ institutional funding was too low and set an objective of
increasing it from some 10% to about 15% of their turnover, in order to strengthen
their underpinning scientific and technological capabilities. Similar discussions take

place in Norway about institutional funding for both universities and institutes, but so
far without a change in policy.
Generally, governments fund the majority of R&D through institutional rather than
project-based funding. Figure 7 shows the division between institutional and project-
based R&D funding for a number of countries overall government spending on R&D.
(The distinction between institutional and project funding has only recently been
introduced in Eurostat and OECD statistics, so these are incomplete and not as
reliable as some of the more established R&D indicators. Nonetheless, they are
probably good enough to show broad trends.)
Figure 7 Government budget appropriations or outlays for research and development
(2013; million EUR)
Country Institutional
Funding
Project Funding Total Share IF : PF
Austria* 1804.787 668.79 2452.955 73.6 % : 27.3 %
Czech Republic 472.75 532.554 1005.303 47 % : 53 %
Germany 15997.2 9356.8 25114.836 63.7 % : 37.3 %
Greece 445.93 265 710.93 62.7 % : 37.3 %
Iceland 99.338 23.485 122.823 80.9 % : 19.1 %
Ireland* 477 283.4 760.4 62.7 % : 37.3 %
Luxembourg 275.42 16.84 292.261 94.2 % : 5.8 %
Netherlands 3257.785 1344.013 4601.798 70.8 % : 29.2 %
Norway 1684.182 1417.436 3132.143 53.8 % : 45.3 %
Portugal 1231.65 347.35 1579 78 % : 22 %
Slovakia* 231.985 62.721 294.706 78.7 % : 21.3 %
Turkey 1425.625 457.091 1882.716 75.7 % : 24.3 %
Source: Eurostat; *data for 2012
A similar pattern is visible at the level of Higher Education (Figure 8), which shows
how the part of the institutional funding (General University Fund) for research
relates to project funding in the universities. The published statistics used in the
Figure do not include data for The Netherlands or the Czech Republic. However, the
NESTI project of the OECD suggested that the share of institutional funding in Dutch
universities was 79% in 20083, while the National Statistical Office places the Czech
share of institutional funding at 50% in 2012. Most of the funding systems involve a
high proportion of institutional funding in the overall mix. The UK, French, Flemish
and Czech systems stand out as only providing 50% or less of institutional funding for
research to the universities.
3 OECD, Working Party of National Experts on Science and Technology Indicators (NESTI) project on
public R&D funding, 2009

Figure 8 Share of GUF versus direct government funding of R&D expenditure in the
Higher Education Sector, 2009 (in millions of €)
PRFS used?
Government
sector funding
General
university
funds (GUF)
Government
competitive
research project
funding
Belgium Flanders 1,117 36% 64%
Denmark X 1,653 72% 28%
Ireland 704 31% 69%
Germany 7,575 71% 29%
Spain X 3,012 66% 34%
France 7,972 50% 50%
Italy X 5,204 85% 15%
Austria 1,669 76% 24%
Finland X 1,033 58% 42%
Sweden X 2,041 57% 43%
United Kingdom X 5,545 48% 52%
Iceland 55 51% 49%
Switzerland 2,000 82% 18%
Norway X 1,380 73% 27%
Source: Statistics from Eurostat
Figure 9 indicates that most PRFS make up a small component of the overall funding
system for research and higher education – a trend that is clearly visible also among
other PRFS-using countries not considered here. Also noteworthy is that a
considerable proportion of many institutional funding systems remains unconditional.
Figure 9 Current university institutional funding systems
Country Overall institutional funding Sub-components
Austria Block grant (unconditional) 80%
Formula funding 20%
Degree exams 60%
Graduates 10%
External funding 14%
Donations 2%
Cooperation 14%
Netherlands Teaching 45%
Research 55%
Bachelors and masters
degrees awarded
16%
No of PhDs awarded 22%
Block grant 62%
Norway Block grant (unconditional) 60%
Teaching 25%
PRFS 15%
No of PhD candidates 30%
EU R&D funding 18%
National R&D grants 22%
Publication points 30%
Sweden Block grant (unconditional) 90%
PRFS – bibliometrics 5%
PRFS – external funding 5%
UK (England)
2013-4
Teaching formula
51%
PRFS 36%
RAE/REF 67%
PhD supervision 15%
Charity support 13%
Business support 4%

Country Overall institutional funding Sub-components
Research libraries <1%
Czech Republic Formula funding 77.5%
PRFS 22.5%
Notes: Where universities charge fees, these are not included in the table. In the UK case,
charity and business support are incentives intended to compensate universities for the fact that
research charities and business tend to be reluctant to pay university overheads.
The introduction of PRFS in the comparator countries has normally resulted in only
marginal changes to the funding mix. The PRFS are generally seen as useful ways to
add an element of competitive pressure to institutional funding – but only at the
margin. The big exception is the UK, where the RAE started partly because of the
massification of higher education4. Government upgraded the polytechnics and
eventually renamed them as universities but lacked the budget for them all to be
research universities. In fact, it cut the research budgets of the universities and the
research councils in the early 1980s. The university cuts were very uneven, with the
‘establishment’ universities tending to suffer least. This led to a lot of public criticism
and the RAE was in effect introduced in order to make future allocation and
reallocation of institutional funding transparent and therefore defensible.
Experience in the comparator countries indicates that performance contracting brings
with it a significant need for monitoring. The research performers need to report at
least yearly and there tend to be monitoring or ‘dialogue’ meetings at regular intervals
so that both the principal and the RO are clear about the extent to which the
requirements of the contract are being fulfilled. None of the countries that use
performance contracts appears to have clear expectations about the consequences of
sub-standard performance. The use of performance contracts in principle provides the
principal with opportunities to manage the portfolio of activities taking place across
the ROs. However, this ability to manage is generally constrained by the autonomy of
the ROs – especially the universities. For some years, therefore, the Norwegian
education ministry has tried to encourage thematic coordination among the
universities through a dedicated coordination process (SAK), involving dialogue with
and among the universities.
PRFS appear to reduce the need for monitoring, since the principal is in principle only
interested in results, rather than the processes through which the RO produces the
results. However, for those PRFS that – like the UK REF – are only operated at
extended intervals, this does mean that the responsible ministry can only periodically
understand the research performance of the universities.
2.1.5 Effects of funding systems
Performance contracting is generally an incremental change to previous systems for
funding and steering. It therefore tends not to attract a great deal of evaluative
attention. The Austrian system’s initial period of university performance contracting
between 2007 and 2012 has been evaluated. Key findings are that the indicators used
were too complex to have a steering effect. They have since been simplified, giving the
universities a clearer understanding of the behaviour required so that it is simpler for
them to translate this into action.
The Swedish PRFS has yet to be evaluated.
4 Martin, B., Whitley, R. 2010. (Martin & Whitley, 2010)The UK Research Assessment Exercise: a Case of
Regulatory Capture?, in: Whitley, R., Gläser, J., Engwall, Reconfiguring knowledge production: changing
authority relationships in the sciences and their consequences for intellectual innovation. Oxford
University Press, Oxford, pp. 51-80

An evaluation of the Norwegian university PRFS5 found that
• Defined in terms of indicators of journal quality, the quantity of research
publication has gone up (during a time when the Norwegian HE system was in any
case expanding rapidly) but average quality has not
• The proportion of HE faculty publishing has increased rapidly and the share of
publication points going to organisations with an historically weak publication
record has increased – suggesting that the biggest effects of the PRFS have been to
boost research performance in the weaker parts of the university system, rather
than to increase the performance of the leading organisations
• Since the number of publications has been rising faster than the funding
distributed through the system, the ‘price’ of a publication has declined, so the
researchers have to run faster in order to stand still in economic terms
The coverage of the system was extended (with modifications) to the regional health
authorities and research institutes in 2008. An evaluation in 20126 found that
• The volume of papers published in the scientific literature by the institutes grew
rapidly (10% per year) between 2007 and 2011
• There were significant changes in research and human resource management
within the institutes, aiming to achieve such an increase in publication volume.
These included publication incentives for individual authors
• Institutes regarded these changes as leading to increased research quality – but at
the time of the evaluation it was not possible to support this proposition with
harder evidence
• While international income was rewarded by the new funding system, this did not
grow – suggesting that other factors are more important than the funding
incentive in determining the level of this income
• The new system provided incentives for research cooperation with universities,
but these appeared to have no effect on a level of cooperation that was already
rather high
So the effects on the Norwegian system have been more capacity building than
redistributive. At the level of behaviour and management, there have been big effects
on processes intended to increase publication, but apparently no effects on research
quality or on structural aspects of the research system, such as inter-linkages.
The UK RAE is seen as having increased the quality of UK university research,
encouraged universities to develop research strategies and maximised research
returns for limited funding. A reason for the RAE’s success is said to be that there was
a gap of several years between successive exercises, allowing time for universities and
individual researchers to change their behaviour.
The RAE has evolved. In each round, the formula connecting performance to money
was revealed after university submissions were assessed, making it harder to ‘game’
the system. Universities therefore tried to obtain the highest possible grades in all
units of assessment. The proportion of institutional research funding distributed
through the RAE rose. Funds allocation has become non-linear, so that high-ranking
groups get much more money than medium-ranking ones while low-ranking groups
get nothing. Over time, fewer papers had to be submitted per researcher and the
5 Aagaard K, Bloch C, Schneider J. W, Henriksen D, Kjeldager T. R, & Lauritdsen P. S. (2014) Evaluering af
den Norske Publiceringsindikator. Aarhus: Dansk Center for Forskningsanalyse
6 DAMWAD (2012) Evaluering av basisfinsnsieringen til norske forskningsinstitutter som omfattes av
rettningslinjene for basisfinansiering,. Oslo: DAMWAD

universities could decide which members of staff to include in their submissions, so
university research managers were increasingly able to influence faculty organisation
and work patterns. Efforts to include industry in panels failed, owing to the high
workload involved7. Pressure of work meant that, even though panellists ostensibly
reviewed articles submitted by the universities in order to judge quality, they
increasingly relied on journal impact factors as indicators of the quality of articles
submitted for review8.
The RAE has had a number of potentially problematic effects. The RAE made intensive
use of learned societies, subject associations and professional societies to nominate
potential panelists. More broadly, those who publish in and edit high impact factor
journals dominate panels. High impact factor journals are used either directly or via
the intermediate use of a list of highly regarded journals as evidence of high quality for
the purposes of selecting submissions to the RAE. University recruitment and
promotion criteria then adjust towards publication in these journals and, in the case
of the UK, it appears that a ‘transfer market’ of people publishing in such journals
develops ahead of each Research Assessment Exercise.
The bias of the RAE in favour of monodisciplinary, ‘basic’ research is widely
acknowledged. Rafols et al9 show statistically that interdisciplinary research (in this
case innovation studies, which is part of the same ‘unit of assessment’ as management)
is systematically excluded from the highest impact-factor journals, which are
dominated by monodisciplinary management papers. Lee10 shows statistically that
heterodox economics is adjudged poorly in the RAE in competition with the
mainstream, neoclassical school to which members of the Royal Economic Society and
the editors of leading high impact factor journals tend to adhere.
Sastry and Bekhradnia11 demonstrate an almost perfect correlation (0.98) between the
way research funding was distributed in the RAE and through the research council
system, arguing that the same institutions dominate both systems. Barker12 shows that
the relative outcomes of the UK RAE rounds have been rather stable, largely
reinforcing the established order. In the 1996 RAE, 26 of the 192 submitting
institutions got 75% of the money. In the 2001 RAE, 174 institutions submitted returns
and 24 of them secured 75% of the money. Overall university rankings have changed
little. Especially the former polytechnics have not been able to emerge from the ‘tail’ of
the RAE system’s funding distribution. This has meant a reduction of the proportion of
overall university teaching in the UK that is research-led, as those institutions at the
tail of the funding system struggle to raise research funding and thus become teaching
rather than research institutions.
7 Martin and Whitley, Op Cit
8 Bence V & Oppenheim C (2004) The influence of peer review on the research assessment exercise. Journal
of Information Science , 30 (4), 347-368
9 Rafols I, Leydesdorff L, O'Hare A, Nightingale P & Stirling A (2012) How journal rankings can suppress
interdisciplinary research: A comparison between Innovation Studies and Business Management.
Research Policy, 41 (7), 1262-1282
10 Lee, FS, Pham, X and Gu, G, (2013) ‘The UK Research Assessment Exercise and the narrowing of
economics,’ Cambridge Journal of Economics, 37, 693-717
11 Sastry, T and Bekradnia, B (2006) Using metrics to allocate research funds, Oxford: Higher Education
Policy Institute
12 Barker K. E. (2007) The UK Research Assessment Exercise: the evolution of a national research
evaluation system. Research Evaluation2007 , 16 (1), pp. 3-12

2.1.6 Some lessons
Evidence from the international context suggests that both the UK PRFS and the
evaluation methodology being implemented in the early 2010s in the Czech Republic
were outliers in terms of their radicalism. The Metodika 2013-2015 is in some
respects a more mainstream solution.
The comparisons in this section suggest that
• The decentralised nature of innovation systems and their governance means that
the mix of policies and funding instruments of systems tends to be emergent’.
That is, it generally results from the interplay of many independent policy actors’
decisions. It is generally hard to coordinate at any level, despite the fact that there
is a genuine need for coordination
• Correspondingly, changes such as the introduction of a PRFS tend to be
incremental. They generally affect only a small part of the overall mix. Our
impression is that the systems that only drive a small part of the institutional
funding nonetheless are effective at changing behaviour and performance. They
are likely to be less contentious because design flaws only has small effects on the
system as a whole
• All the research systems we consider are dominated by the Higher Education
sector. This is less so in the Czech Republic because of the continuing and large
role of the ASCR and this provides a rationale for maintaining a research
assessment and funding system that spans both the university and the scientific
institute sectors
• Elsewhere, RTOs and PSROs do not receive institutional research funding through
the same type of PRFS as other actors. These have rather different research
missions compared with the university and scientific institute sector. Bringing
them together in a single PRFS must be done with some care. Even in the
Norwegian system, which combines all three types of institute, they tend to be
separated into separate ‘competition arenas’ within a uniform PRFS
• Performance contracting is widespread – not only on its own but also in
combination with a PRFS. This involves a burden of monitoring but that is also
part of a necessary dialogue between ROs and their principals and supports the
principals in the implementation of policy
• The comparator countries chosen for this study have well-established research
performance systems and do not have to struggle with the changes needed in
transition states such as the Czech Republic. Transition means that more
decisions have to be made about which new institutions to admit to the
institutional funding system. The international experience is that a bottom-up
approach to admission fragments the research-performing system. Admitting a
new RO to the institutional funding system is inherently a policy decision which
should therefore be taken by the policy system (ministries and Council) that acts
on behalf of society
• Beyond this, however, the need for institutional autonomy implies that principals
should as far as possible use incentives to encourage their agents to attain goals,
rather than try to micro-manage them
• International experience with PRFS is that they tend to increase both the quantity
and the quality of research outputs. It is nonetheless easy to encourage perverse
effects

2.2 The current institutional funding system in the Czech Republic
This chapter summarises the main findings of our analysis of the current governance
system and funding principles in the Czech Republic (CR), including a SWOT analysis,
conclusions, recommendations and main lessons learned from comparison of the
Czech system with above mentioned countries. The full analysis is in the Background
report on the Institutional funding system in the CR (separate documet).
2.2.1 Governance of the institutional funding for research system
The institutional funding for research system is rather decentralised with a high
degree of autonomy for both funding providers and recipients (i.e. the research
organisations).
The RD&I Council is the central co-ordination body. Legally, the Council is an advisory
body of the Government. The International Audit considered that the Council is
increasingly taking the role of executive authority in the RD&I system. Compared to
international practice, the Council has a high degree of autonomy and covers a wide
range of activities, from setting the draft of the total amount of the state budget for
R&D to the formulation of a long-term RDI policy and strategic support to the
Government in the field of R&D. In international practice, entities similar to the
Council have as major task to establish a general framework and principles for the
national RD&I system, while the implementation of specific actions is the task of other
(lower) government/state authorities. In the CR, instead, the Council lays down the
general principles and takes up also their implementation.
The adoption of the Reform of the RD&I system in 2008 reduced the number of
providers of institutional funding and currently, seven ministries are responsible for
this task. In addition, the Academy of Sciences (ASCR) currently acts as the funding
provider for its own research institutes.
The recipients of the institutional funding for research form a heterogeneous group. In
2013, 158 research organisations received institutional funding for research. The
increasing number and wide spectrum of the research organisations supported risks
posing a problem for further dispersion of institutional funding in the future.
Currently, the system for allocation of institutional funding does not reflect the type
and different missions of research organisations.
2.2.2 Methods of institutional funding allocation
Institutional funding is a component of the RD&I state budget expenditures. Two
expenditure chapters determine the annual public RD&I state budget:
• The expenditure for competitive funding, i.e. the funding of on-going and
approved new programmes
• The ‘institutional funding expenditures’ chapter, covering the institutional funding
for research, defined by the results of the ‘Evaluation Methodology’ (further
called: Metodika); the obligations for co-funding of EU programmes and other
international obligations; rewards for exceptional research results; and the
operational costs for the agencies, the ministries and the RD&I Council
In 2014, institutional funding for research accounted for about 73% of the total
institutional expenditures chapter and 35% of the total public RD&I state budget
expenditures. Supposing that 20% of the institutional funding for research was
distributed based on the results of ROs evaluation (application of the Metodika)
carried out in 2013 (according to RIV points, that sum amountedat CZK1.88bn., i.e.
7% of the total public RD&I state budget expenditures.
The preparation of the RD&I state budget expenditures, done by the RD&I Council, is
a political process by nature. According to the Section 5a of the Act No. 130/2002

Coll., the proposal for the total expenditure package for research, development and
innovation is based on an evaluation of the results attained by research organisations
over the past 5 years, on the National Research & Development and Innovation Policy
of the Czech Republic and on the results of an international assessment of research
and development in the Czech Republic. Nevertheless, interviews with representatives
of all providers and the Government Office13 showed that the total amount of
institutional funding allocated to each provider is based more on political decisions
(i.e. on the basis of negotiations within the RD&I Council, between providers and the
Ministry of Finance, among providers as well as between rectors, the President of the
ASCR and the Prime Minister) than on the aggregate research outputs (RIV points
generated by the research organisations). This contradiction between the act’s section
and the actual process of the RD&I state budget expenditures drafting has been
intensively discussed within The Council of Higher Education Institutions14 of the
Czech Republic and the Czech Rectors’ Conference. Both associations have requested
the Government to draw up the RD&I state budget expenditures in line with the Act
and in a transparent manner so that the level of institutional funding corresponds with
the results of the research organizations evaluation (the number of RIV points)15.
According to Act No. 130/2002 Coll., providers of institutional funding have a
relatively high level of autonomy for the allocation of institutional funding to the
research organisations within their competence. They may decide on the amount of
institutional funding for ‘their’ research organisations based on the outcomes of their
own assessment methodology. However, in most cases providers do not make use of
this power. The reason is that either they have not created their own methodology for
evaluating research organisations, or they do not have the personnel, financial and
time capacity for conducting their own evaluation activities. Except for the Academy of
Sciences (ASCR), the funding providers claim that they often use the system of RIV
points for the redistribution of institutional funding to the research organisations
within their competence (based on the proportion principle – “rule of three”).
Currently, 80% of the institutional funding is allocated to research organisations
according to the level of institutional funding in the previous year (i.e. it is distributed
according to performance in 2007-2012 measured by previous variants of the
Metodika), and the remaining 20% is allocated based on the performance (RIV points)
of research organisations in terms of research outputs.
The ASCR has its own methodology for internal evaluation of its research institutes.
The results of the latest internal evaluation in 2011 were used for the allocation of
institutional funding among the research institutes of the ASCR in 2012 and 2013. For
each institute, institutional funding consists of the base funding (three quarters of the
funding) and the PRFS, i.e. funding allocated on the basis of the evaluation results
(quarter of funding).
2.2.3 The amount, distribution and utilisation of institutional funding
The system of institutional funding for research has undergone significant changes in
recent years as a result of the Reform of the RD&I system in the Czech Republic in
2008, including the transition from a research intentions funding system to the
current performance-based model. One of the main consequences of the Reform and
13 Their list is in an annexe of the Background report on the Institutional funding system in the CR
14 Minutes of the 20th meeting of the Board of The Council of Higher Education Institutions taking place on
October 23, 2014 http://www.radavs.cz/clanek.php?c=1637&oblast=16
15 Information on the 9th meeting of the Assembly of the Council of Higher Education Institutions from
November 20, 2014, and the main resolutions adopted
http://www.radavs.cz/clanek.php?c=1648&oblast=17, http://crc.muni.cz/resolutions/128.html

the National RD&I Policy (Measure A3-2) was a relative decrease in institutional
funding, while competitive funding has become more important.
Eight providers distribute institutional funding for research. The Ministry of
Education, Youth and Sport and the ASCR provide the largest amount of institutional
funding; these providers together distribute more than 85% of total institutional
funding.
The total amount of institutional funding has grown (in the absolute value) in recent
years with some differences among providers. Except for the Ministry of Industry and
Trade, the institutional funding of all providers increased considerably in 2011-2014.
In 2013-2014, there was an inter-annual decline of institutional funding allocated to
the ASCR, while the institutional funding assigned to other providers increased.
At the research organisations level, the lion share of institutional funding for research
goes to universities (52.4%), followed by the ASCR institutes (34.2%).
The Act No. 130/2002 Coll. gives research organisations a high level of freedom in
terms of institutional funding utilisation. Generally, institutional funding aims at
funding the long-term development of research organisations. Specifically, the funding
can be used for different types of RD&I cost set by the Act, i.e. personnel cost,
investment cost, operational cost, cost of external services and overheads. In reality,
the institutional funding is used for many activities; the key activity is stabilisation of
research teams (it covers in particular personnel costs).
Institutional funding plays different roles in different types of research organisations.
As for the significance in the total funding, it is the most important public funding
instrument for the public research institutes (organisational units of the Czech
Republic - 59% of their public funding) and for the ASCR institutes (48% on average).
Since the institutional funding is for many of the research organisations a significant
funding stream, there is a general effort to maximise the number of RIV points, which
establish the basis for computation of institutional funding. This effort could affect
strategic behaviour of research organisations negatively, as it could fosters gaming and
an adoption of the behaviour of many researchers towards simple maximisation of
RIV points.
The analysis of data on research organisations funding (data from the RD&I
Information system) shows that the amount of institutional funding allocated to
individual research organisations may vary considerably between years as a result of
the annual evaluation of research organisations. Smaller research organisations are
especially vulnerable to such fluctuations.
2.2.4 Evaluation and control mechanism
Although the former research intentions system was criticised because of the very
formal nature of annual and ex-post evaluation, the control and evaluation
mechanisms of the current funding model seem to be problematic as well. Main
weaknesses can be summarised as follows:
• The evaluation and control system is insufficiently and ambiguously defined in the
national legislation.
• The evaluation of institutional funding at the highest (governmental) level is
missing.
• Many providers complain of insufficient personnel capacities for developing and
implementing their own evaluation system.

2.2.5 Impact of the transition from the research intentions system to the current
institutional funding system
Impact is identified on the basis of interviews with representatives of providers and
research organisations.
The following key impact on providers was found out during interviews.
• Fewer employees were needed to deal with the institutional funding
• Increased autonomy in terms of evaluation and distribution of the institutional
funding
• No power to influence the research topics at research organisations
• Limited decrease of administrative requirements (in terms of yearly evaluation,
assessment of research intentions, etc.)
Impact on recipients can be summarised as follows.
• Limited decrease of administrative requirements (in terms of elaboration of
research projects proposals, reporting, etc.)
• Increased degree of autonomy in terms of how to use the institutional funding
• Increased level of freedom to define research topics and find funding for them
• Increased competition among supported research organisations as well as within
them
• The strategic actions and behaviour of many research organisations and the
researchers aim at simple maximisation of RIV points
The quality and scope of RD&I activities are outcomes of the complex RD&I system,
whose development has been formed by the Reform of the RD&I system, increased
total RD&I budget and the decisions of many stakeholders. Thus, the impact of the
current institutional funding system cannot be seen in isolation from this broader
context. Taking these factors into consideration the following impact was identified:
• Increase of the number of research organisations supported;
• Research organisations gradually increase the quality of their research;
• Modernisation of research facilities (new equipment, modernisation of
laboratories, etc.);
2.2.6 SWOT analysis
The SWOT analysis summarises and sorts out the main conclusions. For better clarity
the analysis is divided into two parts. The first one deals with the overall governance of
the institutional funding system and the roles and power of the funding providers. The
second part focuses on the research organisations as institutional funding recipients.

Figure 10 R&D Governance
Strengths Weaknesses
• The existing institutional funding system is
simpler in comparison with the previous
research intentions funding model and peer
review-based systems used abroad.
• The current institutional funding system is
slightly less administratively demanding
than the previous research intentions
funding model.
• The Act no 130/2002 Coll. (amended the
Act no 211/2009 Coll.) gives providers a
high degree of autonomy in terms of
institutional funding distribution (including
evaluation).
• There is only one research evaluation
methodology valid for all providers (with
the exception of the ASCR).
• Financial and accounting control of
institutional funding is defined by law (Act.
320/2011 Coll., Act no 130/2002 Coll).
• Existence of a national RD&I Information
System, which contains information about
all RD&I outputs generated thanks to RD&I
public support (institutional as well as
competitive).
• The current institutional funding system
creates a competitive environment
stimulating research organisations to
higher performance.
• Existence of the RD&I Information System
containing information on the RD&I public
support (institutional as well as
competitive) and all reached research
results.
• There is no official central co-ordination of the
institutional funding system (no executive body is
responsible for co-ordination).
• The size of the institutional expenditures budget
chapter does not match the results of the evaluation
and previous commitments, and is tstrongly
influenced by negotiations with the Ministry of
Finance and within the Government.
• Methods for the evaluation of research organisations
and distribution of institutional funding (according to
the Metodika) do not reflect differences among
scientific fields, research organisations (including
different missions), RD&I outputs generated,
providers, etc.
• Not all rules, responsibilities and activities of
providers are defined by law - or only in a very general
way.
• Providers do not have internal strategies for their
distribution of institutional funding.
• Frequent changes of the RIV point numbers assigned
to individual types of research results used in the
Metodika have negatively affected the stability of the
whole system.
• Due to the low personnel capacity in R&D
departments, the providers have only limited
resources to develop their own methodologies for
evaluation and distribution of institutional funding,
although the Act no 130/2002 Coll. gives them this
power.
• The RD&I Information System does not contain
detailed information on ROs supported by RD&I
public expenditure (e.g. no information on numbers of
researchers).
• Information sources on RD&I public funding and ROs
provides information, which are not identical and
comparable in many cases (e.g. there are some
differences between the same data from the RD&I
Information System, annual reports of ROs and data
provided by the Czech Statistical Office). .
• Unusual use of single evaluation methodology and
funding principles for universities and other research
organisations.
• In comparison with foreign countries, the number of
institutional funding recipients (supported research
organisations) is relatively high in the CR, and their
variety is very large in terms of size, mission and types
of RD&I activities performed.

Opportunities Threats
• Upcoming amendment of the Act no
130/2002 Coll. could create an opportunity
for improving the institutional funding
system.
• The “IPN Metodika” project will develop a
new evaluation methodology and propose
new funding principles for institutional
funding of research organisations.
• Lack of political will to change the funding system.
• Demands of individual ministries (providers) and
political discussions could negatively affect the
upcoming amendment of the Act no 130/2002 Coll. in
terms of setting up a more effective institutional
funding system.
• Unsystematic interventions of politicians and
ministries in RD&I budget drafting.
• The need to reduce public spending may reduce the
amount of institutional funding.
• The need to reduce employment in the civil service
will not allow for employing staff responsible for
improved management of institutional funding at the
providers’ level.
The current institutional funding system shows a number of strength factors.
Compared to the previous research intentions funding system, it is relatively simple
and slightly less administratively demanding. It creates straightforward rules for
providing institutional funding. First of all, institutional funding is provided to all
research organisations that meet the criteria for being recognised as research
organisations, which are rather formal. Second, research organisations do not have to
elaborate any project proposals; instead, institutional funding is provided on the basis
of evaluation of their research outputs. Therefore, no need for the providers to invest
in, e.g., the appraisal of the project proposals (research intentions) or changes in the
projects, the definition of eligible activities, expenditures, and results expected, or the
involvement in the annual assessment.
Act No 130/2002 Coll. guarantees providers a relatively high degree of autonomy in
terms of institutional funding allocation and distribution. The providers can develop
own evaluation methodologies and set up their own funding principles. However, with
the exception of the Academy of Sciences (ASCR), all providers use the Metodika for
the evaluation of their research organisations and the computation of institutional
funding. The Metodika constitutes a relatively simple institutional funding system
with uniform rules for all providers on the one hand and all recipients on the other.
The Metodika uses information on RD&I outputs generated thanks to RD&I public
support, which is collected in the national RD&I Information System. Compared to
other countries, the information system contains more complete information on RD&I
public support.
These strengths are accompanied by many weaknesses, which have affected
negatively the whole institutional funding system. The crucial weakness of the
institutional funding system, as defined in the Act, is the absence of de-jure central co-
ordination body. There is no official body responsible for governance of the whole
system. In fact, the Council plays this role; it is an advisory body of the Government
without real executive power. However, the Act gives the Council a relatively extensive
power, since it is in charge of drafting RD&I state budget expenditures, the
development of the research organisations’ evaluation methodology, and the
implementation of the evaluation.
The fact that the Council is only an advisory body, without strong political backing, is
reflected in the negotiations on the amount of RD&I state budget expenditures, their
distribution among providers, and more specifically in the amount and distribution of
institutional funding. Consequently, the negotiations with the Ministry of Finance and
within the Government result in the definition of an amount of institutional funding,
and allocation to individual providers, which does not correspond to the evaluation
results or to previous commitments for the funding of RD&I activities. If institutional
funding was distributed proportionally according to the evaluation results (RIV
points), some providers would receive more funding (the Ministry of Education, Youth
and Sport and the Ministry of Health), while the allocation for the other providers

would be lower.16. At the recipients level, it means that some ROs receive lower
institutional funding that they should get according to the number of RIV points they
generated (e.g. although universities generate 63% of RIV points, their share on
institutional funding reaches only to 55%).17 Nevertheless, as regards drafting of RD&I
state budget expenditures including the distribution of institutional funding, the Act
No. 130/2002 Coll. does not explicitly say that institutional funding should be
distributed among providers proportionally according to the RIV points. Thus, it
creates space for various political discussions and negotiations.
The providers are not able to fully utilise their autonomy and have rarely developed
specific strategies for institutional funding distribution. This is mainly due to an
underemployment in the providers’ departments responsible for RD&I support and a
lack of financial sources for purchasing specific services (or for the employment of
experts). In most cases, providers only carry out activities that are strictly defined in
the Act, as they do not have human as well as financial resources for other activities
(especially for conceptual tasks and evaluation).
Another weakness is that many activities are defined only in a general way. This
relates to monitoring and evaluation carried out by providers. Although financial and
accounting controls of institutional funding are rigorously carried out (control is well
defined in the laws), monitoring and evaluation activities are in some cases done only
formally or not at all (again because of the lack of personnel capacities). Only the
ASCR has created a system for monitoring and evaluation, which corresponds to
international standards.
The system for allocating institutional funding among research organisations based on
RIV-points does not reflect differences in the role (mission) of research organisations
and types of research activities. Moreover, there have been almost yearly changes of
the Metodika namely in terms of the number of RIV points assigned to individual
types of research results (e.g. RIV points numbers assigned to proceedings papers
published in the English language changed from 4 in 2006, to 0.2 in 2007, 8 in 2008
and 8-60 in 2013) and their definitions. In addition, the RIV point value of some types
of results (e.g. cultivars) may be questionable, as some interviewed stakeholders
stated. The Metodika influenced individual scientific fields in a different manner.
Preferences of publications and papers in scientific journals indexed in the Web of
Science in general encourage fields whose main results are scientific papers in Web of
Science indexed journals (in this sense ROs and especially faculties of natural sciences
performing higher quality research outputs received higher institutional funding),
whilst the influence on medical and technical sciences (characterised by different types
of results) is rather problematic. The Metodika has a negative impact on the quality of
the results18 in social sciences and humanities.
The Metodika used only data from the RD&I Information System, which contains
relatively detail information on public funding and all research results created through
the RD&I public support. (The system also provides some information on providers,
ROs, programmes and completed research intentions.) Nevertheless, the system does
not contain information on a number of researchers (work-loads, FTE numbers) in
evaluated ROs. This information can be obtained from ROs’ annual reports and the
16 Prof. Opatrný’s presentation on funding of research, 20th meeting of the Board of The Council of Higher
Education Institutions taking place on October 23, 2014,
http://www.radavs.cz/prilohy/20p5_EIPRVS231014.pptx
17 Minutes of the 20th meeting of the Board of The Council of Higher Education Institutions taking place on
October 23, 2014 http://www.radavs.cz/clanek.php?c=1637&oblast=16
18 Münich, D. (2014): Dobrý a zlý kafemlejnek II, http://metodikahodnoceni.blogspot.cz/2014/12/dobry-
zly-kafemlejnek-ii.html

Czech Statistical Office’s surveys (databases). However, usage of these data for
potential ROs evaluations or setting of an amount of institutional funding is rather
limited, because the data from these sources (not only data on researchers’ numbers
but on funding as well) differ substantially compared with other sources, probably due
to different reporting methodologies or some mistakes in reporting.
Other weaknesses appear when comparing the Czech system with international
practice:
• In international practice, there are different evaluation methodologies and
funding principles for universities and other research organisations.
• In international practice, research organisations focused on basic research
activities have higher share of institutional funding (Max Planck institutes, NIH,
Austrian Academy of Sciences), whereas organisations aiming at applied research
and cooperation with a private sector have lower shares of institutional funding
and the majority of their financial sources comes from projects and contracts
(Fraunhofer institutes, VTT Technical Research Centre of Finland).
Figure 11 Recipients
Strengths Weaknesses
• The current institutional funding system has
created a more competitive environment
stimulating researchers and research
organisations to higher research performance.
• Research organisations have developed their own
internal mechanisms for distribution of
institutional funding.
• By law, research organisations have a relatively
high degree of autonomy in terms of
management, distribution and use of institutional
funding.
• Some research organisations (large organisations
and universities) have their own internal audit
units and carry out evaluation of research.
• Short (annual) cycle of research organisations
evaluation and distribution of institutional
funding does not allow for developing long-term
strategic plans within research organisations.
• Usage of the outcome that come out from the
evaluation based on the Metodika for strategic
managements of research organisations is rather
limited because of the general and summative
character of the evaluations.
• The Metodika and its funding principles had
negative impact on the behaviour of researchers,
who adapted their outputs to the Metodika in
order to maximise the number of RIV points.
• Usage of institutional funding is bound to year-
long state budget cycles.
Opportunities Threats
• The upcoming amendment of the Act no
130/2002 Coll. and the Higher Education Act can
create favourable conditions for use and
distribution of institutional funding within
research organisations (including internal
evaluations of research and research teams).
• Research organisations will implement principles
of professional management at all levels.
• Research organisations will develop an evaluation
and monitoring system resulting in more effective
use of institutional funding. Foreign experts will
be involved into the evaluation.
•
• Ongoing conservative attitudes of researchers
towards changes may create resistance towards a
new evaluation methodology, or cause delays and
tensions with possible negative impact on the
actual research..
• The need to reduce public expenditures may
negatively affect the amount of institutional
funding.
• Providers may lay down too detailed or rigid
conditions for the allocation and use of
institutional funding.
•
Compared with the research intentions funding system, the existing institutional
funding system has created a more competitive environment, which stimulates
research organisations and researchers to produce higher quality and/or more RD&I
outputs. Nevertheless, the Metodika is often used not only at the level of the research

organisation but at the lowest level – the level of individual researchers – as well. In
this case researchers are rewarded according to the number of RIV points they
generate.19 Consequently it affects their behaviour in a direction of adaptation of their
outputs to match the Metodika; it also creates a drive for gaming.
Frequent changes of the Metodika have resulted in a situation where the research
organisations management do not know, up to the very last moment, the amount of
institutional funding for the next year. This creates a barrier for elaboration and
implementation of long-term strategic plans. Use of institutional funding for long-
term activities (including large investment activities) is limited because the funding is
bound to the year-long state budget cycles.
Only some universities, ASCR institutes and some large research organisations have
own internal mechanisms for institutional funding distribution. Some research
organisations (namely ASCR institutes and large research organisations) have own
internal audit units and carry out own evaluation of research and research units. Very
comprehensive evaluation system is in ASCR institutes. The ASCR evaluation system
covers all institutes and is carried out centrally in five years cycles. The ASCR has also
an attestation system, which assesses each researcher (this is organised at the level of
institutions).
Institutional funding is provided in good synergy with other funding sources of
research organisations, especially with operational programmes (co-funded by EU
Structural Funds). Whilst institutional funding concentrates rather on stabilisation of
research teams (wages) and operational costs, operational programmes ensure capital
expenditures (purchase of new equipment, modernisation, new buildings, etc.).
Future development of institutional funding will depend on the upcoming amendment
of the Act no 130/2002 Col. and the Act no 111/1998 Coll. (Higher Education Act). The
amendment could remove or at least reduce uncertainty about the amount of
institutional funding for the respective organisations. One of the Metodika project’s
outputs is a proposal about what should be changed in the acts and how the new
evaluation and funding system should be incorporated.
In any case research organisations could implement principles of professional
management, develop monitoring and evaluation systems corresponding to
international standards and benefit from synergies arising from new operational and
international programmes.
Based upon the SWOT analysis, the following recommendations should be taken into
account when developing the new evaluation methodology and funding principles:
1. The amount allocated to providers should be in line with the centrally defined
funding principles, common to all providers.
2. Institutional funding should not be provided based on an annual evaluation. The
evaluation and the resulting amount of institutional funding should be arranged
and provided for a longer period. Similarly, the evaluation methodology and
funding principles should not change each year (or every two years) but be fixed
for a longer period.
3. The evaluation methodology and funding principles should consider qualitative
aspects of RD&I activities performed and take into account the different missions
19 Žížalová, P., Čadil, V., Pokorný, O., Kostić, M. (2011): Podpora vytváření strategií zaměřených na realizaci
výsledků VaV v praxi a ochranu duševního vlastnictví a motivace spolupráce s aplikačním sektorem. Studie
TCAVČR. http://www.vyzkum.cz/storage/att/18C0C6E0BEDC982432D14A62BD739099/A%204-
1%204-2%20Strategie%20a%20motivace%20TT.pdf

of the research organisations. In this sense, a combination of performance-based
funding and research contracts could be desirable.
2.2.7 The Czech institutional funding system in international comparison – main
lessons
Based on the comparison of institutional funding systems in five countries with the
Czech institutional funding system, we can draw following key conclusions that should
be taken into account while drafting revised funding principles for the Czech Republic.
• Unlike the comparator countries with rather stable research performance systems,
the Czech Republic struggles with an increasing number of ROs eligible for
institutional funding and also with a high fragmentation of the whole research
system. This situation results from the bottom-up approach to admission to the
institutional funding system that is in place in the Czech Republic where research
organisations irrespective of their type can apply to the RDI Council through the
respective funding provider to become eligible for institutional funding. Eligibility
is currently assessed against the definition of RO set by the European and Czech
legislation. Although the eligibility of ROs for institutional funding does not
formally entitle them to this type of funding, in fact all eligible ROs receive
institutional funding according to their past research performance.
Lesson 1: Admitting ROs to the institutional funding system is a policy decision and as
such should be taken by policy makers at the highest level in the R&D system, i.e.
ministries and the Council. An evaluation methodology or institutional funding
principles cannot substitute for or replace the policy decision.
• Despite the fragmentation of the Czech research system and heterogeneity of ROs
in relation to their mission, there is one single evaluation methodology and set of
funding principles for universities, institutes of the Academy of Sciences and for
other research organisations – all are evaluated according to the same
methodology and funded according to the same principles. This is a rather unique
system in international comparison. In addition, the current system in the CR
mirrors a system that is typically applied for research councils (and thus, scientific
research), i.e. a primary division of the overall budget into scientific fields. In the
five comparator countries, universities, governmental labs and RTOs do not
receive institutional research funding according to the same principles. In the case
of PRFS there is a strict differentiation between different types of ROs in order to
ensure competition among ROs with the same (similar) function in the research
system.
Lesson 2: The Evaluation Methodology and the funding principles should be designed
taking into account the missions of the research organisations in the RD&I system and
avoid competition between ROs with different missions.
• The fundamental purpose of institutional funding is a long-term strategic
development of ROs. Therefore, the institutional funding systems in comparator
countries are by nature stable and the introduction of any changes tend to be
incremental. The Czech institutional funding system has undergone several rapid
changes over the last 10 years that have negatively affected the stability and
predictability of the funding, especially in smaller-sized ROs. Only recently,
following the International Audit, dampening factors have been introduced in
order to avoid major annual differences in institutional funding.
Lesson 3: Changes in the institutional funding principles should be introduced
incrementally in order to provide sufficient time for adjustment on both sides –
principals (funding providers) as well as agents (ROs).
• Institutional funding systems in comparator countries tend to consist of several
components combining PRFS with other schemes of funding like performance

contracts or block grants. The performance contracts may not even be linked to
funding. Although the significance of single components tends to differ among
different types of ROs the PRFS component is typically relatively small and
consequently it affects only a small part of the total funding for a given RO. The
PRFS is nonetheless effective in changing behaviour and performance of ROs. In
the Czech Republic, there is a single system to allocate institutional funding across
all types of ROs. This system consists of a funding component based on past level
of institutional funding and a PRFS component. The uniformity of criteria for
PRFS component of the Czech institutional funding scheme for all types of ROs
does not take the different role and modus operandi of ROs into account. In
addition, it cannot distinguish the role of the institutional funding for ROs of
different focus (i.e. highly important for basic research, far less so for applied
research ones).
Lesson 4: Introduction of several components of institutional funding allows
combining elements of continuity, stability and incentives for a desirable change in
behaviour. It also enables differentiation of institutional funding schemes for different
types of ROs.
• Performance contracts play an important role in the institutional funding systems
in the comparator countries as instruments for dialogue between ROs and their
principals. They are used to encourage ROs to achieve agreed goals that are in line
with the principal’s policy goals. Inevitably, the performance contracts require
monitoring activities from the ministries/funding providers. The institutional
funding providers in the Czech Republic play a less active role compared to similar
funding organisations in the comparator countries, when it comes to defining long
term strategic directions for research conducted by ROs that the funding providers
are responsible for. Except for the ASCR the providers allocate institutional
funding among the ROs solely based on previous results (RIV points) and without
any future perspective on desired research goals.
Lesson 5: Performance contracts are suitable instruments for activation of a dialogue
between ministries / funding providers, and ROs. Especially, they enable to align the
ROs’ goals with the principal’s policy goals.

3. The revised funding principles for the Czech Republic
This section provides answers to a set of research questions drawn up in the terms of
references. We list them in the following table and direct the reader to the chapter
where we answer these research questions.
Figure 12 Research questions and location of answers
Research question Chapter with
(main) answers
Establish principles for deciding the amount and distribution of
institutional research funding among research funders so as to remove
cross-subsidy to or from other sources of institutional funding (eg for
education)
3.2.1, 3.1.3
Establish coordination mechanisms among research funders to ensure
adequate field coverage and efficient sharing of national and
international research infrastructures
3.1.4, 3.1.5
Establish principles for funding to reflect changing national priorities
while maintaining some stability of institutional funding
3.1.6
Recommend how much money to distribute by formula funding versus
performance contracts
3.2.2, 3.2.3, 3.2.4
Draft rules for allocating formula funding, based on the results of EuV
assessments (WP2)
3.4
Draft a template for performance contracts for different types of
research organisation
3.3.1
Determine how the reporting will work and what sanctions the
ministry can use in order to enforce the contracted performance
3.3.2
Specify a procedure and criteria for deciding when and how to allow
new research organisations to enter the funding system
3.5
As can be seen from the study questions, we were tasked to design a funding system
based on the new Evaluation Methodology, and devise a funding formula based on the
results of the assessments. The new funding system is to be applied for the
“institutional support of the long term conceptual development of research
organisation on the basis of an evaluation of the results“ which is one of several
institutional research funding streams in the Czech Republic. Hence, our concrete
suggestions and recommendations for a funding system to be applied in the Czech
Republic cover this part of institutional funding only (called “institutional funding for
RO” for the sake of brevity). When we speak of institutional funding more generally,
e.g. about its function in the research system, we use the term institutional funding.
In addition to a funding formula based on the results of the assessments, we were also
asked to include performance agreements. Like the new Evaluation Methodology, the
new funding principles are to be valid for all types of research organisations. Based on
this brief, in this chapter we are suggesting funding principles for all, while taking into
account the specifics of different research organisations.
The framework we are suggesting can accommodate various scenarios i.e. different
priorities of R&D policies related to institutional funding for RO. Our basic scenario is
geared towards increasing overall research quality, and giving every research
organisation (and research unit) a chance to get better. This is against the background
that research organisations and research units (RUs) may not do well in the first
evaluation round for various reasons, not least among them that they are unfamiliar
with peer-review based evaluation, and that these reasons ought to be looked into.
However, the framework we are suggesting can also accommodate a scenario for

increasing excellence and societal relevance and a scenario for phasing out
institutional funding for RO to weak research organisations.
In chapter 3.1, we describe the context to the principles for institutional funding for
RO, discussing general concepts for institutional funding (3.1.1), the specific starting
point in the Czech Republic (3.1.2), cross-subsidies between different sources of
institutional funding (3.1.3), adequate field coverage (3.1.4), efficient sharing and use
of national and international research infrastructure (3.1.5) and the need for stability
of institutional funding against the background of changing national priorities (3.1.6).
We then go on to present the core funding principles (3.2), before discussing
performance agreements (chapter 3.3) and the performance-based research funding
system (chapter 3.4) in more detail. Chapter 3.5 describes how new research
organisations can enter the funding system.
3.1 Context to the principles for institutional funding for RO
3.1.1 General concepts for the institutional funding for RO
Institutional funding can be conceptualised as a long-term working relationship
between two organisations: a funding body or provider (the principal) and a research
organisation (the agent). In other words, public institutional funding expresses
responsibility by the state, with the state taking an interest and a stake in the research
organisations it (co-)funds. This aspect seems to be particularly important in the
Czech Republic where the public sector’s ‘ownership’ of research organisations can be
weak. This also implies that context and history have to be taken into account and that
institutional memory is important on both sides, the principal’s and the agent’s.
Rationales commonly cited for granting institutional funding are that it allows
research organisations to behave in a strategic, long-term oriented fashion, reducing
the risk of converging on ‘hot areas’ with a consequent loss of diversity in the wider
research system. It also provides space for researchers to develop ideas which may not
be ready yet for exposure to external competition. In many countries, institutional
funding also pays for equipment and support services of generic benefit that would not
be easily attributable to individual grants.
The most common understanding for institutional funding is that it needs to provide
continuity, stability, sustainability and resilience for institutional development, and
that a long term shortfall of institutional funding leads to a ‘hollowing out’ of research
organisations. 20
In contrast, competitive funding ensures quality and relevance of research through a
peer review or other quality-based selection process and allows the implementation of
selectivity strategies (i.e. only funding the best or certain topics) and can be adapted to
influence behaviour in particular directions such as collaboration or towards thematic
priorities. It has a shorter timescale and increased granularity and thus allows flexible
application of resources as science develops. It incentivises researchers and prevents
senior hierarchy from dominating resources. An advantage of competitive funding is
its straightforward line of accountability for use of resources. However, it is generally
more expensive to allocate than institutional funding as grants involve high
transaction costs in preparation and review.21
20 Luke Georghiou, Effects of Public Funding Systems on Research in the United Kingdom. Presentation at
‘The Impacts of Impact Measurement’, Vienna, 25 September 2014
21 Luke Georghiou, Effects of Public Funding Systems on Research in the United Kingdom. Presentation at
‘The Impacts of Impact Measurement’, Vienna, 25 September 2014

The significance of the different sources of funding (institutional, competitive) differs
for different types of research organisations because they have different missions and
their research is embedded in different contexts, addressing different user
communities (other researchers, industry, public sector etc.). Furthermore, for these
and a number of additional reasons such as location or subject specialisation, different
research organisations do not have equal access to competitive funding. Hence, an
important principle must be that the funding system needs to take into account the
different missions and specifics as well as the whole funding situation of research
organisations. Ideally, the responsible ministries base their funding decisions on a
thorough understanding of the research organisations’ ‘funding profiles’, i.e. a
complete overview and analysis of the various funding sources per research institution
(including competitive funding). This is necessary to understand how changes in
institutional funding for RO affect a research organisation’s budget as a whole.
The share of institutional funding in relation to the overall budget is different for
different types of organisations. The rule of thumb is: the closer a research
organisation is to the market, the less institutional funding it receives (see for example
the Fraunhofer Society compared to the Max Planck Society in Germany). In the end,
it is a policy decision what share of institutional funding a research organisation
should get. In many countries, there are various arrangements at work, making the
institutional funding landscape quite ‘messy’. However, in most countries this is
considered the normality of a differentiated multi-level and multi-actor science and
research system.
Experience from other countries shows that any reform of institutional funding takes
time: a preparation phase is followed by the first implementation of the new funding
system, and this leads to a transition period of new experiences and learning, which
most probably leads to a revision of the funding system based on the experiences made
in the first period. This is followed by the second implementation of the new funding
system etc. All together it will easily take two to three funding periods before the new
system can operate smoothly, meaning that patience and persistence are key
prerequisites of a successful public governance of institutional funding for RO.
3.1.2 The specific starting point in the Czech Republic
Beyond the principles explained above, we have taken into account some key issues
that emerged from an earlier study22 in the design of the funding principles:
• Lack of good governance in the public sector, on the principal’s side
• Lack of ‘ownership’ on both sides but especially the principal’s, but also at many
agents (visible in weak management)
• Lack of trust of citizens in public institutions
The role of institutional funding for RO for the different (types of) research
organisations differs for several reasons: (i) due to their different missions research
plays different roles within their portfolio of activities, e.g. in an Academy institute it is
the primary activity whereas at a museum it is just one of many activities, and
certainly not the most important, (ii) not all (types of) research organisations have
access to the same funding streams, e.g. universities also have teaching funds and
special university funds for PhD training, which the Academy does not have as it does
not award PhDs. Furthermore, the Academy of Sciences has a separate funding stream
for equipment (coming from the “Expenditure related to the activities of the RDI
22 See Erik Arnold, Barbara Good, Fritz Ohler, Brigitte Tiefenthaler, Niki Vermeulen, The Quality of
Research, Institutional Funding and Research Evaluation in the Czech Republic and abroad. Thematic
Report No. 3, International Audit of Research, Development & Innovation in the Czech Republic, 2011.

Council, the GACR, the TACR and the Academy of Sciences of the Czech Republic”). In
contrast, RTOs, public service research organisations and national
resources/infrastructure research organisations have to use institutional funding for
RO for buying research equipment (see section 3.1.3) unless they use other funding
streams for the procurement of equipment which is needed for their main activities.
However, there have already been some positive experiences with negotiating
performance agreements, in the context of the 'Regional Research Centres' and
'Centres of Excellence', funded from the Structural Funds programme "R&D for
Innovation". These are, of course, different from the institutional funding for RO we
are dealing with in this report, but we want to point to the fact that the process of
negotiating, concluding and implementing performance agreements is feasible also in
the Czech environment – which some players have questioned.
3.1.3 Cross-subsidies between different sources of institutional funding
The first reaction of the external observer is one of wonder. Why is this question being
asked? Why is this a problem in the Czech Republic? Are there any legal constraints or
requirements? Is it just a formal problem or a real problem, and if it is a real problem,
what exactly is the problem? Or is it a perceived problem which is based on a variety of
other problems?
Based on the interviews we conducted, there appears to be a link between the issue of
cross-subsidies and the fact that different sources of funding are available for different
types of research organisations or even within types of research organisations (e.g.
universities and the Academy of Sciences). The main problem seems to be the
(alleged) cross-subsidy of teaching through research (and vice versa).
According to interviewees, non-research units tweak research outputs in order to get
RIV points (and ultimately institutional funding for RO), which they then use for
teaching; for example they are said to register textbooks as scientific output. The new
Evaluation Methodology will contribute to solving the problem as such RU will be
evaluated through peers. If they do not conduct any research and use institutional
funding for RO for teaching, then this will come out in the peer review and the RU will
get low scores.23 The low scores will translate into low funding and warrant special
attention by the university (see below section 3.4.3).
Another example where ‘cross-subsidy’ may possibly occur is that universities are said
to use teaching funds for paying for research equipment while the Academy of
Sciences has its own budget for research equipment. The rest of the research
organisations have to use institutional funding for RO or (if available) income from
other sources to pay for research equipment24.
Another example is that universities receive special university funding for PhD
students. PhD students write papers which are put into the RIV system and give the
university additional institutional funding for RO through the Evaluation
Methodology.25 The Academy does not receive funds for PhD students as it does not
award PhDs; hence, it does not normally receive points based on publications by PhD
23 In the case of RU that have too few publications to participate in the evaluation, this fact alone may be an
indication that „something could be wrong“, especially if the unit is a large one. In that case the unit
warrants special attention by the university management.
24 Background report to the 2nd Interim Report: The institutional funding system in the Czech Republic.
25 The same logic applies to competitive funding. Research organisations can win competitive funding based
on which they can write papers or produce other outputs. In the current system, this will give them more
RIV points and therefore more institutional funding for RO.

students26. With the peer review-based evaluation system we are suggesting this issue
will disappear, in case it is an issue at all.
Another angle to look at the issue is to identify how it is solved in other countries and
whether it is considered to be a problem at all. For example, in Austria and the
Netherlands, institutional funding for universities is typically in the form of a block
grant both for teaching and research, so cross-subsidies with respect to institutional
funding are no issue. In the Netherlands, each research university receives a formula-
based lump sum (block grant) for teaching and research; this allocation consists of a
teaching component and a research component but this distinction is for calculation
purposes only. The Executive Boards of Dutch universities and research institutions
have a large degree of autonomy for research and are also free to allocate the funding
as they wish (even across teaching and research activities). In Sweden, cross-subsidy
between research and teaching may occur in HEIs but it is not considered to be a
problem that would require action. In Norway, within limits, universities can use
teaching funds for research, and vice versa. However, this is not seen as a problem.
The universities are encouraged to make these priorities themselves, as they do not
have specific budgets for teaching and research. Some institutions get more research
funds than other, but the balance internally between teaching and research is up to
each institution to decide for itself27.
In fact, in the light of international practice, we think ‘cross-subsidy’ is not a real
problem as long as institutions use the money in line with their missions and in
compliance with the law. After all, the degree of research organisation’s autonomy
with respect to the internal allocation of funding is very high and is not significantly
limited by Acts or providers. We have shown that some of the issues that appear to be
considered a problem will be mitigated through the use of the new peer-based
Evaluation Methodology. Other issues such as the apparent insufficiency of teaching
funds cannot be solved through a new funding system for research as this is outside
the research funding system’s scope.
3.1.4 Ensuring adequate field coverage
In the countries we looked into, the concept of adequate field coverage does not exist.
The typical pattern is that the universities and other research organisations
themselves decide on the thematic foci of their research as long as it is within their
overall mission and specialisation28. So, in line with international practice, we would
recommend abandoning the idea of adequate field coverage at the overall level in
relation to the PRFS. The normal practice is to address policy-driven needs to change
field or problem coverage through competitive project or programme funding.
However, there is an issue of coverage related not to fields or scientific disciplines but
to “tasks that need to be done” in the R&D system, such as the training of young
researchers. One way to address such issues are the performance agreements (see
chapter 3.3), especially with scientific research organisation.
Moreover, some institutions, especially public service research organisations and
national resources/infrastructures often and typically fulfil tasks “that need to be
done” in a state, such as investigating and monitoring invasive species and mitigating
26 If PhD students add their institute’s affiliation to the publication, the Academy (or other research
organisations where PhD students are employed) do receive points for publications written by PhD
students. However, it appears that this does not happen very often.
27 Background report to the 2nd Interim Report: Country analyses Austria, Netherlands, Norway, Sweden.
28 see Background report to the 2nd Interim Report: Country analyses Austria, Netherlands, Norway,
Sweden.

their adverse impact on the native flora and fauna. In many such organisations,
research is the primary task and serves the public sector, typically ministries, and/or
user communities that do not commission research themselves (e.g. apple growers,
pedestrians, drivers etc.), providing them with applied research results to improve the
public sector’s and user communities’ way of doing things. Other public service
research organisation’s primary task is the provision of services (e.g. forensic
investigations, measuring of radioactivity, water management etc.) but research is
central to updating their methods, keeping abreast of current developments and
developing their capabilities. It is the responsibility of the ministry in charge of the
subject, typically the founding ministry, to provide the basic conditions for the tasks to
be fulfilled in a satisfactory manner.
If ministries would like specific fields to be covered more (e.g. research addressing
societal challenges or promising technologies), there is always the possibility of setting
up a thematic programme using targeted funding, similar to the Alfa or the Epsilon
programme implemented by the Czech Technology Agency. This is the way it is done
in other countries as well, e.g. in Norway or Austria29.
3.1.5 Efficient sharing and use of national and international research infrastructure
When we speak of research infrastructure here, we mean research equipment
(instruments, databases etc.) and not national resources/infrastructures such as
museums, archives etc, which are research organisations in our context and are
treated as a separate type of research organisation.
Research infrastructures can in general be categorised by size in terms of costs and
scope
• There is small equipment, like microscopes or data sets that need to be bought. In
most countries, the costs for such equipment are covered by an institute’s or
research group’s ‘petty cash’.
• Then there are the large items of research equipment, e.g. machinery that may
cost several hundred thousand Euros (these are termed “infrastructure” in the
Czech legislation30. Here the funding is more difficult. A growing number of
countries have ut special fudning programme in lace for this, recognising that it
has been a problem that has for too long been igored. They may be able to get
some funding from the research council as part of project funding or within
specific programmes for the funding of research infrastructure. Or they have to
negotiate within their research organisations to get funding from the institutional
funding budget. Normally, in most countries, medium sized pieces of equipment
are funded by institutional funding or competitive funding or a mixture of both.
• Then there are the really large research infrastructures, e.g. particle accelerators
(termed “major infrastructure” in the Czech legislation)31. In the Czech Republic
(like in most other European countries), there is a national road map for such
large-scale research infrastructures, and i there is a separate fund for funding
them, giving out long-term competitive funding.
29 see Background report to the 2nd Interim Report: Country analyses Austria and Norway.
30 See Background report to the 2nd Interim Report: The institutional funding system in the Czech
Republic.
31 see Background report to the 2nd Interim Report: The institutional funding system in the Czech Republic.

In most countries the funding of research infrastructure tends to be discontinuous and
inconsistent. There are periods with large investment followed by periods with low
investment.
In the Czech Republic, the funding of research infrastructure is also patchy.
Universities normally use teaching funds for paying for research equipment while the
Academy of Sciences has its own budget for research equipment.32 Many research
organisations located outside Prague could apply for funding new infrastructures
through the Structural Funds (Czech Operational Programme for RD&I). The rest of
the research organisations have to use institutional R&D funding or (if available)
income from other sources. Hence, there is a situation of inequality. It is our
understanding that the ministries in charge of these research organisations have to
deal with the issue of research equipment in their research organisations.
Efficient use of research infrastructure can be more importantthan sharing. This
applies typically to research infrastructures that are mainly used in-house, especially
the first two categories listed above. There is evidence that the most efficient use of
research infrastructure is characterised by a large share of internal use. On the other
hand, efficient use is characterised by the existence of (i) attractive research
programmes in the organisations where the research infrastructures are located and
(ii) qualified and active key staff working at and with the research infrastructure, and
(iii), if there is capacity for sharing, a clear understanding of how and under which
conditions others should have access to a research infrastructure. Thus, research
infrastructure has to be discussed in the context of attractive research programmes,
credible staff and sound management (see for instance the evaluation criteria applied
in the Czech Operational Programme for RD&I).
The availability and the use of research infrastructure is addressed in the new
Evaluation Methodology, where it is an indicator under the assessment criterion
“Institutional management and development potential”.
3.1.6 Changing national priorities and the need for stability of institutional R&D
funding
In a first step, we need to distinguish different types of priorities, especially making a
distinction between thematic (e.g. nano, energy, life sciences…) and systemic (e.g.
science-industry links, HR development) priorities.
In the countries we investigated institutional funding is not linked to national
priorities or strategies and research organisations can use it freely, within their
mission of course, and set their own priorities. National thematic priorities are (and
should) be mainly funded via competitive funding (programmes).
Systemic priorities are very much linked to institutional development, and the Czech
audit clearly shows that institutional development should a priority in the Czech
Republic33. It is one of the Czech policy goals to create stable conditions for conceptual
development of research organisations34. This means that (some) stability of
institutional R&D funding in this context is in no contradiction to national priorities,
32 see Background report to the 2nd Interim Report: The institutional funding system in the Czech
Republic.
Methodology EM2015, Appendix A-1, pp. 3-7

as it serves a systemic priority, i.e. the institutional development of Czech research
organisations. Other research policy goals are immediately linked to this issue,
especially improving the quality of R&D governance at the national level and
strengthening of research excellence
National systemic priorities can also be funded via competitive funding (e.g. Research
Centres funded from Structural Funds, Centre-of-Competence-programmes,
cooperation programmes, doctoral schools). Systemic priorities with strong
institutional implications should be subject to institutional governance and will
typically be addressed in performance agreements (e.g. HR development) (see section
3.3).
In the long run, we suggest strengthening the link between competitive R&D (project)
funding and institutional development. This means that in the applications of projects
for funding, applicants need to provide strong answers to the question “How will this
project help your organisation achieve its overall goals?”.
3.2 The core funding principles
In this section we present the core funding principles underpinning the new funding
system.
3.2.1 The distribution of institutional funding for RO among research funders
A pre-requisite for the proposed funding model is that are separate ‘pots’ (budgets or
budget lines) for different types of research organisations: scientific research
organisations, RTOs, public service research organisations, national
resources/infrastructures35. The thinking behind this pre-requisite is that different
types of research organisations fulfil different missions and functions in society.
Hence, they should not be made to compete for the same budget pot as this may lead
to some research organisations not being able to fulfil their roles any more. This is a
break with the current funding system where all research organisations compete for
institutional funding for RO irrespective of their type, a practice criticised in the Czech
Audit.36
What needs to be understood is that that typology is based on the function of
organisations, not on their legal form. This ensures that only organisations with the
same function are put in competition with each other. If we use a typology based on
legal form, we start mixing functions in one typology and make research organisations
with one function compete against research organisations with another function.
This pre-requisite is in line with international practice where different budget pots for
different types of research organisations is the common practice. The distribution is
generally contingent on: laws (responsibilities), history (e.g. size of budgets), politics,
policies, (entrepreneurial) individuals (or the opposite). For example, in Austria the
public universities and the non-university research organisations are not funded from
the same budget:. research organisations such as the Austrian Institute of Science and
Technology and the research institutes of the Austrian Academy of Science (individual
35 For the typology of research organisations see R&D Evaluation Methodology and Funding Principles,
First Interim Report: The Evaluation Methodology EM2015, p. 29

pots), for applied research organisations such as the Austrian Institute of Technology,
Joanneum Research (individual pots)37 etc. This is similar in other countries.
In the end, how much money to allocate to the different pots is a policy decision to be
made by policy-makers38. We suggest the following procedure for the transition from
one single budget for institutional funding for RO of all kinds to different budgets for
different types of research organisations: We suggest taking the current expenditure
on institutional funding per research organisation as a starting point (mean values or
weighted mean values over several years). This would be a starting point which
provides continuity and stability although a starting point that may be considered
unfair given the re-allocations of institutional R&D funding entailed by the application
of the coffee mill. Nonetheless, the starting point suggested is transparent and
predictable, and this is important given that the definition of pots for different types of
research organisations ultimately involves a policy decision.
As outlined in the previous section 3.1.1, ideally the responsible ministries should base
their funding decisions on a thorough understanding of the research organisations’
‘funding profiles’. On this basis it will be possible to determine better how large the pot
for ‘their’ type of research organisations should ideally be.
The responsible ministries are the Ministry of Education, the Ministry of Industry, the
Ministry, of Interior, the Ministry of Health, the Ministry of Agriculture, the Ministry
of Culture, the Ministry of Defence and the Academy of Sciences which has the status
of a ministry. Those ministries that lost their R&D budget for their public service
research organisations (e.g. the Ministries of Environment and Transport) and whose
R&D funding went to the Ministry of Education should also be consulted in the
decision making on the size of the ‘pots’ to make sure that the interests of ‘their’
research organisations are adequately represented.
3.2.2 Components of the funding system
Based on our mandate, we have designed a funding system that encompasses three
components: a block grant, a performance agreement and a PRFS (performance-based
research funding system).
In order to ensure trust and stability, part of the institutional funding for RO will be
allocated in the form of a block grant, like in the current funding system based on the
evaluation methodology valid for years 2013-2015 (see 2.2.2).
While the block-grant ensures trust and continuity, the performance-based funding
component reflects both (i) past performance as well as (ii) future developments and
plans. Therefore, we propose two elements for the performance-based part of
institutional funding for RO:
• A performance-based research funding system (PRFS) which allocates funding
on the basis of the new Evaluation Methodology39, and is mainly based on the
judgement of external peers in their respective fields of expertise and types of
research organisations assessing mostly past performance but also future
strategies and
37 see Background report to the 2nd Interim Report: country analysis Austria.
38 Policy decision does not only mean decisions by politicians, it also includes decision by the civil
service/ministries.
Methodology EM2015

• A (negotiated) performance agreement, mainly based on the negotiation between
the government (a ministry or a specialised agency) and the respective research
organisation.
Figure 13 shows on what basis institutional R&D funding is determined and to whom
the funding is allocated. Block grants and performance agreements are allocated to
research organisations as legal entities and are determined on the basis of the budget
allocated to research organisations in the preceding funding period. The thinking for
PRFS money is different. The money is determined on the basis of scores achieved by
research units (RUs) and then aggregated to the level of the research organisation as
legal entity which receives the PRFS money.
Figure 13 Allocation of funding by type of instrument
3.2.3 Shares of the components of the funding
3.2.3.1 Block grant
We suggest that, similar to the current system, in the first funding period after the first
evaluation based on the new methodology, the block grant should make up 80% of the
pot (or pots for the different research organisations) and 80% of what each research
organisation received in the previous funding period.40 The previous funding period
should refer to an average of what research organisations received over a period of 3-5
years (a ‘reference funding period’), in order to take into account trends. Moreover,
these 3-5 years should cover the years when institutional R&D funding was based on
the evaluation methodology valid for the years 2013-2015 (or later), encompassing an
80% block grant and an 20% performance based part. Using the years before the
Evaluation Methodology 2013-2015, when a higher share of institutional R&D funding
40 There is a precedent for this in the UK where in the beginning of the RAE only a small share of
institutional funding was allocated based on the RAE, comment by advisory board, 10 November 2014.
Block grant Performance-based components
Performance
agreement
Performance-based
research funding
system (PRFS)
entity
entity
the level of individual
RUs
legal entity
legal entity
legal entity

was allocated by the ‘coffee mill’ would not make sense because fluctuations were too
large.
The starting point we are suggesting safeguards continuity with the present funding
situation for each research organisation involved. We have opted for it for the sake of
transparency and stability while being aware that the Metodika, through its own
specific problems, has led to an allocation of funding among research providers and
research organisations that some may consider deformed or unfair41. Later on in this
chapter (section 3.4.3), we are suggesting a treatment for RUs that achieve low scores
in the new evaluation, which should mitigate any problems associated with the
starting point.
3.2.3.2 Performance-based research funding system (PRFS)
We suggest basing 15% of total institutional funding for RO on the PRFS. Moving 15%
of institutional funding for RO through a PRFS is in line with some international
practice, e.g. in Norway42 or Denmark. The PRFS, as it is proposed at the moment,
would run every five or six years. Experience with performance based research funding
systems in other countries shows that even smaller shares of budgets allocated on the
basis of performance indicators can have dramatic effects on the research system43.
Experience from other countries shows that the PRFS steers behaviour through two
channels: funding and esteem. Contrary to what might conventionally be expected,
researchers’ behaviour tends to be more strongly impacted by the measure of esteem.
Getting a bad score in the PRFS generally is a enormous incentive to get better. Hence,
a 15% PRFS component is not only in line with practices from other countries,
experience shows that it will have an impact on behaviour.
3.2.3.3 Performance agreement
We suggest linking 5% of funding to a ‘light touch’ performance agreement to promote
institutional development and capacity building. The performance agreements can be
negotiated between a the funding provider (ministry) and research organisation for a
period of two and half to three years, to be compatible with the periodicity of the
PRFS. Those ministries that lost their R&D budget to the Ministry of Education need
also be involved in the negotiation of the performance agreement to ensure its
relevance. In the long term, the possibility to transfer R&D funding responsibility back
to the founding ministries should be examined.
‘Light touch’ in this context means that the performance agreement should not cover
the entire research organisation with all its activities but focus on one or several
strategic projects focusing on well defined issues of institutional development.
The rationale for this ‘light touch’ performance agreement in the funding system is
that principals (providers/ministries) and agents (research organisations) are new to
this instrument. They will need to build up capacity to negotiate, implement, report on
and supervise the performance agreements. How this can be done in practical terms,
we show in more detail below (section 3.3).
42 Background report to the 2nd Interim Report: Country analysis Norway.

We have suggested a 5% share of this funding component because performance
agreements are an instrument that is new to the Czech Republic; this low share will
give research organisations opportunity to learn and introduce a cultural change,
while not risking too much. We have proposed the same share of 5% for all types of
research organisations because all are in need of institutional development.
Together, the performance-based components will make up 20% of total institutional
R&D funding. However, given the high percentage of competitive funding
(approximately 50%) in the Czech R&D system44, we think that the competitive
element in the institutional R&D funding system should not be too pronounced, in
order to guarantee stability and allow capacity building.
In the present system, a formula-based evaluation takes place every year, based on
achievements in the prior period, i.e. the preceding five years. In the future, the new
peer-based evaluation is scheduled to take place every five or six years, which implies
that the overall amount of funds allocated through the corresponding mechanism for
institutional funding for RO is also set for the same period of five or six years45.
Increases in salaries and inflation have to be taken into account, e.g. by taking into
account inflation and increases of wages in the annual instalments.46
3.2.4 Shares of components in future funding periods
We suggest an 80+15+5 allocation of institutional R&D funding to all types of research
organisations for the first funding period because they all are in need of stability.
However, it goes without saying that the shares of the block grant component and the
two elements of the performance based component can be changed in future funding
periods once the PRFS and the performance agreement system are well established.
Moreover, because there are different budget pots for different types of research
organisation, the shares can be varied for different types of research organisations
according to their needs and depending on policy requirements.
However, this requires a policy decision that cannot be delegated to a PRFS or some
other ‘mechanical’ system (or foreign consultants for that matter). The policy decision
has to be taken by the responsible ministries. We suggest the following principles are
considered when thinking about increasing or decreasing the shares of the funding
components.
• One prerequisite is that the PRFS needs to work well47 before its share can be
increased. The same is true for the performance agreements.
• Experience in other countries shows that in countries where there is an increase in
institutional funding, this increase tends to be allocated through the performance-
44 see Background report to the 2nd Interim Report: The institutional funding system in the Czech
Republic.
45 But careful: the UK experience shows that one needs to think about the new round early!
46 This means that by the end of the funding period the total amount of the block grant spent would reach a
total of the original 80% increased by inflation adjustments and wage increases - this needs to be taken
into account in the state budget!
47 Introducing a peer review based evaluation system will be new to some research organisations. For them,
it will primarily mean that they will need to learn how to write a self-assessment report. At the level of the
people administrating the new Evaluation Methodology it will mean that they will have to learn how to put
together panels etc. For the Methodology itself, it means it needs to be tested, for example to make sure
that the indicators are suitable for all types of research organisations, which is being done now and in the
next year. Therefore, enough learning has to occur in the system before the PRFS share of the funding is
increased. This means at least one round of the new evaluation methodology has to be done before
increasing the PRFS share is considered.

based part, which implies that the share of the block grant decreases – while not
decreasing in absolute terms, thus ensuring stability. In countries where there is
little increase in institutional funding, the shares of the block grant and the
performance-based part typically remain fairly stable.
• Because there are different budget pots for different types of research
organisations, the shares of the funding components can be changed separately for
each type of research organisation, depending on their needs. For example, if a
certain type of research organisation displays a particular need for institutional
development, the share of the performance agreement could be increased in order
to set an incentive for such research organisations to focus on institutional
development. Similarly, if a certain group of research organisations displays a
need for more quality and is homogeneous enough to benefit from more direct
competition, the PRFS part of the funding could be increased t. The general
guiding principle would be that changes in funding component shares need to be
in line with the needs and requirements of particular types of research
organisations, as articulated both by the research organisations themselves and
their owners.
• This is linked to the fact that the size of the shares should be in line with
ministries’ strategies. If ministries’ strategy is to increase competition in a group
of research organisations, then the PRFS share should be increased. If ministries’
strategies is to maintain stability, then the block grant should be left at 80%. If
ministries’ strategies is to promote institutional development in a group of
research organisations, then the performance agreement component should be
increased. If several ministries are responsible for a group of research
organisations, their strategies should be aligned in a process of negotiation.
• Another important guiding principle is that the funding mix
(institutional/competitive/contract funding) ought to be part of the decision
making on the shares of the different funding components for different types of
research organisations. If in a group of research organisations there already is a
high share of competitive funding, then the performance-based components, in
particular the PRFS, should not be over-emphasised in order to ensure stability
and opportunities for capacity building. Similarly, if the share of competitive
funding is fairly low, then the performance-based components can be increased
more. The share of competitive funding is typically higher in more applied
research organisations compared to basic research organisations.
Box 1 - Some words about percentages
In this chapter we often talk about different percentages. This might be confusing
because in absolute numbers, some percentages are the same but they refer to
different things, namely to one of the following:
• At a national level, on the expenditure side, we talk about total funding pots
available for each type of research organisation. This is what the 80+15+5 formula
refers to. 80% of the funding available will be allocated through block grants, 5%
on the basis of performance agreements and 15% through the PRFS.
• At an institutional level, on the income side, we compare the sizes of institutional
R&D funding and its three components to the amount of institutional R&D
funding received in the preceding funding period. For the sake of stability we
suggest that each research organisation will receive 80% of the amount it received
in the preceding funding period as a block grant plus 5% if it enters into a
performance agreement with the responsible ministry/funding provider.
Moreover, each research organisation can receive money through the PRFS,
depending on the results of the evaluation according to the new Evaluation
Methodology. This means that 15% of the pots available per type of research
organisation will be redistributed among those research organizations.

As a result, at the level of research organisation, the share of the PRFS money will vary
among research organisations: in the extreme case of an research organisation that
scores “0” throughout its evaluation units, the PRFS component of funding will be
zero, i.e. in total such an research organisation would receive 85% of the institutional
R&D funding it received in the preceding funding period. On the other side, a research
organisation that performs outstandingly well will receive a relatively large amount
through the PRFS, i.e. in total, their institutional R&D funding will grow to more than
100% compared to the preceding funding period.
What has just been said is based on the assumption that the pots of budgets per type of
research organisation are the same size as in the preceding funding period. If these
pots were larger (e.g. due to index alignments), the block grant and performance
agreement component allocated to each RO would be adjusted accordingly. The PRFS
pot as a whole would be larger but allocation of funding would be based on the scores
achieved.
3.3 Performance agreements in more detail
The performance agreement component resembles an existing funding stream among
HEIs called Rozvojové programy (Development Programme) administered by the
Ministry of Education, Youth and Sports (MEYS), which also promotes institutional
development projects. However, while the current programme for HEIs uses a funding
formula based on quality indicators, the performance agreements we are proposing
here will not because in the funding system the quality aspect is covered by the PRFS.
The performance agreement component, however, foresees sanctions in case the
research organisation does not fulfil the agreement (see below).
As mentioned above (see section 3.2.4), if the research organisation enters into a
performance agreement with the relevant ministry, 5% get added to the block grant.
The performance agreements are set up individually, taking into account the specifics
of each research organisation (e.g. role and mission, research profiles, institutional
setup). They are light touch and promote institutional development and capacity
building. Light touch means that the performance agreement does not aim at grasping
and comprehending the research organisation as a whole but that it focuses on one or
several well defined strategic projects in order to support institutional development48.
The main idea behind this approach is that a leverage effect is expected: if the strategic
projecs succeed, they will have a positive influence on the whole or at least essential
parts of the research organisation.
The performance agreement is an agreement between the ministry and the research
organisation on a small number of strategically relevant undertakings that the
research organisation agrees to implement in the performance agreement period in
order to increase R&D capacity and to improve working conditions for research as well
as for support staff. Some research organisations were founded by one ministry and
are now being funded by another (the MEYS). As a rule, the agreement is between the
funder and the research organisation. If funder ≠ founder, then both ministries
represent the principal.
For example, performance agreements can stipulate that
48 The opposite would be a comprehensive agreement, trying to cover “everything” the research
organisation should do and achieve during the funding period. Comprehensive agreements (e.g. in
Austria) tend to weaken the agreement (and the idea behind) as neither the research organisation nor the
ministry are able to oversee the broad range of issues in depth.

• a university improves its doctoral training by linking it better with the research
priorities of the university or the procedures of recruitment, supervision and
career development
• a university should enter a new field or tackle a new set of problems in a way likely
to result in the creation of a new RU over time
• a research organisation needs to set up an internal quality assurance system, or in
case a research organisation already has an internal quality assurance system, has
it audited by an external evaluation panel, e.g. a foreign quality assurance agency
• a research organisation designs and implements an attractive, internationally
compatible career path for research staff, modernising its career management
structure (e.g. through introduction of tenure track positions)
• a research organisation designs and implements a programme to reconcile work
and family life for men and women
• a research organisation implements and funds a continuing education scheme for
their administrative staff or for external user groups.
• etc.
If research organisations do not want to negotiate a performance agreement with the
responsible ministry, or if the two involved parties do not manage to negotiate an
agreement, the research organisation will not receive the 5% top-up to the block
grant49.
The performance agreements are negotiated between a research organisation and the
responsible ministry for a period of three years. The research organisation reports on
progress made in the strategic undertakings agreed on in the performance agreement
on a yearly basis. After three years, the implementation of the performance agreement
is evaluated, and based on the results of the evaluation, a new performance agreement
is negotiated.
In the following two chapters we present a more detailed description of the
performance agreements and the processes involved in setting them up and
implementing them. These chapters are based on our experience with and knowledge
of performance agreements (PA) in several countries, esp. Luxemburg, Austria, the
Czech Republic and Switzerland. As outlined above, each performance agreement as
such takes into account the specific situation of the research organisation concluding
the agreement with its ministry (or ministries50). Therefore, there we do not have to
distinguish between different types of research organisations in the following
explanations.
3.3.1 Scope and content of a light touch performance agreement
In this chapter we provide guidelines for the process of preparing, negotiating and
concluding a performance agreement between a ministry and a research organisation.
Moreover, we provide guidelines and the basic template for the issues to be covered in
the light touch performance agreements we are suggesting.
49 This would raise a question: What happens to the money that does not get distributed because some
research organisations do not want to or manage to have a performance agreement? We recommend
moving the money to the pot of the PRFS.
50 Some research organisations were founded by a specific ministry and are now being funded by another
(mostly the MEYS). As a rule, the agreement is between the founder and the research organisation. If
founder ≠ funder, then both ministries represent the principal.

Basics
• Parties concluding the agreement are the ministry in charge of the research
organisation and the research organisation. There are cases where the founding
ministry is not the funding ministry, in particular for a number of public service
labs (e.g. Water Research Institute, Transport Research Centre). For such cases we
suggest that the founding ministry, i.e. the ministry with an actual working
relationship to the research organisation is involved in the process as well.
• The performance agreement is light touch, i.e. it focuses on a small number of well
defined strategic projects in support of institutional development51.
• The performance agreement makes promises for future performance. They have to
be based on sound analysis and an understanding of the starting point (including
the awareness that this may be missing), on credible past achievements of key
staff, and on suitable governance and management systems. The report by the
peers can serve as an indication where the starting point is and what issues might
need attention in the performance agreement.
• Objectives agreed in the performance agreements need to be manageable and
realistic yet ambitious, and whether this is the case will be assessed on the basis of
past performance. Manageable means that objectives must be attainable with the
resources (money, hierarchy, trust, information, leadership/orientation) available
in-house.
• The performance agreement is binding for both parties, although, strictly speaking
it is not a contract in the legal sense of the word.
• Through the performance agreement a research organisation will receive a 5% top
up to the block grant of institutional funding for RO (see section 3.2.3.3). Unlike
grant-funded projects, the strategic projects do not necessarily have to account for
exactly this amount of funding.
• The first funding period in particular will be a period for learning for both parties.
Although there is some experience in the Czech research system with similar
agreements, not all ministries and not all research organisations can draw on such
experience. Moreover, it will be necessary to improve mutual understanding.
• Most likely some projects agreed upon in the first period will turn out to be either
too ambitions or too simple, i.e. both parties will get a better understanding of
what can and should be included in the following performance agreement.
• All parties involved have to be aware that it will most likely take more than one
funding periods before the new tool is made best use of.
The process
• The ministry in charge invites the research organisations to present a proposal for
the strategic project(s) they want to include in the performance agreement. In this
invitation, the required content (and format) for these proposals is specified (see
below).
• Each research organisation submits its proposal for the strategic project(s) to the
ministry.
• Both parties meet in order to discuss the proposal.
51 The opposite would be a comprehensive agreement, trying to cover “everything” the research organisation
should do and achieve during the funding period.

• In the negotiations, both parties need to be represented by persons authorised to
conclude a performance agreement and, on the ministry’s side, who are familiar
with the research organisation.
• If a major revision of the proposal is needed, one or several follow-up meetings
may be necessary. Minor revisions will typically be feasible in a written procedure.
• Optionally, an external facilitator can support the negotiations. This should be a
person familiar with research and research management at the respective type of
research organisation. If needed, the facilitator mediates between the two parties
as the “advocate” of an agreement that is both ambitious and feasible. Involving a
facilitator may be especially helpful for those ministries that have little experience
with negotiations or feel they are unfamiliar with performance agreements.
• Finally, the performance agreement is signed.
Content of the performance agreement (basic template)
The performance agreement will be set up on the basis of a template issued by the
ministry in charge. The template has to include the following content52.
• Parties concluding the performance agreement (names and contact details of the
institutions and the persons in charge)
• Description of the strategic project(s)
− Starting point: What is the problem to be solved, or the issue to be addressed,
or the opportunity to be grasped? Provide evidence to support this point!
− Overall objective: What would be different at the research organisation if the
problem were solved, or the issue addressed successfully, or the opportunity
grasped? How does this relate to the strategy of the research organisation?
− What does the research organisation propose to do in order to reach the
overall objective? This is the strategic project/these are the strategic projects.
A strategic project may take more than the three years of one performance
agreement period. In this case distinguish between the overall strategic project
and the strategic subproject(s) for the three years of the performance
agreement period.
− How will the strategic project be implemented? Give a work plan including
milestones and estimated timetable.
− Who will be involved? Key staff involved including their relevant
qualifications, roles and tasks in the project.
− Which resources will the strategic project need? Cost estimate (personnel cost,
other costs).
− How will these resources be financed?
− What are the indicators used for monitoring of the strategic projects(s) and
what are the deliverables which should be reached after three years?
− What happens after the strategic project(s) in order to safeguard its
sustainability?
52 This basic template focuses on the content to be covered in each performance agreement. Ultimately, the
actual templates will have to be designed and formatted according to the regulations to be applied by each
ministry.

− Which are the most relevant risks of failure? What would the impacts be? How
are risks mitigated?
• Rules for reporting: frequency and modes of reporting (see guidelines in the
following chapter)
• Rules how extraordinary situations will be handled (e.g. radical change of the
institution’s situation, significant non-achievement of agreed interim and/or final
milestones, loss of key staff, … , embezzlement of funds)
• Amount of funding (block grant and top-up), funding period, and regulation of
instalments
• Signatures
Criteria for the ex-ante assessment of strategic projects
Both, the research organisation (in preparing) and the ministry in charge (in
assessing) shall be guided by the following criteria:
• Does the research organisation have a convincing overall set of goals and related
strategies? Are these goals / strategies based on sound analysis of the current
situation?
• Are the proposed strategic projects most relevant to support the stated goals /
strategies (fit-for-purpose)? Are they feasible in terms of resources and timing?
• Is the key staff credible to successfully implement the proposed projects?
3.3.2 Reporting and implementation
In this chapter we describe how the reporting and monitoring of the performance
agreement will work. We also explain how to deal with deviations from the
performance agreement, including sanctions.
• Research organisations will present a written annual progress report to the
ministry in charge. The progress reports contain information about the work done
and the results achieved to date, and a cost statement as well as a comparison of
the progress made against the intended objectives of the strategic project(s)
including information on and an analysis of deviations from plan.
• The progress report will be discussed during an annual progress meeting between
the head of the research organisation and the ministry (or ministries in case the
founding ministry is not the funding ministry) in charge. In the case of unexpected
events that affect the progress of the strategic project severely, an ad hoc meeting
has to be held.
• Deviations from the agreement (e.g. delays beyond the agreement period) need to
be discussed in progress meetings, with a possibility to agree on a changed agenda
for the project at stake (if there are good reasons!).
• The final progress meeting of a performance period will be used to assess the
results achieved with the strategic projects as well as the experience that have been
made during their implementation. An external assessment may be necessary,
depending on the content of the performance agreement. This could be the case if
the representatives of the ministry feel they are not capable of assessing whether
or not an agreed project has actually reached its objectives. In this case the project
and its results could be assessed by an external expert supporting the ministry.
However, generally the ministry and the research organisation should be capable
of assessing the success of the strategic projects.
• The first round of performance agreement is for learning and capacity building on
both sides, therefore no negative consequences will follow if goals are not

achieved53. The ministry has to stick to the agreement! This is a strong incentive
for the research organisations because they cannot blame the ministry if they fail
to fulfil their part of the agreement.
• From the second round on sanctions can be applied by the ministry if the research
organisation infringes the PA and if the ministry itself has stuck to the agreement.
• After the first period, sanctions need to be defined in the performance agreement,
the most obvious being a decrease of the 5% top up, e.g. down to 2.5% for the
following period.
3.4 Performance-based research funding system (PRFS) in more detail
The key results of any peer review-based funding system PRFS are (i) a set of scores
according to a given set of criteria, and (ii) a verbal description and justification54 of
the respective scores by the peers. This chapter describes how the scores are
transformed into funding for the different types of research organisations.
Changes in the scores of the PRFS after a new evaluation round would be implemented
in the year after the new evaluation round rather than implemented gradually over the
funding period. Put differently, the new scores would be used to calculate the PRFS
part as soon as they are available. It does not make sense to use the new scores
gradually because
• there is already a great deal of stability built into the funding system, so this
‘abrupt’ change in the PRFS funding after the new evaluation round seems
acceptable
• it would complicate the system unnecessarily
3.4.1 Weighting the evaluation criteria for different types of research organisations
The new Evaluation Methodology provides us with scores from 1 to 5 against five
categories of indicators: research excellence, research performance, societal relevance,
institutional management and development potential, membership in the national and
global research community 55.
In order to translate the scores into funding, our proposal is to use the five categories
and define “sub-pots” for them. This implies a weighting of the categories in line
with the organisations’ mission and in line with the PRFS’s objectives. This
in turn implies that the weights have to be different for the different types of
organisations because they have different missions and roles in the Czech R&D
system.
The following table shows how the ‘PRFS pot’ for scientific research organisations gets
divided into ‘PRFS sub-pots’ by category. This gives us the sum of money to be
allocated by criteria (i.e. 5 sub-pots). We have used a fictional example to make the
table more intuitive.
53 With the exception of wilful underachievement.
54 To put it in a formula: While the scores are eminence-based (relying on the experience, knowledge and
reputation of peers), the description should be evidence-based (relying on facts and figures, and
arguments).
Methodology EM2015, p. 62-63.

Figure 14 Evaluation criteria and their weights for scientific research organisations
(default scenario)
Criteria and their weights CZK
Research excellence: 10% 10,000,000
Performance: 50% 50,000,000
Management: 20% 20,000,000
Societal relevance: 10% 10,000,000
Membership in the world research community: 10% 10,000,000
Total (‘PRFS pot’) 100,000,000
Each sub-pot gets allocated to RUs based on the way outlined below in section 3.4.2.
While we can provide guiding principles on which to base the decision on how to
weigh the different categories for the different type of organisations, the weighting
itself is essentially a policy decision that has to be made by policy-makers. Hence, the
decision-making authority will need to undertake a top-down weighting of the five
criteria according to policy priorities, missions of research organisations and, not
least, their history. The decision making authorities will be the responsible ministries.
We suggest the following guiding principles should be taken into account when
determining weights.
• The main guiding principle will be that the weights of the different evaluation
criteria need to be in line with research organisations’ missions.
• In line with their mission, research excellence is of highest relevance for scientific
research organisations. Hence, this category should have a higher weight for
scientific research organisations than for all the other types of research
organisations.
• Research performance is a multi-dimensional category and refers to productivity
and quality. Both are essential to the good functioning of research organisations,
be they scientific research organisations or more applied R&D actors that serve
user communities. As the Czech Audit has shown56, quality and productivity are
important issues in the Czech Republic (and anywhere else) and for all types of
research organisations, that is why they should have a high weight. Therefore, we
would give research performance a fairly high weight.
• Societal relevance is different and in the context of evaluation more important for
research organisations that conduct applied R&D and serve user communities
such as industry sectors (RTOs), the public sector (public service research
organisations) or other researchers (national resources/infrastructures such as
archives and libraries) than for scientific research organisations. For scientific
research organisations, societal relevance lies mostly in the teaching and training
of (PhD) students. Consequently, societal relevance should have more weight for
RTOs, public service research organisations and national
resources/infrastructures.
• Membership in the national and global research community is most important for
scientific research organisations because science is a globalised undertaking and

reputation is the ‘key currency’ in the science community. Hence, the category
should have a higher weight for scientific research organisations than for the other
types of research organisations which often work primarily with and for more local
communities of partners and users.
• Management is important as it is a typical bottleneck in the Czech Republic57 (and
in other countries) and is therefore assessed separately. Since management is
currently often not perceived to be important by many research organisations, it
should not get too much weight in order not to discourage and penalise research
organisations too much. However, it should still have enough weight so as not to
be treated as a ‘quantité négligeable’ by research organisations.
• Weights will also need to be in line with ministries’ strategies for ‘their’ research
organisations. While these strategies should be in line with research organisations’
missions, strategies will typically go beyond the mission, emphasising certain
policy objectives. For example, if it is a strategy to increase excellence in scientific
research organisations, then research excellence should be given a high weight. If
the strategy is to improve management, then management will need to
emphasised.
The weights shall be used solely for the purpose of funding. They will be decided
before the evaluation exercise but they will not be published until after it in order to
avoid gaming by research organisations. This is because if research organisations
know that a certain category has a particularly strong weight for them, they will do
(almost) anything to show themselves in the best light in that category. Evaluation
panels must not know the weights either so that their judgement will not be influenced
by this knowledge.
It is our understanding that the separate budget pots for different types of research
organisations and the different weights attached to the criteria will sufficiently reflect
and take into account the differences between the different types of research
organisations as long as the new Evaluation Methodology does do justice to the
different types of research organisations. What is more, as explained above, since
there are different budget pots for different types of research organisations, in the
future the shares of the components of the funding system (block grant, PRFS,
performance agreement) can be varied for different types of research organisations,
thus further reflecting differences between research organisations.58
Figure 15 underlines that the PRFS funding system we are proposing is two-
dimensional, with one dimension being the types of research organisations and the
other being the different evaluation criteria. The different evaluation criteria have
different weights for the different types of research organisations. This is symbolised
by the different sizes of the spheres in the cells.
57 See Erik Arnold et al., The Quality of Research, Institutional Funding and Research Evaluation in the
Czech Republic and abroad. Thematic Report No. 3, International Audit of Research, Development &
Innovation in the Czech Republic, 2011.
58 As we have explained above, we have not done so in our proposal because at the moment all types of
research organisations are in need of stability, therefore we have set the shares for the funding pot at 80%
(block grant), 15% (PRFS) and 5% (performance agreement).

Figure 15 The two dimensions of PRFS funding system (default scenario)
Scientific
research
organisations
RTOs Public service
research
organisations
National
resources/infrastructures
Research
excellence
¢ ¢ ¢ ¢
Research
performance ¢ ¢ ¢ ¢
Societal
relevance
¢ ¢ ¢ ¢
Membership ¢ ¢ ¢ ¢
Management ¢ ¢ ¢ ¢
Below we are giving examples of how the weights could be set for the different types of
research organisations in order to accommodate the different roles and missions of the
different types of research organisations. We will also show how the weights can be
used to implement different R&D policy priorities, presenting three scenarios below.
These are suggestions only; the actual weights have to be discussed and agreed on in a
political process. Of course, our suggestions may serve as a starting point for
discussion.
3.4.1.1 Default scenario – increasing overall quality in research organisations
Our default scenario aims to increase overall quality in research organisations,
irrespective of the type. That is why research performance has been given the highest
weight (50%), and the same weight across all types of research organisations.
Figure 16 Weights in the default scenario
Default scenario
ScRO RTO PubSL NatRes
Research Excellence 10% 5% 5% 5%
Research Performance 50% 50% 50% 50%
Societal Relevance 10% 20% 20% 20%
Membership in the World Research Community 10% 5% 5% 5%
Management 20% 20% 20% 20%
Total 100% 100% 100% 100%
ScRO=scientific research organisations; RTO=research and technology organisations; PubSL=public service
research organisations; NatRes=national resources/infrastructures

These weights translate into the following shares of total institutional funding for RO.
In other words, Figure 17 shows how much funding is attached to each criterion. While
a sizeable share is attached to research performance, other criteria carry less weight.
However, it has be borne in mind that there is also a psychological effect attached to
the criteria, encouraging RUs to have good scores in a criteria because if they do not it
is a matter of (bad) reputation to them.
type of research organisation
Default scenario ScRO RTO PubSL NatRes
The share of PRFS 15% 15% 15% 15%
Research Excellence 1.5% 0.75% 0.75% 0.75%
Research Performance 7.5% 7.5% 7.5% 7.5%
Societal Relevance 1.5% 3.0% 3.0% 3.0%
Membership in the World Research Community 1.5% 0.75% 0.75% 0.75%
Management 3.0% 3.0% 3.0% 3.0%
Source: Technology Centre. ScRO=scientific research organisations; RTO=research and technology
organisations; PubSL=public service research organisations; NatRes=national resources/infrastructures
• Scientific research organisations: According to the definitions in the First Interim
Report59, scientific research organisations are universities, Academy of Sciences
institutes and university hospitals60. Other research organisations can decide for
themselves whether they belong to this type of research organisation61. The main
rationale for the weights for scientific research organisations in the default
scenario is that for scientific research organisations ‘research excellence’, ‘research
performance’ and ‘membership in the world research community’ should make up
the bulk of the funding because the define quality from different perspectives.
Management is also important, as it is a typical bottleneck in the Czech Republic62
(and in other countries), while societal relevance for scientific research
organisations lies mainly in training of young researchers and the publication of
papers addressed to peers (i.e. other academic researchers) and is thus of lesser
importance.
The weights for the more applied research organisations are all the same in the default
scenario because quality and societal relevance are important to all of them. However,
societal relevance has to be conceived differently for the different types of research
organisations.
• RTOs: RTOs will mostly be sectoral research organisations and industry research
organisations63. This ‘entrepreneurial’ sector includes researchers employed in
manufacturing and service-providing enterprises, but also in a broad range of
Methodology EM2015, p. 26-27
60 In the evaluation, it will be key to include peers from university hospitals in the panel.
Methodology EM2015, p. 28.
62 See Erik Arnold et al., The Quality of Research, Institutional Funding and Research Evaluation in the
Czech Republic and abroad. Thematic Report No. 3, International Audit of Research, Development &
Innovation in the Czech Republic, 2011.
63 Cf. R&D Evaluation Methodology and Funding Principles, First Interim Report: The Evaluation
Methodology EM2015, p. 28.p. 29

private enterprises offering R&D services to industry. The latter include industry-
oriented research institutions that took up the role of public RTOs under
communist times and survived the privatisation wave in the beginning of the
1990s.
Given their mission, quality and relevance appear to be the most important
categories for RTOs.64 Societal relevance of public research laboratories must be
conceived in terms of their target groups, which are mostly industry and industry
sectors, but also public institutions / authorities. Management is important as
well, as it always is. Research excellence is of less importance for RTOs than for
scientific research organisations; nonetheless applied research can be excellent as
well, which is why we have taken into consideration research excellence as a
category but given is less weight than in the case of scientific research
organisations.
• Public service research organisations: Public service research organisations are
typically governed by specific ministries (such as the Ministry of Agriculture or the
Ministry of Transport). In the Czech Republic, some of them have been transferred
to the responsibility of MEYS. Given that public service research organisations
have the public administration (together with user communities) as key target
group, they ought to be tied more closely to ‘their’ specific ministry again65. This
also means that they ought to be appropriately funded and governed. The funding
system proposed here does not define an appropriate level of funding for research
organisations because that is a decision that can only be made policy-makers. So it
is up to the founding ministries to make sure that their public research
organisations receive a level of institutional funding that ensures their long-term
sustainability and institutional development.
Public service research organisations fulfil a specific mission in the Czech system
which goes beyond the ‘pure’ production of research. Typically, their primary task
is to provide expert services essential to the state (e.g. criminal fire investigations),
and research is done for capability development. Other public service labs do
primarily conduct research but have as beneficiaries/applicants the public sector
and/or user communities that would never commission research on their own
(e.g. fish farmers, people living in flood areas). Hence, to some extent, public
service research organisations ought to be protected from science policy
considerations. This does not mean that they do not have to comply with quality
standards in research! On the contrary, quality and relevance are the most
important considerations for public service laboratories and has been reflected as
such in the weights.
Societal relevance of public service research organisations must be conceived in
terms of their target groups, which are ministries, other public administration
bodies, civil society organisations and generally the user communities they are
serving (e.g. famers). The specific role of public service research organisations
implies that other relevant activities aiming at the use of research results need to
64 When composing the panels and selecting the panel members attention and care has to be taken not to
rely on university professors as a default solution. The evaluation of RTOs (like the evaluation of all other
research organisations) must be done by people who are peers not only in the research subject but also in
terms of institutional background, i.e. by representatives from other RTOs (e.g. the director of a foreign
RTO) because only peers are able to truly understand the mission and role of an RTO.
65 This is in line with policy objective no. 8 of the National RDI Policy of the Czech Republic 2009 – 2015,
according to which “the compatibility and linkages of the National Policy with other sectoral policies
[should be] ensured.” This implies in particular that the definition of what is a public service research
organisation should not be left to public service research organisations themselves, the ministries should
be involved in the definition as well.

be taken into account, often more so than commercialisation activities (e.g.
workshops to teach a new pruning method or a publication in a farmers’ gazette).
For public service labs, membership in the world research community may have to
be re-interpreted to take into account that they may (and need to) be members in
more professionally oriented rather than research communities. Finally, the way
research excellence is defined in the new Evaluation Methodology is less relevant
for public service labs
• National resources/infrastructures: These encompass organisations like
museums, libraries, archives or public agencies.66 Often research is not their main
activity, and the research they conduct is geared towards the upgrading and better
exploitation of their collections. As in the case of the other types of research
organisations, quality and relevance are key. National resources/infrastructures
are primarily resources for other researchers, to a lesser extent to other users.
Hence, societal relevance has to be understood in terms of their user communities,
and interaction and collaboration with other researchers is key.
3.4.1.2 The medium scenario – increasing overall quality while more strongly
emphasising the specificity of types of research organisations
In this scenario, research performance still has a highest weight among all criteria,
thus pushing towards an overall increase in quality in research organisations. At the
same time, the specificity of different types of research organisations is emphasised
more strongly. In particular, more weight is given to research excellence in scientific
research organisations while social relevance is more strongly emphasised for the
public service research organisations and the national resources/infrastructures as for
the latter public service is their primary task.
Figure 18 Weights in the medium scenario
Medium scenario ScRO RTO PubSL NatRes
Management 15% 20% 20% 20%
Total 100% 100% 100% 100%
Figure 18 shows the share of total institutional R&D funding each criterion has. The
simulations below will show how large the differences in funding are between the
default and the medium scenarios. While our expectation is that differences will not be
large, the psychological effect must not be underestimated.
Methodology EM2015, p. 29.

Figure 19 Translation of weights into shares of total institutional funding for, by type
of research organisation
Medium scenario ScRO RTO PubSL NatRes
Management 2.3% 3.0% 3.0% 3.0%
3.4.1.3 The radical scenario – pushing research excellence and societal relevance
respectively
Our last scenario emphasises much more strongly the specificities of types of research
organisations. In effect, this means that research excellence is strongly emphasised for
scientific research organisations while societal relevance is strongly emphasised for
the applied research organisations. The rationale behind this scenario is that there
should be a strong push towards achieving the policy objectives of excellence for
scientific research organisations and relevance for the applied research organisations.
For the applied research organisations the second most important criterion is quality;
this is an important added aspect of the radical scenario because relevance without
quality is worthless. For RTOs quality consists of research excellence (5%) and
performance (10%) while for public service research organisations and national
resources/infrastructures quality is solely based on research performance (15%).
The radical scenario has the added advantage of showing the sensitivity of the weights
in the simulations.
Figure 20 Weights in the radical scenario
ScRO RTO PubSL NatRes
Management 10% 10% 10% 10%
Total 100% 100% 100% 100%
Figure 21 shows that more than 10% of total institutional funding for scientific
research organisations will be governed by research excellence, i.e. more than two
thirds of the PRFS part of the funding; hence the radical scenario will have a strong
effect of concentrating funding on the most excellent scientific research organisations.
As for the applied research organisations, their institutional funding will be strongly
influenced by how relevant their work is, concentrating funding on the most relevant
research organisations.

type of research organisation
Radical scenario ScRO RTO PubSL NatRes
Management 1.5% 1.5% 1.5% 1.5%
3.4.2 Options for translating scores into funding
In this section we explain what algorithms can be used for translating the scores into
funding. We present two options, showing their pros and cons at the end of the
section. Initial simulations suggest that the first option can address some weaknesses
of the previous system but that it is very sensitive to the accuracy of data about the
number of full-time equivalent researchers engaged at each research unit being
assessed. It may benefit from a non-linear transformation of scores into funding. The
second option will reward good performance and penalise weak performance but
suffers from its strong dependence upon the funding status quo and will therefore only
very slowly correct past anomalies or misallocations.
Since the translation of scores into funding is quite technical, we will use examples to
illustrate the two options.
3.4.2.1 Option 1 – Allocation of PRFS based on size (person scores)
As an example, we take a fictional pot XY which represents the institutional R&D
funding for scientific research organisations. This pot contains a certain amount of
money, say 100,000 CZK.
Now, institutional R&D funding has different components, which in turn have
different shares of the budget to be allocated (see above):
• Block grant: makes up 80% of total institutional R&D funding
• Performance-based research funding system PRFS: 15%
• Performance agreement: 5%
If the PRFS makes up 15% of institutional R&D funding, this gives us 0.15 * 100,000
crowns to distribute through the PRFS: 15,000 CZK, i.e. the PRFS pot. The PRFS
suggested for the Czech Republic in the First Interim Report67 gives us scores from 1 -
5 against five evaluation criteria68, similar to the UK Research Exercise Framework.
The basic idea is to allocate the PRFS pot among RU based on the scores they
obtained, and taking into account RU size. With regard to normalisation of RU size, we
can use a measure of the number of researchers that work at the RU. In the following
we are presenting two options how this can be done.
Methodology EM2015, p. 69-71
68 Research excellence, Research performance, Management, Societal relevance, Membership in the world
research community

Central register
Ideally, a central register of researchers in public institutions and private institutions
that receive public funding would be (re-)installed, specifying who is employed to what
extent at a certain organisation (e.g. 80% at XY, and 60% at AB) and to what extent
they conduct research (rather than teach or provide services). This would allow to
calculate the research FTEs (full time equivalents) per RU and rule out gaming with
FTEs. However, this is an ideal state which presumably will not be achieved in a short
time because it would require a huge effort on the part of all stakeholders to put
together such a central register.
While from our perspective this is the preferred option because it is the most accurate
and rules out gaming, we are also presenting an option that is less demanding.
Using data from the evaluation
The second option deals with the following main issues related to the use of FTE
researchers as an indicator for the size of the RO in the formula for PRFS
• The reliability of the data
• The multiple contracts per researcher and related risk of gaming
• The calculation of research versus teaching time (and other tasks)
Since there are no reliable official statistics on FTE researchers available at the level of
research organisations/evaluated units (EvU), this option suggests obtaining the data
directly from ROs/EvUs in course of the evaluation process. Specifically, the EvUs will
be asked at the time of registration of RUs to specify for each researcher (head count
(HC)) the time spent on research within the particular EvU (FTE). Alternatively,
stakeholders can reach a consensus based on Czech Statistical Office data (see Figure
22 below), which specifies the amount of time spent on research at sector level. By way
of example, the 0.5 division in the higher education sector (50% teaching, 50%
research) is very much in line with divisions in other countries (e.g. Norway).
Figure 22 FTE parameter at sector level
HC FTE FTE per HC
Business Enterprise sector 16.698 13.958 0.8
Government sector 8.220 6.235 0.8
Higher Education sector 20.732 10.289 0.5
Private Non-Profit sector 251 199 0.8
CZ total 45.902 30.682 0.7
Source: Czech Statistical Office
Names of researchers together with their working capacity spent on research will be
matched with unique identifiers that have already been introduced in the RDI IS (as
an alternative the open researcher identifier ORCID can be used). This enables to track
multiple contracts for each researcher as well as to distinguish between the time spent
on research and teaching (especially at higher education institutions).
The data on HC and FTE researchers submitted by EvUs will be used for both the
evaluation as well as the formula for calculating the PRFS component of institutional
R&D funding. This set up has a built-in correction mechanism against gaming since
there are two opposite forces that discourage EvU from over- or underestimating the
number of FTE researchers. On one hand, overestimation of FTE researchers poses a
risk of penalty for dishonesty in evaluation (zero score) and of a lower score for
research performance (the more FTE researchers for the same research output, the
worse the score for performance). On the other hand, underestimation of FTE leads to
lower institutional R&D funding as well as again to the risk of penalty. Thanks to the
easy control of multiple contracts on the national level, the authorised body for
decision-making (RD&I Council) can be also be notified in cases of multiple 100% FTE

researchers. The authorised body can then take decisions about financial
consequences as a result of any oddness in the evidence of multiple contracts.
Definition of researcher
Both options – the central register and using data from the evaluation – will need to
work with a definition of what a researcher is and what categories of staff are included
as researchers. The first thing to keep in mind is that the PRFS focuses on research,
research quality and research performance; hence its focus is on researchers. Against
this background we propose the following definition of researcher: a researcher is a
PhD holder active in research. This has two implications:
• PhD students are still in research training, so they are not fully fledged researchers
and will not be counted as researchers. This convention is in line with
international practice; other PRFS do not count PhD students as researchers
either.
• Technicians have a support and assistance function in research rather than
conducting research themselves. Hence, they should not be counted as
researchers. Again, this convention is in line with international practice as the
large majority of PRFS does not include technicians towards the counting of
researchers. The only exception is the Italian PRFS69. Some disciplines have more
technicians than others (e.g. the technical sciences compared to the humanities),
which is not reflected by the proposed definition of researcher. Indeed,
institutional R&D funding is granted to cover parts of the costs, and disciplines
that require technicians tend to be more expensive than those that do not.
However, this should be covered by the block grant rather than the PRFS
component of the funding system, which focuses on research quality and research
performance.
• The proposed definition of researchers has the advantage of not overly
complicating the system.
Because the issue of FTEs is fairly tricky, we have devised a second option of
translating scores into funding, which does not use FTEs to calculate the funding
(section
69 The Italian VQR 2004-2010 is the only national PRFS where technicians were also included; however,
this regarded only technicians that had published in the evaluation period (as authors or co-authors) and
excluded those whose tasks were uniquely administrative and/or for service delivery. The body responsible
for the evaluation (ANVUR) explained the inclusion by stating that technicians in research institutions are
normally involved in research activities and therefore appear as (co)authors of publications. The decision
has created quite some problems for the Italian research community and was heavily contested. The main
reasons were:
• The title of “technician” was introduced some years ago and implies that these employees have a
different role in the institution and their activities lead to different outputs than the ones of
researchers.
• Second, the academic profile of technicians is different depending on the institutional context and
needs: while in principle technicians do not have a PhD, in some institutions it includes employees
with and without PhDs
• Third, there are substantial differences in different fields as well as in research institutes versus
universities of the role and function that technicians have, for research and for the institution as such
(in some universities and depending on the field of science, they also have teaching functions)
In the Italian system, technicians fulfil tasks of review of analyses and technical collaboration related to
technological or professional activities.

Example70
Below is an example of how the PRFS pot (15,000 CZK) is divided among two research
units with different FTEs and scores. The PRFS pot consists of 15,000 CZK.
RU1: 10 R&D FTE x score of 4 = 40 [weighted R&D FTEs)
RU2: 18 R&D FTE x score of 2 = 36 [weighted R&D FTEs]
This implies that total weighted R&D FTEs = 76
Share of PRFS pot RU1 gets: 40/76 à translated into money = 15,000 * 40/76 = 7,894
CZK
Share of PRFS pot RU2 gets: 36/76 à translated into money = 15,000 * 36/76 = 7,105
CZK
Institutional R&D funding for RU1 + institutional R&D funding for RU2 = 7,894
+7,105 =15,000 CZK
Figure 23 Translation of scores into points
No. of R&D
FTE
Evaluation
score
Weighted
R&D FTE
Share of
PRFS pot
Amount of money
from PRFS pot
RU 1 10 4 40 40/76 7,894
RU 2 18 2 36 36/76 7,105
Total 28 76 15,000
Unclassified RU (i.e. RU with a score “0”) do not get any money from the PRFS pot as
0 x n = 0.
As we have shown above (section 3.2.2), the recipient of the money is the legal entity,
i.e. the research organisation. However, how large the “PRFS pot” is for the research
organisation is determined at the level of the RU. Thus, for each RU we can calculate
the amount of money its scores are equivalent to; this amount of money can then be
added up at the level of EvU (e.g. a faculty) and then at the level of entire research
organisation (e.g. a university). In other words: the money from the PRFS pot is
allocated to e.g. a university (= governance unit) and the amount of money it receives
is the total of PRFS funds earned by all RU of this particular university together
(Figure 13).
3.4.2.2 Option 2 - Allocating PRFS funding based on relative scores
The outcomes of the evaluation will be scores for individual RUs in five evaluation
criteria. For example:
Figure 24 Examples of scores of RUs, by evaluation criteria
Crit. 1 Crit. 2 Crit. 3 Crit. 4 Crit. 5
RU1 2 4 1 3 2
RU2 3 2 1 5 2
RU3 4 3 2 2 2
Scores for research organisations can be calculated as a weighted average of scores
achieved by individual RUs, where the weights are set based on number of FTE
70 In this example we use only one score to clearly illustrate the basic allocation principle. Below (see
scenario 1) we will describe an allocation mechanism for the five criteria suggested in the new Evaluation
Methodology, based on the mechanism in this example.

researchers in that particular RU. Scores for research organisations will not be made
public.
Following the example:
Figure 25 Examples of average scores of research organisation, by evaluation criteria
Crit. 1 Crit. 2 Crit. 3 Crit. 4 Crit. 5 FTE
RU1 2 4 1 3 2 20
RU2 3 2 1 5 2 8
RU3 4 3 2 2 2 50
RO1 3.38 3.15 1.64 2.56 2.00 78
We can do the same aggregation at the national level for each type of research
organisation with respect to their mission, so that we can get an average “mission-
based” national score (weighted average) for each criterion.
Following the example:
Figure 26 Example of aggregating scores at national level, by evaluation criteria
Crit. 1 Crit. 2 Crit. 3 Crit. 4 Crit. 5 FTE
RU1
RU2
RUx
RUz
National –
ScROs*
2.35 3.07 1.89 3.65 2.19 15 000**
*ScRO=scientific research organisation; ** This is sum of FTE researchers for all RUs at ScRO in the Czech
Republic. The number 15.000 is only an example, the actual sum will probably be higher.
This average national score for the four types of research organisations sets a basis for
decision about the increase, decrease or no change of the PRFS component at
individual research organisations in a particular group. The decision about increase,
decrease or no change is ultimately a policy decision that needs to be made by policy-
makers.
Below we give an example of how the increase, decrease or no change can be
implemented. The actual shares have to be discussed and agreed on in a policy
process. Of course, our suggestions may serve as a starting point for discussion.
• RO scores less than 50% of the national average: it gets 50% less than in the
previous period
• RO scores more than 50% but less than 75% of the national average: it gets 25%
less than in the previous period
• RO scores more than 75% but less than 125% of the national average: it gets the
same as in the previous period
• RO scores more than 125% but less than 150% of the national average: it gets 25 %
more than in the previous period
• RO scores more than 150% of the national average: it gets 50% more than in the
previous period
Following our example the funding implications for the RO1 would be as follows:

Figure 27 Example of funding implications, by evaluation criteria
Crit. 1 Crit. 2 Crit. 3 Crit. 4 Crit. 5
RO1 (ScRO*) 3.38 3.15 1.64 2.56 2
National (ScRO*) 2.35 3.07 1.89 3.65 2.19
RO1/National 1.438298 1.026059 0.867725 0.70137 0.913242
Funding implication +25% Same Same -25% Same
*ScRO=scientific research organisation
As stated in the fundamental principles the total institutional funding for RO is
separated into four pots according to the mission-based classification of research
organisations (section 3.2.1). Furthermore, according to the principles the decision
makers at the national level set weights for different criteria and different types of
research organisations (section 3.4.1).
This enables us to allocate the total PRFS institutional support in each pot (mission-
based) to sub-pots (evaluation criterion-based). Each research organisation then gets
the respective amount (same/higher/lower) of institutional support from each sub-pot
(Figure 28). In the end these amounts of institutional funding for RO will be adjusted
according to the total available PRFS component in each pot by multiplying the
calculated amounts by coefficient c = amount of PRFS in the pot for period 2/ amount
of PRFS in the pot for period 1.
Figure 28 Example of funding implications for research organisations
Funding in
period 1
Results of
evaluation
1x (funding
in period 1 x
result of
evaluation)
Budget
adjustment
coefficient
Funding in
period 2
RO1 20 +25% 25 30.3
RO2 180 +25% 225 272.7
RO3 70 -50% 35 42.4
RO4 80 -25% 60 72.7
RO5 100 +50% 150 181.8
Total in pot 450 495 1,212121 600
An important feature of option 2 is that we do not use FTE for calculating funding. The
funding is based, first, on the previous PRFS part of funding, second, on relative scores
of research organisations compared to the national average in respective groups of
research organisations and, third, on the total amount of PRFS funding. FTE are used
only for the purpose of weighting scores achieved by individual RUs, i.e. calculating
average scores for research organisations. If FTEs are not quite precise, this will only
have a small effect on average scores. Therefore, the space for gaming with FTEs is
negligible and we can simply use the FTEs showed by the RUs in the evaluation forms
(in SPE it is the form 3b). However, because the second scenario relies on relative
changes with respect to the current situation, it is likely to engender less change than
the first scenario (see also simulations below).
Compared with option 1, option 2 requires an additional decision by policy-makers
about the increase, decrease or no change of the PRFS component at individual
research organisations in a particular group of research organisations (see list above
Figure 27). It needs to be decided what the limits of the bands are (e.g. between 50%
and 75% from the national average) and how large the changes relative to the previous
funding period in these bands are (e.g. if RO scores more than 50% but less than 75%
of the national average, it gets 25% less than in the previous period).

What is more, scenario 2 does not take into account research active FTEs in the
calculation of funding, hence the growth of research organisations during a funding
period (e.g. through competitive funding) is not considered in the following funding
decision. In contrast, scenario 1 takes into account research active FTEs in the
calculation of funding. Therefore, if a RU grows from one evaluation round to the next
based on competitive funding, the increased number of research active staff will be
used for calculating PRFS funding for the second funding period.
In both scenarios, we are presenting a linear allocation of funding. A non-linear
allocation of funding would of course also be possible. In the case of non-linear money
allocation, as in the British Research Excellence Framework (REF), the top band gets a
larger share of money than the ones below, which leads to a concentration of funding
on the best performing research organisations. Whether or not to use a non-linear
allocation, is a policy decision because it has strong effects on the R&D system as this
leads to a concentration of funding on the best research organisations. For the first
funding period, we would advise using a linear allocation in order to allow for stability,
continuity and in particular organisational capacity building, which is needed for
improved governance of institutional R&D funding.71
3.4.3 The issue of RUs with low scores
RUs that receive low scores, in particular 0 (‘unclassified’), in one or several evaluation
criteria ought to get special attention as the low scores are indicative of a problem. The
immediate consequence is, of course, that the RU with low score will contribute little
or nothing to the PRFS share of funding allocated to the research organisation it
belongs to. This is already an incentive for the management to get active.
If RUs belong to a university then we would recommend the university management to
take a closer look at the RU. The evaluation report by the peers will explain why they
have given a RU a low score in a certain area. So this will be the starting point for the
university management. The range of options the university management has in
reaction to bad scores is large, ranging from better funding the RU (because its low
scores might be due to lack of funding) to phasing out the RU. In other words, there is
no single reaction to low scores. The same holds for RUs which are one of several RUs
at larger research organisations.
If a RU is equivalent to a research organisation, for example an RTO, and receives low
scores in the evaluation, then we would invite the principal – the responsible ministry
(or founding ministry) – to take a closer look at the research organisation. Again, the
evaluation report would be the starting point. As in the case of university management
that is faced with a RU with low scores, the principal can react in a variety of ways.
However, as a guiding principle we would recommend research organisations’
management and principals to answer the following questions before they take action
of any kind:
71 In a second funding period, a non-linear allocation for funding could be used for scientific research
organisations if the policy objective is to boost excellence in scientific research organisations. However,
one should be aware that there are two parameters that can be influenced in order to support excellence:
a) a stronger weight of the evaluation criterion ‘research excellence’ (see the radical scenario in section
3.4.1.3); and b) a non-linear allocation of funding favouring the top-performers. We have presented two
options for translating scores into funding, the first based on person scores and the second on relative
changes. We have illustrated both with examples using a linear allocation of funding. Of course, a non-
linear allocation of funding can be used too. In the first scenario this would for example mean that a score
of 4 would count double. In the second scenario, this would mean that the group of scientific research
organisations in the highest band would get a overly large share of the PRFS pot. We strongly recommend
to simulate any possible effects beforehand.

• Does the RU play a specific and unique role in the Czech Republic? This might be
the case e.g. for the only Czech research organisation dealing with indispensible
issues (“somebody has to count the fish”) or for the only institution providing
tertiary education in certain subjects in a remote region or for an institution which
plays a special role in regional development.
• If yes: Does this specific role require (i) public (ii) institutional (iii) research
funding? If yes again, it makes sense to give the RU a chance.
− If the RU=research organisation, we would suggest that the problems be
addressed in the performance agreement that the principal (ministry) and the
agent (the research organisation) negotiate. The performance agreement
should specify measures (including organisational / managerial actions as an
option) that will “change the course” of the RU. This agreement needs to be
based on an analysis of the reasons behind the poor performance, taking into
account the findings of the peers. After three years the RU should be evaluated
again. If the assessment is still ‘unclassified’, then we would recommend more
drastic measures, such as phasing out the institutional funding for RO.
− If the RU is part of a larger organisation, e.g. a university, then the university
management should negotiate measures with the RU and/or take action that
would “change the course” of the RU. Again, this pre-supposes an analysis of
the reasons behind the poor performance, taking into account the findings of
the peers. After three years the RU should be evaluated again. If the
assessment is still ‘unclassified’, then we would recommend more drastic
measures.
The special treatment of RUs with low scores will also take care of RUs that did and do
well in the Metodika but do badly in the new Evaluation Methodology. If they achieve
unclassified or 1 scores, the special procedure applies to them, sorting out
discrepancies between the results of the Metodika and the new Evaluation
Methodology.
Box 2 – Scenario for phasing out institutional funding for RO to weak
research organisations
We have heard voices in favour of phasing out institutional R&D funding to weak
research organisations. In line with these voices, we are sketching a scenario that
foresees a strict treatment for RUs with low scores. We believe our default scenario
(4.5.3.3) is already a fairly strict system but of course it can be made stricter.
Of course, RUs gets scores for five evaluation criteria. So a decision needs to be taken
to which RUs the strict treatment applies. We suggest the following procedure:
• If a RU gets ‘unclassified’ or 1 in more than one evaluation criteria, or
• If a RU receives ‘unclassified’ or 1 for research performance, or
• If a RU in a scientific research organisation receives ‘unclassified’ or 1 for research
excellence, or
• If a RU in a RTO, public service research organisation or national
resources/infrastructure receives ‘unclassified or 1’ for societal relevance,
it should get into the group of RUs to whom the ‘strict treatment’ applies.
We would suggest conducting a first evaluation round and giving weak RUs with
scores ‘unclassified’ and 1 a chance to get better during the first funding period.
After three years the RU will be subject to an additional evaluation. If the assessment
is still ‘unclassified’ or 1, institutional R&D funding – the PRFS part, block grant and
performance agreement – to the weak RU (or more precisely its research organisation)
will be phased out by the provider if the RU=research organisation. Two rounds of

evaluations are necessary in order to have sufficient evidence and more political
legitimacy to phase out institutional R&D funding to weak RUs and their research
organisations.
The scenario needs to respect research organisations’ autonomy in terms of allocating
institutional R&D funding internally. If RUs belonging to a larger research
organisation have low scores in the second round of evaluation, they will not receive
any PRFS money. However, the research organisation’s autonomy means that its
management cannot be forced to phase out internal institutional R&D money to weak
RUs. However, providers can decide to reduce the block grant allocated to a research
organisation whose majority of RU(s) have had low scores in two evaluations in a row.
The difference between this scenario and the default scenario is
a) that phasing out of (parts of) institutional R&D funding to weak RUs (or their
research organisations) occurs mandatorily after two evaluation rounds and
b) that this treatment applies to both RUs which are ‘unclassified’ and those that get
scores of 1. Whether or not this treatment should be extended to RUs with scores of 1,
also depends on available resources and on the overall distribution of scores across the
same type of research organisations, thus on the average performance of similar
research organisations in the Czech Republic.
3.4.4 The issue of small RUs
One question is still open: How to deal with RUs which are too small to participate in
the new Evaluation Methodology exercise and thus do not have any access to the PRFS
pot of money?
Small in this context means that the researchers of an RU have published fewer than
50 outputs in one scientific field in the five or six years preceding the evaluation
according to the new Evaluation Methodology72. At the moment this holds for a total
of 98 RUs out of 841 RUs, thus one out of eight73. The present list contains units which
may not actually be involved in research (e.g. universities’ internal computer service
departments, support centres for students with special needs, language centres). These
units appear to have been registered as special parts of universities in the RIV
database but of course they are not research organisations and should not take part in
the new Evaluation Methodology. This needs to be verified. The definition of RUs in
the course of the new EM should take care of this problem. Hence, the problem of
small RU will be smaller than it appears to be now.74
Moreover, our current list of small RUs is based on publication data (only the
'scholarly' outputs) for the years 2008-2012, that is one to two years less than the
envisaged time frame for the evaluation methodology and the funding system. This
will lower the number of small RUs further.
Small RUs will not participate in the evaluation, which implies that they will not have
access to the PRFS pot of money. Without any regulation beyond those outlined above,
these RU will lose a potential 15% of their institutional funding for RO.
Methodology EM2015
73 The present list might contain units which are not actually involved in research (e.g. internal computer
service departments of universities). This needs to be verified. Such units should not qualify for
institutional R&D funding and need not be considered further.
74 A quick scan of the list shows that about ten RUs are service organisations. In the case of at least another
ten to fifteen RUs the status oft he RU is not clear.

Setting a minimum threshold for RUs to participate in the Evaluation Methodology is
sensible because it contains the cost of the assessment and creates an incentive for
RUs to grow or to merge.
That said, we suggest the following approach, distinguishing between two different
cases:
• Case 1: There is only one RU in the research organisation. Typically, this is the
case for national resources/infrastructures like archives or museums or public
service research organisations, i.e. organisations where research is not the main
task and activity and where research is often done in order to develop their
capabilities and to serve the main purpose of the institution. Here, research staff is
often small and academic publications are not their number one priority.
We suggest that in the first funding period of the new system, these organisations
start with a block grant of 80% of their institutional research funding for RO of the
preceding period (calculated in the same manner as the 80% block grant for RU
participating in the evaluation) plus 5% if a performance agreement is concluded
with the provider. Moreover, within a period of three years, the research
organisation and the funding provider in charge analyse the situation. We propose
that the research organisation fills in the self-assessment template used in the new
Evaluation Methodology (or a simplified variation thereof), which the provider
will then check and discuss with the research organisation. If the provider feels it
does not have enough expertise to check and discuss the self-assessment report, it
can call in an external expert. Together the research organisation and the provider
develop a strategy for the future, including the future of the public institutional
funding for RO (e.g. switching to performance agreement altogether, growing into
the new Evaluation Methodology within a certain period of time). This strategy for
the future should mandatorily be included in the performance agreement, as
described above (section 3.3).
• Case 2: The RU is part of a larger research organisation, e.g. an institute or
department of a university which also comprises other, larger RUs that do
participate in the evaluation. In this case the allocation of institutional R&D
funding happens like in case 1, (i.e. 80% or 85% in case the research organisation
concludes a performance agreement with the ministry), with the management of
the research organisation and the RU analysing the situation within a period of
three years and developing a strategy for the future. While this strategy for the
future cannot be in the form of a performance agreement (because there are no
performance agreements between research organisation management and RUs), it
should be laid down in writing. The issue of the small RU is mandatorily
addressed in the performance agreement concluded between the funding provider
and the research organisation.
In both cases, the specific situation of “small subjects” has to be taken into account in
order not to unintentionally reduce subject coverage or variety in the Czech R&D
system75.
75 This could also affect emerging fields and supporting disciplines. See also the German debate on „Kleine
Fächer“ („small subjects“): http://www.kleinefaecher.de/

Box 3 – Which parameters can be varied in the funding systems
To recapitulate, the following parameters can be varied in the funding system
• The shares of the different funding components (block grant, PRFS, performance
agreement. These can be varied for the different types of research organisations.
For the first funding period, we have suggested an allocation of 80% through block
grant, 15% through PRFS and 5% through performance agreements for all types of
research organisation because after the changes entailed by the coffee mill all
research organisations are in need of stability.
• The weights attached to the different evaluation criteria. These weights have a
direct impact on funding and need to be set in line with the missions of research
organisations and in line with policy objectives. We have suggested three scenarios
above: the default scenario aiming to increase overall quality in research
organisations; a radical scenario pushing research excellence in scientific research
organisations and societal relevance in applied research organisations; and a
medium scenario located in the middle between these scenarios aiming to increase
overall quality while more strongly emphasising research excellence for scientific
research organisations and societal relevance for applied research organisations.
• A linear or non-linear allocation of funding. For the first funding period we
recommend a linear allocation of funding in order to allow for stability, continuity
and in particular organisational capacity building. For the second funding period,
a non-linear allocation of funding may be considered for scientific research
organisations in order to boost excellence and concentrate funding on the top
performers.
• Treatment of RUs with low scores. We have suggested a default scenario and a
stricter scenario. While we prefer the default scenario because it does not
mandatorily prescribe phasing out of institutional R&D funding, given the
problems in the Czech Republic (e.g. fragmentation of funding, deformation of
funding allocation due to the problems of the Metodika) the stricter scenario may
also be applied.
In other words, we have proposed three scenarios – default, medium, radical, with
different weights attached to evaluation criteria. In these scenarios the rest of the
parameters are the same: 80% of available funding distributed through block grant,
15% through PRFS, 5% through performance agreement; linear allocation of funding
and default treatment of RUs with low scores.
The setting of these parameters in the future needs to be in line with research
organisations’ needs and requirements and with ministries’ strategies for research
organisations.
3.5 How can new research organisations enter the funding system
What happens if a new research organisation enters the stage? A new research
organisation could be e.g.
• A new public university
• A new public service research organisation
• A new private not-for-profit research institute

• Etc.
‘New research organisation’ in this context means a research organisation that is a
legal entity not eligible for public institutional R&D funding at the moment.76
The first guiding principle we are proposing is that the founder of a new research
organisation will be responsible for the funding of the new organisation (including
institutional R&D funding), either by funding the organisation or by providing access
to other funding sources or by a combination of both. This holds for any founder of
new research organisations, be it a ministry, the Academy of Sciences in its role as a
funding provider, a private company or a university. The advantage of this guiding
principle is that it creates responsibility and ownership for the founder.
This is the normal case in most countries. For example, in Austria, the federal
government together with the regional government of Lower Austria decided to set up
the graduate research institute IST Austria (Institute of Science and Technology
Austria), with both governments providing funding to the research organisation; the
new institute has been set up by a new law.
As a second guiding principle we propose that the decision of whether or not a new
research organisation should receive public institutional R&D funding has to be taken
outside the system established for allocating this institutional R&D funding (i.e.
outside the PRFS). In fact, a PRFS system cannot accommodate the exit and entry of
new research organisations as PRFS systems in other countries show. For example in
the UK, if a university closes a research unit the university gets funding for it until the
new assessment takes place. In a similar logic, organisations that newly enter the
research system are ignored in the UK performance based system until the next
exercise when their research performance is assessed. Before they enter the UK PRFS,
they are funded from special grants as the REF cannot accommodate the entry and
exit of research units or organisations.
The second guiding principle means that a procedure and criteria have to be set up by
which to decide about the eligibility of a research organisation for public institutional
R&D funding. In the Czech Republic at present there is an existing (but not fully
operational) two-step procedure defining the eligibility of research organisations to
receive institutional R&D funding, with the first step assessing the formal features and
the second step assessing the scientific features of research organisations (presently
research organisations are defined in the Act No 2011/2009 Coll. on the support of
research, experimental development and innovation77)78. This is quite an open
procedure developed by the R&D&I Council together with the Office for the Protection
of Competition. Either this methodology (perhaps in an adapted version) will be
followed under the new funding system to check the eligibility of research
organisations or a new methodology will replace it. Eligibility would, of course, have to
be verified for existing RO as well at the beginning of each funding period.
76 We do not consider new units within existing research organisations: If an existing research organisation
establishes a new unit (e.g. a university establishes a new institute), this lies in the autonomy of the
research organisation. The same rules apply as outlined for RU too small to participate in the new
Evaluation Methodology from the beginning: As soon as a (new) small RU has grown enough to pass the
size limits it can participate in the new Evaluation Methodology (see 3.4.4). Moreover, we do not consider
the 'Regional Research Centres' and 'Centres of Excellence', funded from the Structural Funds because
they are not independent research organisations and are funded from other sources than the institutional
R&D funding also in the current period (2014 – 2020).
77 Background report to the 2nd Interim Report: The institutional funding system in the Czech Republic.
78 The requirement that the research organisations generated at least 1,500 RIV points in the last five years
was relinquished in early November 2014.

A third guiding principle is that policy makers need set up guidelines as to how and
when a new research organisation enters the system of institutional funding for RO
(after having passed the eligibility test). For example, a new research organisations
might come into the new Evaluation Methodology once they have passed the threshold
of 50 outputs in the preceding five years. Thus they would enter into the new system of
institutional R&D funding we propose and they would have access to PRFS money.
The state budget (or the relevant chapters respectively) might need to be adjusted
accordingly if the institutional R&D funding available to the research organisations
that are already in the system is to be stable.
The country case studies show that in the countries we studied the decision how new
organisation enter the institutional funding system is decided outside the
(performance-based) funding system by policy decision. The case studies also show
that guidelines set up are typically for one type of organisation only.
For example, in Norway, the Ministry of Education and Research has set up guidelines
for the research institute sector which specify that national institutional funding can
only be given to institutions which fulfil a list of requirements, some of which are
performance-related (e.g. income from national and international commissioned
projects must represent at least 25 per cent of total R&D income). For the higher
education system, there is a separate accreditation system.79
In Sweden, in the higher education sector, all public HEIs receive institutional funding
whereas non-public HEIs must be approved by the government. In order to be
approved, HEIs must be accredited according to the Higher Education Act and offer
education free of charge for students. There are examples of new HEIs in Sweden,
both private and public. Those which were accredited receive institutional funding,
and those who were not been accredited do not. 80
In the Netherlands too, there is an accreditation system for higher education
institutions which entitles them to institutional funding. The universities that are
entitled to receive public institutional funding are mentioned in the Law on Higher
Education and Research. There are no other bodies, e.g. centres of excellence,
competence centres that receive institutional funding. 81
79 Background report to the 2nd Interim Report: Country analysis Norway.
80 Background report to the 2nd Interim Report: Country analysis Sweden.
81 Background report to the 2nd Interim Report: Country analysis Netherlands.

Appendix A - List of Abbreviations
Abbreviation Description
ASCR Academy of Science of the Czech Republic
BIS Department for Business, Innovation and Skills
BMVIT Federal Ministry for Transport, Innovation and Technology (Austria)
BMWFW Federal Ministry of Science, Research and Economy (Austria)
BOKU University of Natural Resources and Life Sciences (Austria)
CR Czech Republic
CRDI Council for Research, Development and Innovation
CZK Czech Crown
EvU Evaluated unit
EZ Ministry of Economics (NL)
FFG Research Promotion Agency (Austria)
FTE Full Time Equivalent
GACR Czech Science Foundation (Grant Agency)
GOVERD Government Expenditure on R&D
GTIs Large Technological Institutes
HC Head count
HEI Higher education institution
HR Human resources
KNAW Royal Dutch Academy of Arts and Sciences (NL)
MEYS Ministry of Education, Youth and Sports (CR)
NatRes National resources/infrastructure research organisation
NESTI
National Experts on Science and Technology Indicators - OECD Working
Party
NL Netherlands
NWO National Research Council (NL)
OCW Ministry of Education and Research (NL)
OECD Organisation for Economic Cooperation and Development
PhD Doctor of Philosophy
PRFS Performance-based Research Funding Systém
PSRO Public service research organisation
R&D Research and development
R&D&I Research, Development and Innovation
RAE Research Assessment Excercise (UK)
RCN Research Council of Norway

RCUK Research Councils UK
RD&I Research, development and innovation
RDI Council Council for Research, Development and Innovation
REF Research Excellence Framework (REF)
RIV Register for Information on Results (CR)
RO(s) Research organisation(s)
RTI Research, Technology and Innovation
RTO(s) Research and Technology Organisation(s)
RU(s) Research Unit(s)
ScRO(s) Scientific Research Organisation(s)
TACR Czech Technology Agency
UK United Kingdom
VINNOVA Swedish Governmental Agency for Innovation Systems
WP Work package

Draft of the Second Interim Report

More Related Content

What's hot (13)

Viewers also liked (20)

Similar to Draft of the Second Interim Report (20)

More from MEYS, MŠMT in Czech (20)

Recently uploaded (20)

Draft of the Second Interim Report