EUROPEAN COURT OF AUDITORS

12, rue Alcide De Gasperi

1615 Luxembourg

LUXEMBOURG

Tel. +352 4398-1

E-mail: eca-info@eca.europa.eu

Internet: http://eca.europa.eu

Twitter: @EUAuditorsECA

YouTube: EUAuditorsECA

More information on the European Union is available on the internet (http://europa.eu).

Luxembourg: Publications Office of the European Union, 2015

Print | ISBN 978-92-872-3435-3 | ISSN 1831-0834 | doi:10.2865/507985 | QJ-AB-15-019-EN-C
PDF | ISBN 978-92-872-3417-9 | ISSN 1977-5679 | doi:10.2865/619824 | QJ-AB-15-019-EN-N
EPUB | ISBN 978-92-872-3372-1 | ISSN 1977-5679 | doi:10.2865/609385 | QJ-AB-15-019-EN-E

© European Union, 2015

Reproduction is authorised provided the source is acknowledged.

For any use or reproduction of the Annex on p. 41, permission must be sought directly from the copyright holder.

HOW TO OBTAIN EU PUBLICATIONS

Free publications:

•one copy:
via EU Bookshop (http://bookshop.europa.eu);

•more than one copy or posters/maps:
from the European Union’s representations (http://ec.europa.eu/represent_en.htm);
from the delegations in non-EU countries (http://eeas.europa.eu/delegations/index_en.htm);
by contacting the Europe Direct service (http://europa.eu/europedirect/index_en.htm) or
calling 00 800 6 7 8 9 10 11 (freephone number from anywhere in the EU) (*).

(*)The information given is free, as are most calls (though some operators, phone boxes or hotels may charge you).

Priced publications:

•via EU Bookshop (http://bookshop.europa.eu).

EN
2015

NO 21

Special Report

Review of the risks related
to a results-oriented
approach for EU
development and
cooperation action

(pursuant to Article 287(4), second subparagraph, TFEU)

Audit team

The ECA’s special reports set out the results of its performance and compliance audits of specific budgetary areas or management topics. The ECA selects and designs these audit tasks to be of maximum impact by considering the risks to performance or compliance, the level of income or spending involved, forthcoming developments and political and public interest.

This performance audit was produced by Audit Chamber III which specialises in external actions spending areas. The Chamber is headed by ECA Member Karel Pinxten who is also the Reporting Member of this report. The audit was led by Philippe Froidure, director of Chamber III, supported by Hubert Devillé, assistant to the director.

From left to right: P. Froidure, K. Pinxten, H. Devillé.

Contents

Executive summary

1. Introduction

1.1. Why results matter

1.2. What is a result?

1.3. The overall framework relative to EU development and cooperation results

2. Results policy and regulatory background

2.1. Policy background

2.1.1. High-level development goals

2.1.2. High-level forums on aid effectiveness

2.1.3. EU communications

2.2. Legal and regulatory background

3. Our review

4. Key risk areas

4.1. Formulation of objectives and indicators

4.1.1. Terminology

4.1.2. Document alignment and consistency

4.1.3. Setting SMART objectives and clear indicators

4.1.4. Defining a partner country’s commitment to implement reforms as an objective

4.2. Establishment of the results chain

4.2.1. Linking actions and results through a clear logical chain

4.2.2. Contribution versus attribution

4.3. Increased complexity due to the integration of cross-cutting issues

4.4. Harmonisation between development partners’ instruments and frameworks

4.4.1. Alignment with partner countries’ results frameworks

4.4.2. Harmonising the instruments used by development stakeholders

4.4.3. Coping with the juxtaposition of different accountability structures

4.5. Weaknesses in evaluation and results reporting

4.6. Data consolidation and reporting

4.7. Data quality

4.8. Budgetary outturn

4.9. Changes in EU actions’ context

4.9.1. Risk management and choice of implementation modalities

4.9.2. Exogenous factors

5. Conclusions and recommendations

5.1. Conclusions

5.2. Recommendations

Annex
—
Comprehensive Evaluation Framework for Budget Support Evaluations

Reply of the Commission

Executive summary

I

Each year, the EU dedicates substantial resources to assisting the development of the world’s poorest countries. In order to establish the credibility of development assistance, it is essential that the results achieved with these resources can be demonstrated.

II

Recently, there has been renewed interest on the part of the European Parliament and the Council in shifting the focus of EU development and cooperation policies from activities to results. Also, in December 2013, the Commission issued a Staff Working Document on ‘Paving the way for an EU Development and Cooperation Results Framework’ and, in March 2015, presented the framework by issuing another Staff Working Document on ‘Launching the EU International Cooperation and Development Results framework’.

III

In our review, we identified nine key risk areas related to a results-oriented approach for EU development and cooperation action.

IV

Our review showed that these risk areas had been correctly identified by the Commission in a wide range of documents. It also showed the interest in the actions that are being undertaken by the Commission to launch an EU international cooperation and development results framework.

V

Nevertheless, further actions remain to be taken in order to harness the full potential of the Commission’s initiatives to improve EU development and cooperation results. In this context, we have formulated a number of recommendations that the Commission should take into account.

1. Introduction

1.1. Why results matter

01

Substantial resources are devoted to assisting the development of the world’s poorest countries. While emphasising the fact that there is no single source of data on development finance, the Commission estimated in July 2014 that the cumulative resources available to developing countries over the 10-year period from 2002 were in the region of 2 171 billion USD, including domestic and international public and private finance.

02

As part of the commitment to attain the internationally agreed Official Development Assistance (ODA) target of an ODA/GNI ratio of 0.7 %, in 2013, collective EU (including Member State) net ODA spending was 56.5 billion euros, which is equivalent to an ODA/GNI ratio of 0.43 % of the EU’s collective GNI¹.

03

The amounts scheduled to be made available for development and cooperation over the 2014-2020 period are 19 661 million euros through the EU general budget², and 30 506 million euros for ACP countries through the European Development Fund³.

04

In order to establish the credibility of development aid, in particular as regards the instruments used, aid delivery methods and the funds concerned, it is essential that the results achieved with this support can be demonstrated.

05

Public accountability is also an important issue. In a report published in January 2014, the European Parliament pointed out that ‘activity-based budgeting [was] still the fundamental principle when drafting the budget of the Union’, and regretted that some of the proposals for the 2014-2020 programming period remained ‘fundamentally input-based (expenditure oriented) and, therefore, still focused on compliance with the rules rather than on performance’. The Parliament also pointed out that the performance framework used by the Commission for the multannual financial framework 2014-2020 should encompass the following three main elements⁴:

(a)achievement of programme objectives (results);

(b)sound programme management by the Commission and the Member States;

(c)the contribution of programme results and sound management to the Union’s main objectives.

1.2. What is a result?

06

A result is a measurable consequence deriving — directly or indirectly — from a cause and effect relationship. A results-oriented approach to development and cooperation relies on the principle that the focus of development and cooperation interventions should be on the delivery of results, rather than on activity or process management. Results usually cover three categories⁵:

(a)output: what is produced or accomplished with the resources allocated to an intervention (e.g. training courses delivered to unemployed people, roads built);

(b)outcome: change that arises from an intervention and which normally relates to its objectives. Outcomes may be expected or unexpected, positive or negative (e.g. trainees who have found employment, improved accessibility to an area due to the construction of a road);

(c)impact: longer-term socioeconomic consequences that can be observed for a certain period after an intervention has been completed, which may affect direct beneficiaries of the intervention or other indirect beneficiaries.

Figure 1

Notion of results

1.3. The overall framework of EU development and cooperation results

07

Besides existing mechanisms aimed at measuring and reporting on results, a notable initiative taken by the Commission in December 2013 was the production of a Staff Working Document on ‘Paving the way for an EU Development and Cooperation Results Framework’. This initiative aimed to:

‘set out the processes (…) taken forward by the Commission to deliver against the ‘Agenda for Change’ commitment to strengthen its ability to monitor and report operational results achieved through the implementation of EU-funded development and cooperation projects and programmes’⁶.

08

This Staff Working Document took into account the conclusions of a study, undertaken by the Commission in 2013, which aimed to provide an analysis of lessons learnt from five development agencies which were already implementing their own results framework. It presented a preliminary approach to drafting an overall EU development and cooperation results framework. It described such a framework as follows:

‘A results framework is a tool that is used by different development partners to measure results achieved against strategic development objectives. The results framework can be seen as an articulation of the different levels, or chains, of results expected from the implementation of a particular strategy, programme or project’.

09

The EU development and cooperation results framework will have two main purposes: as an accountability tool — in order to inform stakeholders of the results that have been achieved — and as a management tool — in order to provide performance data as a basis for management decisions.

10

In June 2014, another Commission Staff Working Document was issued, aiming to describe ‘the framework for monitoring, evaluation and reporting to the end of the next multiannual financial framework that results from the legal instruments supporting the next generation of programmes and the additional work of the Commission to complete the framework’ — the ‘MORE framework’. This Staff Working Document provides, for each policy area, an overview of the general and specific objectives of the instruments, the main indicators attached to each objective, monitoring arrangements, and the timing and content of evaluations and reporting⁷.

11

In April 2014, the European Parliament expressed its support for ‘the Commission’s continuing efforts to shift from an input-based to a performance-and impact-oriented approach’⁸. In May 2014, the Council welcomed the Commission’s Staff Working Document of December 2013 on ‘Paving the way for an EU Development and Cooperation results framework’ and expressed its interest in the ‘results of the piloting phase, with a view to the timely finalisation of the EU development and cooperation framework, so that reporting on results achieved during 2014 can be presented in the first semester of 2015’⁹.

12

In March 2015, the Commission issued a new staff working document on ‘Launching the EU International Cooperation and Development Results Framework’. This document aimed to translate the approaches set out in the 2013 Staff Working Document (see paragraph 7) into operational DG International Cooperation and Development plans to report annually as of 2015 in order to demonstrate how funds spent contribute to the achievement of the policy objectives¹⁰.

1SWD(2014) 235 final — ‘EU Accountability Report 2014 on Financing for Development — Review of progress by the EU and its Member States’.

2http://ec.europa.eu/budget/mff/figures/index_en.cfm#documents, January 2015.

3Commission publication on 'Multiannual Financial Framework 2014-20 and EU Budget 2014 — The figures'.

4Report on the evaluation of the Union’s finances based on the results achieved: a new tool for the European Commission’s improved discharge procedure — A7-0068/2014, 30 January 2014.

5For the purposes of this report, we use the terminology for these three categories of results as stated in external action financing instruments. Similar terms may be used differently in other EU documents.

6SWD(2013) 530 final of 10 December 2013.

7SWD(2014) 200 final of 26 June 2014 ‘Overview of the Monitoring, Reporting and Evaluation Frameworks for the MFF 2014-2020 Programmes — Accompanying the document Report from the Commission to the European Parliament and the Council on the evaluation of the Union’s finances based on the results achieved’.

8European Parliament Decision of 3 April 2014 on discharge in respect of the implementation of the general budget of the European Union for the financial year 2012, Section III — Commission and executive agencies — ref. P7_TA-PROV(2014)0287, section ‘External relations’, paragraph 264.

9‘Council conclusions on an EU Development and Cooperation Results Framework’ — Council of the European Union Foreign Affairs (Development) meeting, 19.5.2014.

10SWD(2015) 80 final of 26 March 2015.

2. Results policy and regulatory background

2.1. Policy background

13

The role of the Commission on the international stage is set out in Article 17 of the Treaty on the European Union. The Commission works, in particular, with the United Nations and the Organisation for Economic Cooperation and Development (OECD) on development cooperation issues. In this context, high-level development goals are agreed, together with principles for providing more effective aid¹¹.

2.1.1. High-level development goals

14

In September 2000, the General Assembly of the United Nations adopted the United Nations Millennium Declaration, which committed its members to a new global partnership with the aim of reducing extreme poverty and set a series of time-bound targets — with a deadline of 2015 — which are commonly referred to as the Millennium Development Goals (MDGs).

15

In June 2012, the Rio+20 Conference agreed to launch a process to develop a set of Sustainable Development Goals (SDGs), which will build upon the Millennium Development Goals¹². In September 2013, world leaders renewed their commitment to meet the MDG targets and agreed to hold a high-level summit in September 2015 to adopt a new set of goals to build on the achievements of the Millennium Development Goals¹³. Global discussions on the wider agenda beyond 2015 are currently ongoing through various processes. A Synthesis Report on the post-2015 sustainable development agenda was published by the UN Secretary-General in December 201414,15.

2.1.2. High-level forums on aid effectiveness

16

The European Commission and EU Member States have taken part in several high-level forums aiming to improve the effectiveness of development aid programmes:

(a)the Rome Declaration on Harmonisation (2003);

(b)the Paris Declaration on Aid Effectiveness (2005);

(c)the Accra Agenda for Action (2008);

(d)the Busan partnership for effective development cooperation (2011);

(e)the Mexico First High-Level Meeting of the Global Partnership for Effective Development Cooperation: Building Towards an Inclusive Post-2015 Development Agenda (2014).

17

In these forums, a consensus emerged among donors and partner countries, and a number of principles relating to development results were agreed, including:

(a)strengthening partner country governments’ ability to assume a greater leadership role and take ownership of development results;

(b)working with partner countries to rely, as far as possible, on partner countries’ frameworks for results-oriented reporting and monitoring;

(c)harmonising donors’ monitoring and reporting requirements;

(d)improving management for results by donors and partner countries;

(e)increasing the focus on development results.

2.1.3. EU communications

18

The European Union’s commitment to internationally agreed development goals and making aid more effective has been documented in several public statements and communications, all of them expressing, in one way or another, the importance of development and cooperation results:

(a)Joint statement by the Council and representatives of the Member States’ governments on the European consensus for development (ref. 14820/05 of 24 February 2006);

(b)Supporting developing countries in coping with the crisis (COM(2009) 160 final of 8 April 2009);

(c)A 12-point EU action plan in support of the Millennium Development Goals (COM(2010) 159 of 21 April 2010);

(d)Operational Framework on Aid Effectiveness — General Secretariat of the Council (ref. 18239/10 of 11 January 2011);

(e)Increasing the impact of EU Development Policy: an Agenda for Change (COM(2011)637 final of 13 October 2011);

(f)A decent life for all: Ending poverty and giving the world a sustainable future (COM(2013) 92 final of 27 February 2013);

(g)Beyond 2015: towards a comprehensive and integrated approach to financing poverty eradication and sustainable development (COM(2013) 531 of 16 July 2013).

2.2. Legal and regulatory background

19

The main provisions concerning accountability for results and public audit¹⁶ can be found in the Treaty on the Functioning of the European Union (TFEU), in financial regulations for the EU general budget and European Development Funds, and in the regulation on common rules and procedures for the implementation of the Union’s instruments for financing external action.

(a)The Treaty on the Functioning of the European Union provides that:

(i)the Commission shall submit to the European Parliament and to the Council an evaluation report on the Union’s finances based on the results achieved (Article 318(2)); and

(ii)the Court of Auditors may, at any time, submit observations, particularly in the form of special reports (Article 287(4))¹⁷.

(b)The Financial Regulation applicable to the general budget of the Union (the Financial Regulation) stipulates the following18,19.

(i)Specific, measurable, achievable, relevant and time-bound objectives must be set for all sectors of activities covered by the budget. The achievement of those objectives shall be monitored by performance indicators for each activity (Article 30(3)).

(ii)In order to improve decision-making, institutions must carry out ex ante and ex post evaluations of all programmes and activities which entail significant spending (Article 30(4)).

(iii)The Commission must attach to the draft budget a number of documents, including activity statements containing information on the achievement of all previously set specific, measurable, achievable, relevant and timed objectives for the various activities, as well as new objectives measured by indicators (Article 38(3)(e) (i)), a full justification including a cost-benefit analysis for proposed changes in the level of appropriations (Article 38(3) (e)(ii), and a summary of evaluation results when relevant to budget changes (Article 38(3) (e)(v)).

(iv)The authorising officer by delegation must report to his or her institution on the performance of his or her duties in the form of an annual activity report containing financial and management information and indicating the results of the operations by reference to the objectives set, the risks associated with those operations, the use made of the resources provided and the efficiency and effectiveness of internal control systems (Article 66(9)).

(v)The Court of Auditors’ special reports must be transmitted without delay to the European Parliament and the Council (Article 163).

(c)The regulation on common rules and procedures for the implementation of the Union’s instruments for financing external action stipulates the following²⁰.

(i)Financing decisions must include in an annex a description of each action, specifying its objectives, main activities, expected results, methods of implementation, budget and indicative timetable and performance monitoring arrangements (preamble, paragraph 5).

(ii)The Union’s external action under the Instruments must contribute to clear results (covering outputs, outcomes and impacts); the results of the Union’s external action and the efficiency of a particular Instrument must be monitored and assessed on the basis of predefined, clear, transparent and, where appropriate, country-specific and measurable indicators (preamble, paragraph 12).

(iii)Action programmes must specify for each action the objectives pursued, the expected results and the main activities (Article 2).

(iv)From 2015, the Commission must submit to the European Parliament and to the Council an annual report on the achievement of the objectives of each regulation by means of indicators, measuring the results delivered and the efficiency of the relevant instrument; this report must assess the results of the Union’s financial assistance using, as far as possible, specific and measurable indicators of its role in meeting the objectives of the instruments (Article 13).

11http://ec.europa.eu/europeaid/policies/international-development-dialogue_en

12Rio+20 outcome document ‘The Future We Want’.

13United Nations — General Assembly — Integrated and coordinated implementation of and follow-up to the outcomes of the major United Nations conferences and summits in the economic, social and related fields — Follow-up to the outcome of the Millennium Summit — ref. A/68/L.4.

14United Nations — General Assembly — The road to dignity by 2030: ending poverty, transforming all lives, and protecting the planet — Synthesis report of the Secretary-General on the post-2015 sustainable development agenda — ref. A/69/700.

15See also COM(2015) 44 final of 5 February 2015 ’A Global Partnership for Poverty Eradication and Sustainable Development after 2015’.

16Accountability refers mainly to the democratic oversight of policies and activities of public bodies, whereas public audit refers to the financial and performance audits of policies and related public funds, and their link to the accountability process. See European Court of Auditors’ report on ‘Gaps, overlaps and challenges: a landscape review of EU accountability and public audit arrangements’, 2014 (http://eca.europa.eu).

17See also Article 163 of the Financial Regulation.

18Regulation (EU, Euratom) No 966/2012 of the European Parliament and of the Council of 25 October 2012 on the financial rules applicable to the general budget of the Union and repealing Council Regulation (EC, Euratom) No 1605/2002 (OJ L298, 26.10.2012, p. 1).

19Similar provisions are contained in European Development Funds’ financial regulations.

20Regulation (EU) No 236/2014 of the European Parliament and of the Council of 11 March 2014 laying down common rules and procedures for the implementation of the Union’s instruments for financing external action (OJ L 77, 15.3.2014, p. 95).

3. Our review

20

Our review aimed to provide in a single document an overview of what the Court considers to be the main risk areas related to a results approach for EU development and cooperation action. It also illustrates the risks concerned on the basis of experience acquired by the Court through its audits in the field of external actions. It is particularly relevant in the context of current discussions related to the Commission’s performance framework (see paragraph 5) and the implementation of a results framework for EU international cooperation and development action (see paragraph 11).

21

Our review was guided by the following question: did the Commission adequately address the key risks entailed by a results approach for EU development and cooperation measures?

22

We carried out our review work between October 2014 and January 2015. It consisted of a desk review of documentation from EU institutions and other major multilateral and bilateral aid donors, as well as interviews with EU Commission staff, including from the EU delegation in Cambodia, and officials from two international organisations and several implementing bodies. A screening of ECA special reports published in recent years was also carried out in order to illustrate the key risk areas identified with specific examples. The timescale of development actions is such that these examples sometimes relate to actions undertaken several years ago.

4. Key risk areas

23

On the basis of its review work, the Court identified nine key risk areas, which are detailed and illustrated below. They relate to:

(a)the consistent use of results related terminology (section 4.1);

(b)the establishment of a clear results chain (section 4.2);

(c)increased complexity due to the integration of cross-cutting issues (section 4.3);

(d)the harmonisation of instruments and frameworks between development partners (section 4.4);

(e)results reporting and evaluation (section 4.5);

(f)data consolidation and results overview (section 4.6);

(g)data quality (section 4.7);

(h)focus on budgetary outturn (section 4.8); and

(i)changes in the context of EU actions (section 4.9).

4.1. Formulation of objectives and indicators

4.1.1. Terminology

24

Given the number and diversity of parties involved in, or associated with, implementing a new framework for EU development and cooperation, it is crucial for them to be able to employ a common and consistent terminology.

25

However, the Court found that results-related terminology is not always used consistently (see Box 1).

26

A lack of clarity and consistency in terminology employed can cause misunderstanding and confusion among those involved in EU development and cooperation activities, notably in the context of evaluating actions.

Box 1

Examples of terminology issues

While the regulation on common rules and procedures for the implementation of the Union’s instruments for financing external action²¹ provides that results cover outputs, outcomes and impact (see paragraph 19), EuropeAid’s management plan for 2014 uses a different categorisation in which outcomes include results and impact²².

While regulations provide for specific, measurable, achievable, relevant and timed (SMART) objectives, EuropeAid’s Results Oriented Monitoring Handbook, which constitutes a reference document for all EuropeAid’s results-oriented monitoring activities, refers to ‘SMART OVIs’, where ‘OVIs’ stands for objectively verifiable indicators; it thus confuses objectives and indicators; similarly, the Commission’s budget circular for 2014 referred to SMART as the ‘minimum conditions to be fulfilled by objectives and indicators’²³.

While regulations provide for predefined, clear, transparent and — where appropriate — country-specific and measurable indicators, other sets of quality criteria are sometimes used, as EuropeAid pointed out in its 2012 guidelines for budget support²⁴: ‘Different sets of criteria exist to assess indicator quality: SMART indicators are Specific, Measurable, Achievable, Realistic and Time-limited. CREAM criteria refer to indicators that are Clear, Relevant, Economic, Adequate and Monitorable. Eurostat indicator guidance focuses on logic, relevance, the possibility of setting a target, frequency of data collection, appropriateness and possibility of estimating precision. RACER indicators refer to indicators that are Relevant, Accepted, Credible, Easy, and Robust.’

4.1.2. Document alignment and consistency

27

Objectives — together with their indicators — are easier to understand and more likely to be achieved when they are formulated in a clear and consistent way in all interrelated documents.

28

Objectives and indicators for EU development and cooperation are specified in a broad variety of documents. These include:

(a)the Treaty on the Functioning of the European Union;

(b)high-level development goals (see section 2.1.1);

(c)regulations on instruments for financing external action²⁵;

(d)the Commission’s regional and country strategies and multiannual indicative programming documents;

(e)annual action programmes annexed to the Commission’s financing decisions;

(f)financing agreements with partner countries, administration/contribution agreements with international organisations, grant agreements and contracts with non-governmental organisations and service providers;

(g)activity statements (or programme statements) as part of the budgetary procedure;

(h)annual management plans and annual activity reports of the Commission directorates-general concerned.

29

The administration of all these documents, which originate from different authorities and are subject to different adoption cycles and procedures, necessarily leads to adjustments to objectives and indicators and, in some cases, inconsistencies. In this connection, a results reporting pilot exercise carried out by the Commission in 2014 in the preparation of the results framework found that, where logical frameworks existed at both decision and contract levels, ‘the degree of coherence between them was mixed — at best’²⁶.

30

In previous audits, the Court also found inconsistencies between objective and indicator-setting documents (see Box 2).

Box 2

Example of inconsistencies between documents

‘The annual activity reports contain seven indicators for Central Asia. Though of general relevance to the success of EU development assistance in Central Asia, these indicators are different from those set in the Regional Strategy Paper and Multi-annual indicative programmes; nor are they directly linked to the intended results of the Council’s new partnership strategy for Central Asia.’²⁷

31

Inconsistencies in the formulation of objectives and indicators can give rise to confusion about what should serve as a basis for assessing what has been achieved.

4.1.3. Setting SMART objectives and clear indicators

32

Setting specific, measurable, achievable, relevant and time bound (SMART) objectives is a legal requirement for all EU development and cooperation activities. Moreover, the results of EU external action — including EU development and cooperation — and the efficiency of a particular instrument should be monitored and assessed on the basis of predefined, clear, transparent and — where appropriate — country-specific and measurable indicators (see paragraph 19)²⁸.

Figure 2

Need for SMART objectives and clear indicators

33

In previous audits, the Court often found that the objectives set were too vague or unrealistic (see Box 3). Similarly, the Court often found that indicators were missing or unclear (see Box 4). Even when indicators are well defined, their interpretation can be relative and their value can vary considerably according to the context in which they are used (see Box 5).

34

The results reporting pilot exercise carried out by the Commission in 2014 in the preparation of the results framework also found that, in many cases, where the logical frameworks existed in decisions and contracts, they were not of good quality, i.e. objectives were not SMART and indicators were not well defined or were based on unclear data sources.

35

As well as contravening regulations, failing to set SMART objectives and clear indicators undermines performance monitoring and results assessment, which is detrimental to accountability and public audit. The results of an intervention also risk being limited if its objectives are not well defined from the beginning.

Box 3

Example of objectives that were too vague or unrealistic

‘The lack of clear definition of project objectives relates mainly to equipment planned, its level of operation (quantity and quality) and the target population expected to benefit from project results. The lack of clear and quantified measures significantly hampered the Commission’s ability to monitor progress during implementation and later to measure and check whether the results achieved were sustained over time or not (…).’²⁹

‘The objectives are also sometimes overly ambitious, in particular in the case of NGO projects which have modest budgets and a relatively short implementation period. This was particularly true of NGO projects funded under the Food Facility: they had approximately 20 months for implementation, which in practice was reduced to more like 15 months as the first months were given over to procurement and other preparatory activities. In some cases, the interventions are based on unreasonable assumptions concerning the conditions necessary for successful implementation, e.g. the institutional and management capacities of the national or local authorities, the quality of rural infrastructure or the availability of suitable staff (e.g. nutritionists and extension workers).’³⁰

Box 4

Examples of indicators that were missing or unclear

‘No performance indicators were included in the financing agreements for Pegase Direct Financial Support (Pegase DFS), which makes it harder for the EEAS, the Commission and the Member States to assess the concrete results of the support (…). The lack of performance indicators also makes it more difficult to demonstrate results in order to attract new funding. Moreover, this does not comply with the ENPI regulation’s explicit requirement that performance indicators be established for monitoring purposes for all forms of ENPI assistance.’³¹

‘(…) in Kyrgyzstan the food security information programme contained vague indicators, such as ‘food security information is generally accepted to be accurate and timely’, ‘adequate numbers of staff are trained’ and ‘usefulness of crop forecast’.’³²

Box 5

Example of the changing relevance of indicators

The EU provides budget support to Cambodia for the development of its education sector. In recent years, grade 12 school exams in Cambodia were undermined by cheating and corruption. The pass rates were then around 80 %. In 2014, the Cambodian authorities undertook to prevent and eventually eliminate irregularities in these exams, in particular by introducing measures at testing centres to prevent cheating. A first exam organised in August 2014 led to a 25 % pass rate; a second-chance exam organised in October brought the total pass rate for 2014 up to 41 % of candidates, roughly half the 2013 pass rate. At first sight, the change in this indicator could be interpreted as a deterioration; however, it can alternatively be regarded as reflecting an improvement in the reliability and credibility of the exams.

4.1.4. Defining a partner country’s commitment to implement reforms as an objective

36

For an action’s objectives to be significant in terms of development, they should correspond to the achievement of a result, i.e. a measurable change resulting from the action concerned.

37

In previous audits, the Court found that the objectives of EU actions are sometimes formulated in the form of a commitment to be taken by a partner country at a given moment. However, in some of these cases, even if the stated objective had been met, i.e. even if the partner country appeared to have made the expected commitment, the action had then no or little significant effect over time; in other words, it did not produce any significant result in terms of development (see Box 6).

38

The effectiveness of EU development aid and coordination actions can be undermined when the partner country’s apparent commitment to achieving results, without further evidence that significant reforms are actually being implemented, is considered as a sufficient condition for payment.

Box 6

Example of intervention where the partner country’s commitment to implement reforms did not translate into development results

‘The Commission disbursed its budget support in Kyrgyzstan in three tranches, each of which was preceded by an assessment of progress towards public financial management (PFM) objectives. The Commission assessed that progress had been slow, but justified the disbursement of each of the three tranches mainly because it considered that the Kyrgyz government remained committed to further reforms. There was no clear evidence for this opinion. A core part of the Commission’s reasoning for its disbursement decisions is based on its assessment of the partner countries’ commitment to reform rather than on the actual progress achieved’.³³

‘The Commission did not succeed in ensuring that the issues it brought up in the subcommittee dialogue were subsequently followed up by the Egyptian authorities with concrete actions. Even when issues raised were included as action items at the end of the meeting, they were not generally implemented. For example, the 2009 subcommittee included as an action point ‘exploring the possibility to use EU technical assistance to advise Egypt on NGO legislation in Member States’. This was not followed up. (…) The Commission made no link between its criticisms of human rights violations made in the progress reports and the option of reducing or suspending EU assistance.’³⁴

4.2. Establishment of the results chain

4.2.1. Linking actions and results through a clear logical chain

39

In order to obtain the expected results, it is necessary to identify appropriate actions and thus to establish a clear logical link between activities, outputs, outcome and impact35,36. Establishing such a logical chain is also essential to demonstrate how EU development and cooperation interventions contribute to results³⁷.

40

Some previous Court special reports and Commission evaluation reports found that the link between action and results was not always clearly established (see Box 7).

41

For some actions, results-based payment conditions can be set in financing agreements linking payment directly to the achievement of results in order to create incentives for improved performance through full or partial payment for full or partial achievement of results38,39. In previous audits, the Court sometimes found that such a link was not made or enforced (see Box 8).

42

In the absence of a clear link between actions — or conditions for payment — and their intended results, the interventions’ objectives can be lost sight of, which is detrimental to aid effectiveness; the EU’s capacity to demonstrate and account for the results of its actions is also undermined. Similarly, seemingly separate policies can in reality be closely linked⁴⁰. When designing an action in the context of a given policy, its possible results in terms of other policies may be given insufficient consideration; for example, a project could contribute to health-related objectives, such as combating malaria, while its stated objectives only concern improving access to safe drinking water.

Box 7

Examples of an unclear link between action and results

‘In the 24 interventions examined, the ENP strategy is clearly referenced as the umbrella under which the cooperation takes place. Reference to one or several ENP objectives is clearly made in the programming documents and the interventions are in line with the strategy documents. However, the documentation reviewed does not show nor explain in an explicit manner how the objectives will be achieved through the specific interventions, and how the interventions programmed will contribute to achieve the results.’⁴¹

‘Budget support programmes in Malawi and Rwanda make reference to the objectives of the national poverty reduction strategies and sector policies and programmes. They do not state how they aim to contribute to the governments’ objectives.’⁴²

Box 8

Examples of financial interventions with ineffective payment conditions

‘One objective of the two targeted budget support programmes was to contribute to improving public financial management (PFM). However, the programmes did not establish any links with the government’s PFM reform plan, nor did they identify which priorities they were intended to support. It is unclear how programmes of this sort can achieve such an objective, given that funds were disbursed in a single fixed tranche, with no performance-related conditions or policy dialogue requirements.’⁴³

‘For road infrastructure investment projects, the Commission uses conditions in a way that gives partner countries little incentive to comply with them. The conditions set are not legally binding but presented in the financing agreements as ‘accompanying measures’ to be taken up by the partner country. In practice, the Commission generally endorses works contracts between partner countries and contractors regardless of whether these accompanying measures have been taken. It is then under an obligation to pay EDF money under these contracts, and it has little leverage to compel the partner country to comply with the conditions.’⁴⁴

4.2.2. Contribution versus attribution

43

EU development and cooperation interventions take place in a complex realm, where many other donors and many exogenous factors, besides EU intervention, contribute — positively or negatively — to the achievement of results. For this reason, as well as establishing a causal link between EU development and cooperation action and its results (see section 4.2.1.), it could be desirable to assess the strength of this link, i.e. which part of the results can be attributed to EU action.

44

However, as the Commission pointed out in its Staff Working Document on ‘Paving the way for an EU development and Cooperation Results Framework’ (see paragraph 7), ‘it is difficult to measure contribution of one level of the results hierarchy to the next’, i.e. from output to outcome, and from outcome to impact. Indeed, while an intervention’s outputs can usually be largely attributed to the intervention, this will often not be the case for its outcome, and even less for its impact, as the Court found in some of its audits (see Box 9).

45

In the particular case of budget support interventions, it is not normally possible to identify the outputs, due to the fungibility of EU financial support within the partner country’s budget. Indeed, the partner country’s budget supports the whole country’s development strategy, and the outputs of EU budget support interventions thus relate to changes of government policies and services, which are not the consequence of the budget support programme alone but also of various other government actions and decisions, other external assistance programmes (…) and external non-government-related factors⁴⁵. This is well illustrated in the OECD’s comprehensive evaluation framework for budget support evaluations (see Annex).

46

Consequently, even where a clear logical link between action and results can be established, it can be difficult — if not impossible — to assess which part of the results is attributable to EU interventions and which is due to other donors’ interventions or other factors. For this reason, it is more relevant to speak in terms of contribution rather than attribution. Besides its technical aspects, this issue also relates to the more political question of results ownership. Indeed, when many stakeholders are involved, including the EU, partner countries and other donors, there can be several claims for the same positive results, or no acknowledgement of responsibility for negative results.

Box 9

Example of difficulties in attributing results to EU intervention

‘The Court cannot assess the extent to which the EU budget support programmes in Malawi and Rwanda have contributed to the results of the governments’ programmes since there are multiple and complex intervening factors (…).’⁴⁶

4.3. Increased complexity due to the integration of cross-cutting issues

47

In its communication on ‘Increasing the impact of EU Development Policy: an Agenda for Change’ (see paragraph 18), the Commission proposed that EU activities in each partner country should be concentrated on a maximum of three sectors. Focusing EU interventions on a small number of sectors can increase their impact and leverage and help to keep EU assistance manageable not only for partner countries, but also for the Commission and EU Delegations⁴⁷.

48

Meanwhile, according to EU instructions for establishing multiannual programming documents for the 2014-2020 programming period, sector policy commitments must also take into account nine cross-cutting issues, including the promotion of human rights, gender equality, democracy, good governance, children’s rights, persons with disabilities, indigenous peoples, environmental sustainability and combating HIV/AIDS⁴⁸ (see Box 10).

Figure 3

Focal sectors and cross-cutting issues

49

Previous Commission evaluation reports pointed out difficulties in mainstreaming human rights through EU development actions (see Box 11).

50

The mainstreaming of cross-cutting issues in EU interventions can make the prioritisation of these interventions’ objectives more complex. The increased complexity due to the integration of cross-cutting issues in EU programmes represents a supplementary risk to the achievement and reporting of results.

Box 10

Integration of cross-cutting issues in a programme — the example of Cambodia

The Commission’s Multiannual Indicative Programme for Cambodia for the 2014-2020 period provides for three focal sectors: Agriculture/Natural Resource Management, Education/Skills and Good Governance. It also provides for the mainstreaming of several cross-cutting issues: gender equality and empowerment of women, nutrition, the environment, and information and communication technologies.

Box 11

Examples of difficulties in addressing cross-cutting issues — the case of human rights

‘The EU has sought to further translate its commitment and mainstream human rights at programming level, including regional strategy papers, bilateral action plans, and thematic programmes. Human rights and democracy are entrenched in the policy documents as guiding principles. Nevertheless, one can observe a phenomenon of ‘dilution’ of the democracy and human rights component as the cycle moves ‘downstream’, i.e. from broad political pledges to policy frameworks down to programming, choice of instruments, allocation of funding and selection of projects. This ‘dilution’ appears under various forms including: (i) the lack of coherent strategy to address problems through concrete interventions; (ii) the limited funding for related democratic and human rights aspects in other thematic important sectors; (iii) the tendency to confine human rights work to thematic instruments.’⁴⁹

‘The desk analysis of six concrete areas where European Commission (EC) sought to mainstream human rights shows the phenomenon of ‘dilution’ of the human rights component as the cycle moves downstream (…). All this is compounded by a lack of even basic systems to monitor and evaluate progress in relation to mainstreaming. This dilution is related to political resistance of partner countries but also to major internal weaknesses at EU level, such as limited political leverage and inconsistent decision-making with regard to the place and weight of human rights in EU external action. The EU’s difficulties to reconcile values and interests have been widely documented and criticised by media and civil society. The Arab Spring was an eye-opener on the limits of the ‘stability versus human rights approach’ and may open perspectives for a more serious approach to mainstreaming human rights.’⁵⁰

4.4. Harmonisation between development partners’ instruments and frameworks

4.4.1. Alignment with partner countries’ results frameworks

51

In line with the principles agreed at high-level international forums, donors, including the EU, should rely as far as possible on partner countries’ frameworks for results-oriented reporting and monitoring (see section 2.1.2). In particular, donors should minimise their use of additional frameworks, refraining from requesting the introduction of performance indicators that are not consistent with countries’ national development strategies⁵¹.

52

However, previous Commission evaluation reports found issues related to non-alignment between donors and partner countries’ frameworks (see Box 12).

53

When partner countries face many different types and levels of reporting obligations and payment conditions from various donors, these can present an unmanageable burden, which is detrimental to both aid efficiency and effectiveness.

Box 12

Example of the lack of alignment with the partner country’s results framework

‘(…) far from being a Government-led, genuinely harmonised framework, the Performance Assessment Framework (PAF) had developed into ‘the sum total of all Development Partners’ individual preferences and requirements’. (…) The effective ownership of Government was lost and the PAF assessment process became little more than a compulsory process, which needed to be respected in order for Budget Support disbursements to continue (…). Simultaneous with this loss of ownership and engagement, there has been a significant increase in the transaction costs associated with the annual assessment process and the related policy dialogue. This is a perception reported unanimously by all relevant stakeholders interviewed. It is manifested most obviously in the steady growth in the number of indicators incorporated within the PAF.’⁵²

4.4.2. Harmonising the instruments used by development stakeholders

54

In previous Court special reports, issues were found related to the lack of harmonisation of instruments used by development partners (see Box 13).

55

The failure to harmonise instruments used by development partners can be detrimental to results monitoring and reporting effectiveness and efficiency.

Box 13

Examples of issues due to the lack of harmonisation of instruments used by development partners

‘The United Nations Development Programme (UNDP) has estimated that there are more than 50 international public funds, 45 carbon markets and 6 000 private equity funds providing climate change finance, each with its own governance structure. As the World Bank notes, fragmentation of this sort threatens to reduce the overall effectiveness of climate finance (…). The Commission and Member States use both bilateral and multilateral channels to disburse climate finance. In 2010 they used no less than 22 multilateral channels.

(…) No attempt has been made to reduce the proliferation of climate funds, which involves serious risks of inefficiencies, inadequate accountability and fragmentation of aid (…)’⁵³

4.4.3. Coping with the juxtaposition of different accountability structures

56

As part of the EU’s development and cooperation action, the European Commission entrusts some budget implementation tasks under indirect management to international organisations⁵⁴. For these tasks, accountability relies on the accountability structures of the international organisation concerned, including governance, results frameworks⁵⁵ and external audit, as well as on the EU’s own audit and accountability provisions.

57

In previous audits, the Court found that the juxtaposition of different accountability structures for tasks under indirect management with international organisations sometimes created accountability gaps (see Box 14).

Box 14

Example of distinct accountability architecture — United Nations

‘Although the Commission verifies the financial management systems of its UN partners (using a four-pillar analysis), the Commission depends on UN reports for confirming the practical operation of control systems and the achievement of results. At the time of the audit the Commission had not yet succeeded in obtaining adequate information from the UN in this regard.’⁵⁶

‘The Commission does not receive sufficient timely information from UN reports. A large proportion of reports are still delayed, not detailed enough and focus on activities rather than results.’⁵⁷

4.5. Weaknesses in evaluation and results reporting

58

Legal provisions for results monitoring, evaluation and audit provide for two different types of timing:

(a)periodic timing: for example, Article 318 TFEU lays down that the Commission must submit annually (…) an evaluation report on the Union’s finances based on the results achieved, in particular in relation to the indications given by the European Parliament and the Council in the context of the annual discharge procedure;

(b)action-based timing: for example, Article 18.3 of the rules of application of the Financial Regulation provides that results of multiannual programmes where the resources exceed 5 000 000 euros must be periodically evaluated in accordance with a timetable which enables the findings of that evaluation to be taken into account for any decision on the renewal, modification or suspension of the programme.

59

In previous audit reports, the Court found that, in some cases, the achievement of projects or programme objectives could not be assessed before they finished, which undermined periodic assessment of the achievement of results (see Box 15).

60

The Court also found weaknesses in the timely delivery of evaluations, i.e. strategic evaluations often took much longer than planned and programme evaluations were sometimes postponed or cancelled⁵⁸. A recent study requested by the Commission also found ‘important disconnects’ between evaluations and policy formulation, programming, monitoring and result-oriented management. In particular, it found missed opportunities — in terms of uptake — where previously learned lessons were ignored because evaluation findings did not synchronise with the management decision-making process, (…) or because the context had changed by the time the findings were made available⁵⁹.

61

Weaknesses in results reporting and evaluation timing can prevent the feedback of relevant knowledge and lessons-learnt into reporting and decision-making processes.

Figure 4

The time cycle of an action

Box 15

Example of results which could only be assessed at the end of the action concerned

‘The Court assessed the effectiveness of a sample of 19 projects by comparing the intended objectives in the contribution agreement with the actual achievements. (…) Due to insufficient information, the results could not be assessed in the following cases: (…) One major project was a trust fund for which the objectives stated in the contribution agreement concerned the fund as a whole and were therefore of a more general nature. Although more specific objectives were defined for the underlying projects, there can be no clear link between the achievement of the project objectives and the general objectives in the contribution agreement. Consequently the effectiveness can only be assessed when the fund itself will be closed, which was not yet the case at the time of the audit.’⁶⁰

4.6. Data consolidation and reporting

62

In order to demonstrate the results achieved by EU development policies, data on results achieved by EU development and cooperation actions needs to be collected and consolidated through information systems and presented and disseminated through appropriate reporting channels.

63

However, in previous audit reports, the Court found issues with the capacity of EuropeAid’s information systems to collect results-related data (see Box 16). In 2014, the Commission also pointed out that, for the EU to be able to report results, data collection systems and tools would need to be further rationalised and developed⁶¹.

64

Data on EU development aid and cooperation activities serve as a basis for a broad variety of reports, the main ones being:

(a)the Commission’s annual accounts and report on budgetary and financial management⁶²;

(b)EuropeAid’s Annual Activity Report and the Commission’s synthesis report⁶³;

(c)annual reporting to the OECD’s Development Assistance Committee (DAC), in line with DAC’s directives⁶⁴;

(d)a monthly data contribution to the IATI Data Registry, in line with IATI standards⁶⁵;

(e)the annual report on the evaluation of the Union’s finances based on the results achieved⁶⁶;

(f)the annual report on the European Union’s development and external assistance policies and implementation thereof;

(g)regular updates to the Commission’s ‘EU Aid Explorer’ website⁶⁷.

65

Currently most of these reports focus on activities, while none of them provides consolidated data on the results achieved by these activities across the board⁶⁸. A lack of consolidated reports on and overview of the results achieved by EU aid can impair decision-making and accountability.

Box 16

Example of weaknesses in EuropeAid’s information system

‘The main reporting tool used by the Commission between delegations and EuropeAid is the External Assistance Monitoring Report (EAMR) (…). However, these reports largely describe activities carried out and implementation problems encountered. Moreover, information provided to a certain extent overlaps with information available in CRIS. They provide little indication on the actual results of the aid, either in terms of assessments made by the delegation itself or by reporting on the results of evaluations made after aid interventions have closed. (…)’⁶⁹

‘(…) CRIS cannot easily be used to compute the total amount spent on a given policy or financed from a given financial instrument. Indeed, the list of domains used to associate records from many CRIS modules to a given domain mixes geographical zones (Asia, for example), financial instruments (…) and thematic policies (food security, for example).

As a result of this situation, the consolidation of data — notably financial data — contained in CRIS for reporting purposes is rendered particularly complex. This situation is detrimental to CRIS’s efficiency and effectiveness as a reporting and management tool.’⁷⁰

4.7. Data quality

66

Results monitoring and assessment requires the collection of sufficient, relevant, reliable and up-to-date data. In line with commitments made in high-level forums on aid effectiveness (see section 2.1.2), data collection should rely as much as possible on partner countries’ own frameworks for results-oriented reporting and monitoring.

67

However, previous Court audits and Commission evaluations sometimes found that data sources did not provide sufficient, relevant, reliable and timely information on results (see Box 17). Data collection is sometimes costly — notably in the case of censuses — and thus not frequently carried out by partner countries.

68

A lack of sufficient, relevant, reliable and up-to-date data can significantly weaken the quality and reliability of reported results.

Box 17

Examples of data-quality weaknesses

‘Generally the indicators used by the Commission have an appropriate focus on the [Millennium Development Goals], although insufficient attention is paid to indicators for education quality. However, the national education management information systems that the Commission relies on do not consistently provide sufficient, reliable and timely information.’⁷¹

4.8. Budgetary outturn

69

In previous years, there has been pressure to increase the amounts devoted to EU development and cooperation, particularly as part of the context of the commitment to attain the internationally agreed ODA target of an ODA/GNI ratio of 0.7 % by 2015⁷².

70

Article 317 TFEU stipulates that the Commission must implement the budget in accordance with the financial regulations and taking into account the principles of sound financial management. Article 38(3) of the Financial Regulation stipulates that the Commission must attach to the draft annual budget, among other things, information on the implementation rates for the preceding year’s activity and implementation rates for the current year.

71

Budgetary outturn can be perceived as an objective in itself and is sometimes stated as such in Commission management plans (see Box 18).

72

For this reason, there can be tension between the need to implement the budget and the need to implement it in accordance with the principle of sound financial management (effectiveness, efficiency and economy)⁷³. In particular, achievements and failures in terms of budgetary implementation are more directly visible than achievements in terms of performance (results). For this reason, there can be a natural tendency to focus on budgetary implementation to the detriment of focusing on results, as the Court found in its annual report on the implementation of the budget in 2013: ‘When spending EU funds in the 2007-2013 programming period, the focus was on absorption (the need to spend money) and compliance rather than good performance’⁷⁴.

73

Focusing on budgetary outturn as an objective can be detrimental to sound financial management and the achievement of results.

Box 18

Example of a budgetary execution objective

EuropeAid’s management plan for 2014 sets the following objective for the European Development Fund (EDF) in Africa⁷⁵: ‘Levels of budgetary execution — Maintain high levels of budgetary execution including budget support payments’.

EuropeAid’s management plan also contains, for external financing instruments other than EDF, an annex entitled ‘Sound financial management and effective use of EC resources’, which sets several indicators and targets for budgetary execution:

| Indicator | Target (2014)
1 | Execution of initial annual financial forecast: Payments | From 90 % to 110 %
2 | Execution of annual financial forecast for contracts | From 90 % to 110 %
3 | Execution of annual financial forecast for decisions | From 90 % to 110 %

4.9. Changes in EU actions’ context

4.9.1. Risk management and choice of implementation modalities

74

EU development and cooperation actions are implemented in a large number of countries with different levels of development, many of which are characterised by critical social, political or humanitarian conditions or even volatile security situations. The choice of an aid implementation modality to achieve an action’s objectives should be based on its expected capacity to deliver results in an effective and efficient way given the context of the action. In some cases, the choice of an implementation modality that may appear riskier than others may sometimes be motivated by the expectation that it will deliver better or quicker results.

75

Budget support illustrates this situation well. Indeed, while it can be conceived as a means of supplying large volumes of aid in a more predictable manner than the traditional project approach, budget support is often provided to countries with weak public financial management (PFM) systems. Given that the funds transferred under budget support operations are merged with other budget resources within the country’s budget (known as ‘fungibility’), they are also exposed to the same PFM weaknesses, including risks of fraud and corruption⁷⁶. In addition, results achieved through budget support and the sustainability of such results can be put at risk in the event of political instability in the partner country, for example due to a lack of consideration for good governance, major economic and social issues or human rights. The decision to provide EU budget support should thus be taken on a case-by-case basis and supported by an assessment of the expected benefits and potential risks⁷⁷.

76

In previous audit reports, the Court sometimes found that the choice of an aid implementation modality had not been preceded by a suitable analysis of the risks entailed with regard to the expected advantages (see Box 19).

77

Failing to compare the expected benefits with the potential disadvantages of the selected implementation modalities can lead to a wrong choice of modality or expose the EU budget to unreasonably high risks.

Box 19

Examples of a lack of risk analysis for the chosen implementation modality

‘(…) the Commission does not convincingly demonstrate, before deciding to work with a UN organisation, that it has assessed whether the advantages offset any disadvantages. The choice of a UN organisation is not based on sufficient evidence that this approach is more efficient and effective than other ways of delivering aid.’⁷⁸

4.9.2. Exogenous factors

78

Most EU development and cooperation actions are planned with a view to producing results over a period of several years. During this time, actions can experience significant changes or instability in:

(a)the political or economic context at partner-country level — this was the case, for example, in Egypt⁷⁹, the Democratic Republic of the Congo⁸⁰ and Palestine⁸¹ — or at project level (see Box 20);

(b)the development context in respect of natural disasters, e.g. in Haiti, in January 2010, when the capital and the surrounding area were hit by an earthquake of magnitude 7.0 on the Richter scale⁸²; or

(c)EU aid management and control systems (see Box 21).

79

Such changes in the context of EU development and cooperation actions can make it difficult to track and report on results in a consistent and meaningful way.

Box 20

Example of instability in the economic context of a project

In Cambodia, the Commission financed a project to promote Waste to Energy technology in the rice milling sector. However, in recent years, Cambodia has seen major investments in larger and more efficient rice mills that have led to a four-fold capacity increase between 2009 and 2013⁸³. It is now expected that most rice millers require more powerful systems than those initially planned for promotion by the project, at much higher costs. For this reason, the project’s initial objectives have lost some of their relevance and are likely not to be achieved as initially planned.

Box 21

Example of changes in management and control systems

On 1 January 2015, the Commission’s ‘Support Group for Ukraine’ and ‘Neighbourhood’ Directorate were transferred from EuropeAid to the Directorate-General for Neighbourhood and Enlargement Negotiations, DG Neighbourhood and Enlargement Negotiations (formerly the Directorate-General for Enlargement, DG Enlargement)⁸⁴. This will affect both the geographical coverage of EuropeAid’s results framework and the way in which neighbourhood policy results are monitored and reported.

21Regulation (EU) No 236/2014.

22See, for example, the DCI (Development Cooperation Instrument) intervention logic in section 4.1.4 of EuropeAid’s management plan for 2014.

23European Commission — Budget circular for 2014 — Standing instructions, 14.12.2012.

24Budget Support Guidelines — Programming, Design and Management — A modern approach to Budget Support, September 2012.

25OJ L 77 2014, 15.3.2014.

26A logical framework is a document that provides an overview of a project’s objectives, activities and resources. It also usually outlines critical assumptions on which the project rests and provides information on how the project will be monitored, including through the use of indicators.

27Special Report No 13/2013 ‘EU development assistance to Central Asia’, paragraph 79 (http://eca.europa.eu).

28In April 2014, the European Parliament urged the adoption of specific, measurable, achievable, relevant and timed benchmarks for all programmes in Heading 4 — Global Europe — of the multiannual financial framework 2014-2020. See the European Parliament Decision of 3 April 2014 on discharge in respect of the implementation of the general budget of the European Union for the financial year 2012, Section III — Commission and executive agencies — ref. P7_TA-PROV(2014)0287.

29Special Report No 13/2012 ‘European Union development assistance for drinking water supply and basic sanitation in sub-Saharan countries’, paragraph 50 (http://eca.europa.eu).

30Special Report No 1/2012 ‘Effectiveness of European Union development aid for food security in sub-Saharan Africa’, paragraph 52 (http://eca.europa.eu).

31Special Report No 14/2013 ‘European Union direct financial support to the Palestinian Authority’, paragraph 31 (http://eca.europa.eu).

32Special Report No 13/2013, paragraph 52.

33Special Report No 13/2013, paragraph 57.

34Special Report No 4/2013‘ EU Cooperation with Egypt in the Field of Governance’, paragraphs 39 and 41 (http://eca.europa.eu).

35Study on Legal Instruments and Lessons Learned from the Evaluations Managed by the Joint Evaluation Unit — Final Report Volume 1: Final Summary Report, July 2011, p. vi: ’Formulating clear logic chains in regulations is key. This can be encouraged by avoiding unstructured lists of activities and unclear links between activities, outputs and their intended impacts. The relationships between each step in the chain should be clear and logical so as to avoid gaps in the logic and ‘orphan’ outputs or impacts.‘ — ref. evaluation DEVCO 1292.

36A detailed explanation of a results chain is provided in United Nations Development Group’s results-based management handbook, October 2011, p. 14.

37Study on the uptake of learning from EuropeAid’s strategic evaluations into development policy and practice, Final report, June 2014, p. vii.

38Commission Budget Support Guidelines — Executive Guide — A modern approach to Budget support, September 2012.

39See EU Common Position for the Fourth High Level Forum on Aid Effectiveness (Busan, 29 November — 1 December 2011) — Council conclusions — 3124th Foreign Affairs Development Council meeting, Brussels, 14 November 2011: ‘The EU calls on partner countries and development partners to increase the emphasis on harmonised and results-based conditionality’.

40Thematic evaluation of the European Commission support to the health sector — Final report — Volume 1, August 2012, p. xiv.

41Evaluation of the European Union’s Support to two European Neighbourhood Policy Regions (East and South) — Final report — Volume 1, June 2013, p. 24.

42Special Report No 1/2012, paragraph 53.

43Special Report No 9/2013 ‘EU support for governance in the Democratic Republic of the Congo’, paragraph 65 (http://eca.europa.eu).

44Special Report No 17/2012 ‘The European Development Fund (EDF) contribution to a sustainable road network in sub-Saharan Africa’, paragraph 27 (http://eca.europa.eu).

45OECD — DAC Network on Development Evaluation — Evaluating Budget Support Methodological Approach, page 11. See also page 12, on budget support outcomes: ‘Outcomes are the result of the entire set of policies, strategies and spending actions of the government (…), of the reactions/responses of the relevant stakeholders to changes in public policy making and resource allocation decisions, and of the influence of other external factors. These outcomes are thus only partly influenced by the BS provided. The causal relationship between the provided budget support and the outcomes will therefore be (strongly) diluted with other influencing factors.’

46Special Report No 1/2012, paragraph 57.

47EEAS and Commission joint ‘Instructions for the programming of the 11th European Development Fund (EDF) and the Development Cooperation Instrument (DCI) — 2014-2020’.

48See also, for example, the Development and Cooperation Instrument (DCI) Regulation (EU) No 233/2014 which lays down that: ‘Throughout all programmes, cross-cutting issues as defined in the European Consensus shall be mainstreamed. In addition, conflict prevention, decent work and climate change shall be mainstreamed, where relevant’ (Article 3).

49‘Evaluation of the European Union’s Support to two European Neighbourhood Policy Regions (East and South) — Final report — Volume 1, June 2013’, p. 62.

50‘Thematic evaluation of the European Commission support to respect of Human Rights and Fundamental Freedoms (including solidarity with victims of repression)’ — Final Report — Volume 1, December 2011, p. 15.

51Busan Partnership for effective development cooperation.

52Joint Evaluation of Budget Support to Tanzania: lessons learned and recommendations for the future Final Report: Volume 1, 2013, paragraphs 74-75.

53Special Report No 17/2013 ‘EU climate finance in the context of external aid’, paragraphs 57 to 58 and paragraph 68 (http://eca.europa.eu).

54Financial Regulation, Article 58 — methods of implementation of the budget.

55See, for example, United Nations Development Group’s results-based management handbook, United Nations Population Fund’s integrated results framework, or United Nations Development Programme’s handbook on planning, monitoring and evaluating for development results.

56Landscape review 2014 ‘Gaps, overlaps and challenges: a landscape review of accountability and public audit arrangements’, Box 1 (http://eca.europa.eu).

57Special Report No 3/2011 ‘The efficiency and effectiveness of EU contributions channelled through United Nations organisations in conflict-affected countries’, paragraph IV (http://eca.europa.eu).

58Special Report No 18/2014 ‘EuropeAid’s evaluation and results-oriented monitoring systems’, paragraphs 27 to 29 (http://eca.europa.eu).

59Study on the uptake of learning from EuropeAid’s strategic evaluations into development policy and practice — Final report, June 2014, p. vii.

60Special Report No 3/2011, paragraphs 25 and 27 (b).

61SWD(2013) 530 final, p. 12.

62Financial Regulation, in particular Articles 141 to 142.

63Financial Regulation, in particular Article 66.9.

64‘Converged statistical reporting directives for the creditor reporting system (CRS) and the annual DAC questionnaire’ and addendums — reference DCD/DAC(2013)15/FINAL, 11.6.2013.

65Internet site of the International Aid Transparency Initiative (www.iatiregistry.org).

66Treaty on the Functioning of the European Union, Article 318.

67http://tr-aid.jrc.ec.europa.eu

68In particular, the annual report on the evaluation of the Union’s finances based on the results achieved does not provide a clear view of the extent to which the Union’s main objectives have been achieved. See Report on the evaluation of the Union’s finances based on the results achieved: a new tool for the European Commission’s improved discharge procedure — A7-0068/2014, 30.1.2014.

69Special Report No 1/2011 ‘Has the devolution of the Commission’s management of external assistance from its headquarters to its delegations led to improved aid delivery?’, paragraph 59 (http://eca.europa.eu).

70Special Report No 5/2012 ‘The Common External Relations Information System (CRIS)’, paragraphs 40 and 41 (http://eca.europa.eu).

71Special Report No 12/2010 ‘EU development assistance for basic education in sub-Saharan Africa and south Asia’, paragraph V (http://eca.europa.eu).

72‘Member States undertake to achieve the target of an ODA/GNI ratio of 0.7 % by 2015, while those which have achieved that target commit themselves to remaining above that target; Member States which joined the EU after 2002 will endeavour to increase their ODA/GNI ratio to 0.33 % by 2015’ — see Brussels European Council 16 and 17 June 2005, Presidency conclusions — reference 10255/1/05 REV 1, paragraph 27.

73‘One perennial question (…) is how to reconcile the long-term nature of development cooperation, calling for multi-year planning horizons, with the normal practice of aid appropriations lapsing each year. In many DAC member countries, general government procedures require that funds appropriated in a given fiscal year be disbursed within that year or, as a minimum, committed within that year and spent soon afterwards. As a consequence, aid managers in some countries operate under considerable pressure to commit and disburse funds rapidly, promoting undue emphasis on the financial inputs of development activities, rather than desired outcomes and actual results.’ — See OECD — Development Assistance Committee — Managing aid: practices of DAC Member countries — reference DCD/DAC(2004)40.

74European Court of Auditors’ Annual report concerning the financial year 2013, paragraph 10.56 (OJ C 398, 12.11.2014).

75Management Plan 2014 — Directorate-General for Development and Cooperation — EuropeAid.

76European Court of Auditors’ Annual report concerning the financial year 2011, paragraph 7.10 (OJ C 344, 12.11.2012).

77COM(2011) 638 final of 13 October 2011 ’The Future Approach to EU Budget Support to Third Countries‘.

78Special Report No 15/2009, paragraph III.

79Special Report No 4/2013.

80Special Report No 9/2013.

81Special Report No 14/2013.

82Special Report No 13/2014 ‘EU support for rehabilitation following the earthquake in Haiti’ (http://eca.europa.eu).

83World Bank policy note on Trade Development — Turning Cambodia into a leading rice exporter, September 2013.

84Minutes of the 2104th meeting of the Commission held on Wednesday 5.11.2014 — PV(2014) 2104.

5. Conclusions and recommendations

5.1. Conclusions

80

On the basis of our review, we identified nine key risk areas related to a results-oriented approach for EU development and cooperation action.

(a)The use of inconsistent results-related terminology or failure to establish a clear logical chain between action, outputs, outcome and impact can be a source of confusion and undermine the assessment of what has been achieved (sections 4.1 and 4.2).

(b)Increased complexity due to the integration of cross-cutting issues in EU programmes represents a supplementary risk for the achievement and reporting of results (section 4.3).

(c)The lack of harmonisation between development partners’ aid delivery instruments, results frameworks and accountability structures can generate inefficiencies and accountability gaps (section 4.4).

(d)Weaknesses in evaluation and results reporting can prevent the feedback of relevant knowledge and lessons-learnt into reporting and decision-making processes (section 4.5).

(e)A lack of consolidated reports on or an overview of results achieved by EU aid can impair decision making and accountability (section 4.6).

(f)A lack of sufficient, relevant, reliable and up-to-date data can significantly weaken the quality and reliability of reported results (section 4.7).

(g)Focusing on budgetary outturn as an objective can be detrimental to sound financial management and the achievement of results (section 4.8).

(h)Changes in the context of actions can undermine the achievement or sustainability of results, and make it difficult to assess the results achieved (section 4.9).

81

Our review showed that these risk areas had been correctly identified by the Commission in a wide range of documents. It also showed the interest for the actions that are being undertaken by the Commission to launch an EU development and cooperation results framework.

82

Nevertheless, further actions remain to be taken in order to harness the full potential of the Commission’s initiatives for improved EU development and cooperation results.

5.2. Recommendations

83

In order to mitigate the risks identified, we formulate the following recommendations.

Recommendation 1
Improve the guidelines for terminology usage and formulation of objectives and indicators

The Commission should improve its guidelines to ensure the consistent use of terminology regarding results (outputs, outcomes, impacts) and, where relevant, to formulate SMART objectives and clear indicators for its interventions at all levels.

Recommendation 2
Establish a clear link between actions and expected results

Through its internal guidelines, practical tools, project formulation and monitoring processes, the Commission should ensure that a clear link is established between actions and expected results, including cross-cutting issues.

Recommendation 3
Improve the information system for reporting results and lessons-learnt

The Commission should improve its information system in order to be able to effectively consolidate and report on the results of its interventions. This should be complemented by measures enabling the Commission’s evaluation activities to identify useful lessons for subsequent legislation, programming or action design in the areas concerned.

Recommendation 4
Ensure data availability and quality

In all its interventions, the Commission should assess data availability and quality, and strive to have sufficient, relevant, reliable and up-to-date data on results achieved. This should be reflected in its systems and guidelines.

Recommendation 5
Assess the risks inherent to the implementation modality before committing financial resources

Before committing its financial resources, and when considering the expected results, the Commission should also assess the risks inherent to its choice of a particular implementation modality.

This report was adopted by Chamber III, headed by Mr Karel PINXTEN, Member of the Court of Auditors, in Luxembourg at its meeting of 17 November 2015.

For the Court of Auditors

Vítor Manuel da SILVA CALDEIRA
President

Annex

Annex

Comprehensive Evaluation Framework for Budget Support Evaluations

© OECD — DAC Network on Development Evaluation — Evaluating Budget Support Methodological Approach.

Reply of the Commission

Executive summary

III

The nature of the Court’s review is such that it includes examples which sometimes relate to actions undertaken several years ago. This is in particular true in relation to the examples relating to budget support, for which a number of risks have been addressed through the adoption of revised budget support guidelines in 2012 (see the replies provided to various of the boxes hereafter), as well as for the examples relating to multiannual programming documents (strategy papers or indicative programmes), for which the Commission would like to point out that it has systematically included indicators in the programming documents for the period 2014-2020.

1. Introduction

07

The Results Framework (RF) is one of the tools for reporting results at output and direct outcome levels. Evaluations provide judgements on the causal, logical chain between the different level of results up to the impact and the achievement of ‘strategic development objectives’ be it at country or at global level. Furthermore, several initiatives have been taken to improve the whole Monitoring and Evaluation (M&E) system to improve the Commission's results-oriented approach.

10

The Staff Working Document (SWD) on the ‘MORE framework’ describes the various existing frameworks for monitoring, evaluation and reporting on the implementation of the multiannual financial framework (MFF). In terms of objectives and indicators, it describes what has been set up in the legal acts establishing 2014-2020 financing instruments and complemented under the 2014 and 2015 budgetary programme statements as well as the monitoring, evaluation and reporting tools as included in the Annual Management Plans (AMP) and Annual Activity Reports (AAR). For external assistance instruments, it also refers to the new Results Framework as one of the tools for monitoring and reporting on the implementation of the MFF.

3. Our review

22

The nature of the Court’s review is such that it includes examples which sometimes relate to actions undertaken several years ago. This is in particular true in relation to the examples relating to multiannual programming documents (strategy papers or indicative programmes), for which the Commission would like to point out that it has systematically included indicators in the programming documents for the period 2014-2020, as well as in relation to the examples relating to budget support, for which a number of risks have been addressed through the adoption of revised budget support guidelines in 2012.

The Commission also refers to its replies to the Court’s special reports in question and to the additional replies to the Court’s observations in the present report with respect to some of the examples so as to underline one or more aspects it considers more specifically relevant in the context of its reply to those observations.

4. Key risk areas

26

The Commission is aware and agrees with the Court on the need to ensure consistency in terms of the terminology used. In the Staff Working Document (SWD) of 26 March 2015 ‘Launching the EU International Cooperation and Development Results Framework’, the terminology input-output-outcome-impact is used, corresponding to the internationally practised OECD-DAC results terminology and being also in line with the one used in the 2014-2020 financing instruments. The same terminology is used in the new Action Document template (applicable as of 1st January 2015) and in the new Results Oriented Monitoring (ROM) Handbook (‘Handbook on ROM reviews and support to end-of-project results reporting’) (March 2015).

Box 1

The terminology used in EuropeAid’s Annual Management Plan is following the one foreseen in the Commission’s Secretariat-General and DG Budget overall instructions for the establishment of the management plans by the Commission’s general directorates and services.

The risk as set out in the example given in Box 1 refers to the terminology used in the Results Oriented Monitoring (ROM) Handbook applicable to the previous ROM system which ended in spring 2014. It has been replaced by a new ROM system and a new ROM Handbook introduced in spring 2015, which no longer contains such reference and is fully in line with internationally practiced language applying the reference to SMART objectives and not to indicators. Accordingly, the Commission will modify the next budget circular.

Moreover, the Commission intends to refine the language and instructions related to the objectives and indicators to be used in the context of the budgetary circular.

The Commission agrees that objectives and expected results should as much as possible be SMART but also considers that this is in many cases not easy to implement. This is particularly true where expected results can only reasonably be measured in qualitative terms and cannot be quantified which is inter alia the case for results of broad policies or financing instruments.

Corporate results frameworks, by nature, focus on quantitative results, as mentioned in the Staff Working Document ‘Launching the EU International Cooperation and Development Results Framework’ (EU RF). Indicators in the EU RF have been set as much as possible following RACER criteria (‘Relevant, Accepted, Credible, Easy, and Robust’) — as set out in Eurostat’s indicator guidance, referred to by the Court.

29

The Commission agrees that the broad variety of the documents referred to by the Court increases the risk of inconsistencies. Taking into account that the documents are adopted at different stages of policy formulation and the project and programme management cycle (including the programming phase) it necessarily results in adjustments and more specific types of focus in implementation.

The Commission has taken measures to reduce the risks mentioned by the Court. Thus, in the 2014-2020 programming exercise, country, regional and thematic programming documents have been systematically reviewed in order to ensure expected results and indicators are of appropriate quality (i.e. measurable) and of a manageable number (see the programming documents at http://eeas.europa.eu/around/index_en.htm).

The revision of the Action Document template (in force as of 1 January 2015) and the ongoing revision of the related internal Quality Support Group documents aim to ensure that expected results and indicators are properly included in Action documents.

Furthermore, the Commission’s services are working on the improvement of the Monitoring and Evaluation system, including room for adjustment that may be needed over implementation of the actions and the flexibility that is required to implement such adjustments in order to adapt to changing country contexts and policies.

32

The Commission has taken and is taking measures to reduce the risks mentioned by the Court. See Commission reply to paragraph 29.

Box 4

In compliance with recommendation 2 on the development of performance indicators made by the Court in the Special Report 14/2013, the Commission developed a results-oriented framework that includes indicators to monitor the progresses of the Palestinian Authority in its reforms.

The results-oriented framework, which has been developed in coordination with the World Bank and with EU Member States in the framework of the EU Local Development Strategy, was agreed with the Palestinian Authority through a Memorandum of Understanding signed on 25 March 2015. The areas covered in the pilot phase include Macroeconomic/Fiscal indicators, PFM, public administration reform (including measures addressing the reintegration of non-working Gaza civil servants), education, health and social protection. An evaluation of the pilot phase will take place end 2015/early 2016 and additional sectors could be added for the next phase. The text above shows that indicators have been established, which is in line with the ENPI regulation.

Box 5

The Commission sees the case as an example where it ultimately had to adapt the understanding of the indicator to the conditions of specific local context and accordingly to accept the adaptation of the target value of the indicator. The example demonstrates the importance of being continuously attentive to such situations but does not relate to a risk which the Commission can manage otherwise than in that way.

38

Institutional reforms are a long and difficult process where helping the government of a partner country to progress step by step can be more sustainable than a quick win of project. See reply to Box 6.

Box 6

The Commission notes that the quoted payments (referring to June 2008, December 2010, July 2011) largely precede the new policy on budget support which introduced a much more rigorous assessment of eligibility criteria, notably the PFM criterion. The risk which is to be illustrated by this example is reduced through and since the application of the 2012 budget support guidelines.

In its reply to the Court's findings the Commission also noted that ‘A distinction must be made, on the one hand, between the eligibility criteria for budget support, which require a holistic approach based on credibility and positive implementation and which are by nature general, and, on the other hand, the specific conditions, attached to variable tranches, which must indeed be clear and specific …. Donors must acknowledge the complexity of reforms and define targets that are more reasonable and better sequenced. For example, progress in PFM reform in both countries, Kyrgyzstan and Egypt, receiving budget support has been slower than expected. Indeed PFM reform plans, initially largely prepared by external donors, were far too ambitious and underestimated the local capacities to implement them. To remedy this, PFM reforms and action plans were revised to be more realistic and additional Technical Assistance was provided.’

As a result, the case is rather an example of overly ambitious objectives. In addition, it has to be underlined that the reform of the system of public finance management is a long-term process. Within the timespan of one operation of 3 years, only very limited results can be detected.

The case of Egypt rather relates to an example where the issue is not about defining a partner country’s commitment to implement reforms as an objective (see title of section 4.1.4) but, in terms of the 2012 budget support guidelines, about the precondition for disbursement related to essential elements like human rights.

Following the Council conclusions of August 2013/February 2014 and the recommendations of the Court, the Commission has focused its assistance on socioeconomic interventions and support to civil society.

There has been no new budget support operation in Egypt since 2011, and no budget support disbursement has been made on any of the five ongoing budget support operations since 2012.

Box 7 — Second indent

The budget support operations in Malawi and Rwanda are dating before 2010 and largely precede the new policy on budget support which introduced a much more rigorous approach to clarifying the link between the operation financed by the EU and the national policies.

The risk which is to be illustrated by this example is reduced through and since the application of the 2012 guidelines.

Box 8

The operations targeted emergency budget support measures to allow macro-economic stabilisation of the country with the global objective of sustaining the Democratic Republic of Congo in the economic and food crisis, and consequently, the results analysis has to be seen in this context.

The PFM reform plan the Court refers to was indeed mentioned in the Financing Agreement and its establishment was one of the milestones to measure progress.

Fixed tranches are also subject to eligibility, i.e. performance related. Therefore the payment conditions were linked to objectives including PFM improvements.

46

EU contributions to country development results will be reported as country results supported, thus opting for a contribution approach (see Commission’s March 2015 Staff Working Document ‘Launching the EU International Cooperation and Development Results Framework’). To quantify such contribution, two methods which are currently being used by donors are possible:

—the ‘overall method’, where the total (or overall) results the EU achieved jointly with others (such as the partner country government and other donors) would be reported, rather than results linked to the share of funding that has been provided by the EU;

—the ‘proportional method, where results would be reported based on the EU’s proportion of total funding. Under this approach results of projects and programmes are calculated as a pro rata share equal to the level of financial inputs provided by the EU. In this approach, the reporting proceeds implicitly on a direct causal link between funding provided and results achieved.

At an aggregate level, like in the case of reporting on the basis of the new results framework, one method needs to be chosen. As detailed budget information on inputs by others, including partner governments, is not always easily available to allow proportional results to be calculated, the Commission has opted for reporting following the overall approach as being for the time being the only realistic option to report results achieved in an aggregated way. As equally set out in the March 2015 Staff Working Document, the overall method would also be more appropriate from an aid effectiveness perspective, with the strong focus it places on country ownership.

Box 9

Whereas it is possible to assess in general whether a national policy to which a budget support programme provided support, has contributed to certain well defined results achieved at country level, it is not possible to attribute a particular proportion of the result to the EU funding.

Box 11

The example set out in Box 11 rather focuses on the insufficiency of integrating the human rights perspective in programmes than on the increased complexity due to the integration of cross-cutting issues or on the additional resulting risks for appropriate definition of objectives and indicators and related results measurement.

Box 14 — Second indent

In 2015, the Commission has introduced specific provisions on results reporting by EU implementing partners in the general conditions of the Administration Agreement (covering the World Bank as implementing partner) and of the ‘PAGODA’ Agreement covering all pillar assessed organisations — including UN Agencies. This should ensure greater availability of results data in progress and final reports transmitted by EU implementing partners.

59

The fact that in certain cases, by the nature of the project or programme, the results can only be identified and reported at the end of the project or programme, and therefore do not allow for periodic assessment of their achievement, does not inevitably lead to weak reporting on results. In such cases milestones or benchmarks can be identified which assist in tracking whether the project or programme is well under way to the effective achievement of results.

60

The Commission has already taken steps to improve evaluations by:

—drafting and issuing an ‘Evaluation Policy for the European Union development cooperation’ (2014);

—improving the quality of the planning, implementation, monitoring of evaluations of projects and programmes (ongoing);

—increasing the involvement of the management in strategic evaluations already at the Inception phase (ongoing); and

—applying the ‘Evaluation first principle’ in accordance with the Communication COM(2013) 686 final.

Box 15

The Commission does not consider the example as a case of weakness in results reporting or evaluation with respect to the action as a whole, the effectiveness of which can per definition only be assessed towards the end of or after the implementation of all its components or sub-projects.

65

In 2015, the Commission will issue its first consolidated results report in relation to the assistance provided within the context of the EU international cooperation and development policies. This will take the form of a complementary report to the Annual report.

Box 16 — Third indent

Improved monitoring and reporting systems and the implementation of an operational information management system (replacing also CRIS) will significantly step up EuropeAid’s use of information systems for reporting, in particular results reporting.

Box 17

The weaknesses of national data collection and information management systems in the countries to which the EU provides assistance is a factor which is inherent to the developing status of the countries concerned and a permanent point of concern on all donors’ side. This can only be sustainably addressed by building up appropriate national statistical systems. This risk factor is mitigated on the EU side as part of the sector and other support the EU is providing. It can however only be addressed sufficiently by sustained collective efforts of the country and the donor community.

72

The MFF 2014-2020 introduces a number of new features particularly to stimulate spending programmes' performance. Building on this, a Commission initiative lead by Vice-President Kristalina Georgieva, Budget and Human Resources aims at a shift from spending to an EU Budget Focused on Results. The objective is to create the conditions for further strengthening the budget’s performance in terms of both programme results and sound financial management of funds in a balanced, comprehensive approach. This aims at reducing the risk of focusing on budgetary implementation to the detriment of programme results, including spending areas and methods by which the EU budget can maximise results and rigorous application of cost-effective controls for enhanced compliance.

73

The Commission addresses the risk by multiannual programming, by 24 months planning of the implementing individual actions and by the internal quality support group process. These actions are complemented by further measures such as the upcoming EU Results Framework. See also reply to paragraph 29.

75

For budget support programmes, the Commission has developed a risk management framework (RMF) to identify major risks and mitigate them. The RMF is taken into account in the decision-making process notably in the enhanced governance through the internal Budget Support Steering Committee. The RMF informs systematically the policy dialogue with partner countries on the strategic level, as well as by focussing on key issues. The RMF is used to balance expected benefits, as defined in the action document, with identified risks and proposes mitigating measures.

Internal analysis (budget support report 2013 and 2014) shows that EU budget support countries clearly show better and improving PFM performance compared with non-budget support aid recipients.

The risk which is to be illustrated by this example is reduced through and since the application of the 2012 budget support guidelines.

Box 19

In 2015, the Commission has decided to step up on a case-by-case basis the underlying assessment whether or not to work with an international organisation as an implementing partner.

Box 20

The Commission recognises that changes in the political and economic context represent risks in terms of results to be attained. This type of risks are the reason why the context analysis is an important feature of the standard identification and formulation process applied by the Commission’s services and EU Delegations in the preparation of projects and programmes.

In order to adapt to unforeseen changes of the aid context a certain degree of flexibility is needed during the implementation phase. This has been ensured by simplifying relating rules and procedures as of July 2014. These simplification measures included changes relating to the use of indicators so that modifications of specific priorities or in relation to the collection and availability of data can be taken into account.

It can however not be excluded that the project’s initial objectives cannot be achieved, in which case they need to be adapted to the changing circumstances and context.

Box 21

The reorganisation of January 2015 will as such affect neither the definition of objectives nor the attainment of related results within the framework of specific actions.

The Commission, irrespective of its own administrative organisation, and in line with the provisions of the common implementing regulation covering the ENI countries (in particular articles 12 and 13), will respect its obligations to monitor the implementation of projects and report to the Member States, the EP and other EU institutions.

The Commission will be reporting on the basis of the results framework also for the Neighbourhood countries, consistent with the Council conclusions n. 9145/15 of 26.05.2015. A specific tool to provide support to ‘end of project results reporting’, has been put in place within the framework of the new Result Oriented Monitoring (ROM) system to help consolidating data on end of project results data, in particular in relation to the indicators based on the agreed results framework.

As regards evaluation, DG NEAR has its own Evaluation Function, addressing since 2015 both the Neighbourhood and the Enlargement regions.

5. Conclusions and recommendations

80(a)

At various levels, the Commission services have been working on ensuring consistent terminology: the sector indicator guidance 2013, the action document template and related guidance applicable as of 1 January 2015, as well as the new International cooperation and development results framework and the related instructions and guidance for reporting on the basis of the results framework.

The Commission services are furthermore in the process of strengthening support to internal quality assurance with respect to these aspects of individual action documents, considered important by the Commission both for monitoring and reporting on implementation, performance and results and for the evaluation of the actions.

They are all also in line with the terminology used in the basic regulations which constitute the regulatory framework for the EU external assistance.

The Commission agrees that establishing ‘a clear logical chain’ from inputs to impact is of utmost importance at project level. This has been reinforced with the introduction of the Action Document template as of 1 January 2015. However, establishing a clear logical chain at strategic or at policy level will never reflect the complexity of development context. In the international evaluation world, the use of the linear logical framework at such strategic/policy level is therefore more and more challenged.

80(b)

The integration of cross-cutting issues presents indeed several challenges to appropriate formulation of the specific objectives and results of the actions and to the collection of corresponding data allowing to monitor their implementation.

80(c)

The risk of inefficiencies and gaps to which the Court points would seem most present at country level. This demonstrates the importance of the alignment by donors on the country system/policy/results framework as the preferred approach compared to only harmonising among donors, as already pointed out by the Court in paragraph 17(a) and (b).

By applying this approach, the Commission has already been making important efforts in the context of the 2014-2020 multiannual programming to align as much as possible on country policies and frameworks, in accordance with the Busan commitments on aid effectiveness. This has also been the Commission’s approach in setting up the new corporate results framework. At country level, the Commission is together with other donors in favour of the definition of country-owned results frameworks and sees joint programming with EU Member States as one of the means to promote such frameworks.

80(h)

Changes in the context of actions may lead to changes of the indicators used or of the initial target values of the indicators.

Recommendation 1

The Commission accepts the recommendation and has already been working on ensuring consistent terminology: the sector indicator guidance 2013, the action document template and related guidance applicable as of 1 January 2015, as well as the new International cooperation and development results framework and the related instructions and guidance for reporting on the basis of the results framework.

They are all in line with the terminology used in the basic regulations.

The Commission services are furthermore in the process of strengthening support as part of internal quality assurance with respect to these aspects of individual action documents, and will also continue to improve its general guidance on project and programme management and staff training modules in this area.

Recommendation 2

The Commission accepts the recommendation.

It has been in the process over the past 2 years of preparing various measures to strengthen its systems, guidance and processes in this respect. Please see also reply to Recommendation 1.

The Commission has agreed (see reply to paragraph 80 a) that establishing ‘a clear logical chain’ from inputs to impact is of utmost importance at project level.

Recommendation 3

The Commission accepts the recommendation.

Actions are under way to improve the whole monitoring and evaluation system, enhancing also the Commission’s capacity to identify lessons (see also the Commission’s replies to the Court’s Special Report 18/2014).

As regards more specifically the information system, the actions the Commission has been taking so far is the establishment of a preliminary system to ensure consolidated results reporting for its first results report in 2015 prior to the development of the first modules of its new operational information system which are planned to comprise also the reporting on results.

Anticipating the new system and in line also with one of the recommendations of the Court in its Special Report 18/2014 on Evaluations and ROM, the Commission is in the final stage of the development of a specific information module relating to evaluations, foreseen to be put into production in the first half of 2016, which will progressively improve the Commission’s capacity to identify lessons learnt for the various relevant purposes.

Recommendation 4

The Commission accepts the recommendation. It has already stepped up its efforts in that sense as mentioned in its reply to Recommendation 1.

Recommendation 5

The Commission accepts the recommendation.

The implementation of the recommendation will be part of the general assessment of the risks inherent to an action (e.g. choice of a particular implementation modality, please see also section 2 ‘Risks and assumptions’ of the template of the Action Document applicable since 1 January 2015).

For budget support programmes, the Commission has developed a risk management framework (RMF) to identify major risks and mitigate them. The RMF is taken into account in the decision-making process notably in the enhanced governance through the internal Budget Support Steering Committee. The RMF informs systematically the policy dialogue with partner countries on the strategic level. The RMF is used to assess the balance to be ensured between expected benefits of the envisaged action, to identify risks and to propose risk-mitigating measures.

Internal analysis (budget support report 2013 and 2014) shows that EU budget support countries clearly show better and improving PFM performance compared with non-budget support aid recipients.

In 2015, the Commission has decided to step up the underlying assessment whether or not to work with an international organisation as an implementing partner on a case-by-case basis.

QJ-AB-15-019-EN-E    ISSN 1977-5679

In our review, we identified nine key risk areas related to a results-oriented approach for EU development and cooperation action.
Our review showed that these risk areas had been correctly identified by the Commission. It also showed the interest in the Commission’s initiatives to improve EU development and cooperation results.
Nevertheless, further actions remain to be taken in order to make these initiatives fully effective. We formulate a number of recommendations in this regard.