1 Introduction to smart specialisation

Within the recent EU regional and urban policy reforms, the emergence of smart specialisation as a policy prioritisation framework is by now well documented (Foray 2015: McCann and Ortega-Argilés 2013a, b, 2015a; McCann 2015), and importantly for our purposes here, this approach also brings entrepreneurship and SMEs centre-stage in EU development policy thinking. The approach is one of the key elements of the reforms to EU regional and urban policy, or more accurately EU Cohesion Policy. Yet, understanding the origins and the emphases of the smart specialisation way of thinking is critical in understanding how such an approach helps to address some of the key development challenges facing European regions.

As is by now well-documented, the original conceptual framework for smart specialisation was developed by the ‘Knowledge for Growth’ K4G expert team of analysts advising the EU Commissioner for Research Potočnic (McCann 2015). Their work focused on understanding and responding to Europe’s weaknesses in developing new technologies and ideas and also Europe’s weaknesses in translating these ideas and technologies into commercial applications. In particular, the efficacy of the role played by entrepreneurship in driving innovation was considered as being central to the European issues, because entrepreneurship was understood as being key to fostering not only innovation, but also innovations that could be successfully nurtured, disseminated and taken up within the wider EU economy. Therefore, finding ways to enhance the scale and effectiveness of entrepreneurial processes was seen as being a critical policy challenge.

In terms of enhancing EU growth, the various briefing papers produced by this expert advisory group highlighted the importance for Europe of maximising the alignment of incentives and linkages between actors, institutions and policy-settings in order to best facilitate entrepreneurial search processes (Foray et al. 2009; David et al. 2009). Entrepreneurial actions in this broad sense can be understood either in terms of classic business start-ups, or the venturing of SMEs into new activities and technologies, or even the efforts by large firms to build new networks with SMEs. A common feature here must be that the entrepreneurial actions contain a sufficient degree of experimentalism and self-discovery (Hausmann and Rodrik 2003) as is essential in all forms of innovation. Within this arena, smart specialisation argues that finding ways to link new technologies and activities to as wide a range of sectors and activities is critical. Although the original ideas were developed in a non-spatial setting, it became increasingly clear to the proponents of this approach that it was especially pertinent to the EU regional context (Foray et al. 2012). Indeed, translating the non-spatial smart specialisation logic to an explicitly spatial setting is not so difficult, because many parallel developments in economic geography had been moving in very similar directions (McCann and Ortega-Argilés 2015a). Indeed, one of the observations of the OECD (2013) is that an important aspect of the smart specialisation approach is that it brought together into a single and simple framework many different elements which were already evident in a diverse range of literatures but which had not been coherently integrated in any systematic format.

The basic argument of the smart specialisation approach is that policy resources must be prioritised on those activities, technologies or sectors where a region has the most realistic chances to develop wide-ranging and large-scale impacts which also develop and build on many different local and interregional linkages and connections (Foray et al. 2012). Such an approach requires that many of these activities and technologies to be prioritised are already partly embedded in the region’s existing industrial fabric and that as many local actors and institutions are engaged in the policy design and delivery process. Importantly, policies should be focused on stimulating and facilitating entrepreneurial actions, and with local SMEs being seen not only as the key priority in their own right but developing links between SMEs and other larger local actors is a critical agenda. The argument here is that such linkages can provide the platform and network effects which local entrepreneurial initiatives need in order to build requisite scale.

This line of thinking inherently involves an entrepreneurial ecosystems type of approachFootnote 1 in which the role of entrepreneurship in driving local innovation is seen as being critical for enhancing regional competiveness. These systems type of thinking implies that policies may be targeted on any of the technological, financial, institutional, or skill-related elements within the ecosystem, either to enhance certain features of the local entrepreneurial system, to overcome key constraints and bottlenecks, or to bridge any perceived missing links. Indeed, one of the key observations of the K4G expert group was that the widespread misalignments evident local and regional policy-making the application of the smart specialisation approach as a key element of the EU Cohesion Policy reforms signals a clear intention to put entrepreneurship and the role of SMEs in fostering growth and development as the central tenets of the new generation of regional development policies. For scholars working in the fields of entrepreneurship and small businesses, these have long been advocated as the most realistic way forward for fostering development. However, the smart specialisation approach also signals a movement away from regional development policies emphasising flagship high-technology initiatives or the advocacy of large-scale infrastructure building and instead focuses on fostering development via the promotion of local entrepreneurial processes aimed the technological upgrading of the existing industrial fabric of the region.

In order for the smart specialisation approach to underpin good policy prioritisation processes, it is essential that the approach becomes explicitly outcome-oriented or result-oriented and this was clearly recognised by the proponents of the approach (David et al. 2009). As such, this also sets smart specialisation squarely within the new generation of policy thinking in which outcomes, results and the expected theory of change are all to be clearly articulated in advance. Yet, how to do this itself represents a challenge, and the aim of this paper is to outline these challenges and to identify ways of making progress in overcoming such challenges.

The rest of the paper is organised as follows. In the next section we discuss the role of entrepreneurship and SME policies in the broader context of the EU policy agenda as well as in the specific context of EU regional policy. In Sect. 3 we discuss the requirements of outcome-oriented or result-oriented policies in terms of data, indicators, monitoring and evaluation, and in Sect. 4 then discusses evaluation issues specific to policies aimed at fostering entrepreneurship and SMEs. Section 5 provides some brief conclusions.

2 Entrepreneurship policies and the European context

The fostering of SMEs, and in particular of entrepreneurship, are critical factors in driving economic development because of their impacts on wealth generation, innovation, skills and capabilities, the opening up of new markets, job creation and job satisfaction (Van Praag and Versloot 2007; Feldman et al. 2011; European Commission 2012). SME and entrepreneurship policies are governmental initiatives aimed at positively influencing the formation, viability and commercial success of new and smaller scale firms. However, in the majority of the cases these policy initiatives are developed at different governance levels—local, regional, national and supra-national, so that policy is able to best access the right small-scale stakeholders at the local and regional levels. Indeed, this multi-level aspect of entrepreneurship and SME policies often makes their monitoring and evaluation somewhat harder than for other types of policies.

There are various well-known market failure arguments for entrepreneurship policy (Storey 2008) relating to private versus social benefits, asymmetric and imperfect information, knowledge externalities, and barriers to entry (Stevenson and Lundström 2007). Moreover, depending on the issues being addressed there are also different types of entrepreneurship policies (Stevenson and Lundström 2007) with different intended objectives, namely extension policies, new firm creation policies, ‘niche’ target group policies and ‘holistic’ entrepreneurship policy. Entrepreneurship extension policies have as the main objective to improve the access and services to start-up supports through existing SME support structures. New firm creation policies focus in reduce the barriers to firm entry and exit, increase the start-up rate and reduce the red tape and administrative burden. ‘Niche’ target group policies are focused in the group that are underrepresented in entrepreneurship. Finally, the most comprehensive entrepreneurship policies are the ‘holistic’ entrepreneurship policies whose main objectives are to increase the entrepreneurial culture. Allowing for these different types of policies Hart (2003) argues that the two distinct aspects to entrepreneurship policy relating to public policy and public governance need to be differentiated, whereas public policy includes actions taken by the government and institution, public governance focuses on more informal means of supporting entrepreneurs. These two aspects combined together help to provide a sounder base for strong entrepreneurial growth by providing official as well as communal support.

However, as we see in Table 1 there is still a clear distinction between SME and entrepreneurship policy. SME policy applies to existing enterprises, whereas entrepreneurship policy relates to policies seeking to enhance the creation of new enterprises. At the national level many policy initiatives around entrepreneurship and SMEs are taking place in Europe. For example, Denmark has introduced various initiatives that target entrepreneurs in their country including the Danish Foundation for Entrepreneurship or the Global Entrepreneurship Week (Danish Business Authority 2015) while in Sweden, the Swedish Entrepreneurship Forum was founded to serve as a source of information and it also has a strong focus on research and on connecting the academic and the real life facets of entrepreneurship. Indeed, there are many more examples of specific initiatives taken by governments to support the issue, not only at national but also at other more local and regional levels (McCann and Ortega-Argilés 2013a, b, c). As we see from Table 2, many European regions are also heavily engaged in various policy initiatives and programs focused on fostering entrepreneurship and enhancing SMEs.

Table 1 Features of SME and entrepreneurship policy measures and examples
Full size table
Table 2 Logic of intervention features
Full size table

From the perspective of EU Cohesion Policy, these initiatives are very important. Cohesion Policy is today the largest policy source of funding for SMEs in Europe. However, the broader background context in which the smart specialisation approach to EU Cohesion Policy has also emerged in the post-crisis period as one in which the fostering of entrepreneurship and SMEs is nowadays seen as being critical for the whole future of Europe, in ways which were not quite the case in earlier years.

Today, all of the EU policies related to SMEs and entrepreneurship, including those emerging from EU Cohesion Policy, fall under the broader Europe 2020 pillar of Smart Growth. From this broader umbrella emerges the Entrepreneurship 2020 Action Plan which is a blueprint for decisive action to reignite the entrepreneurial spirit in Europe and this Action Plan acts as a follow-up to the Small Business Act (SBE) review of April 2011. The Entrepreneurship 2020 Action Plan is built on three main pillars: entrepreneurial education and training to support growth and business creation; strengthening framework conditions for entrepreneurs by removing the existing structural barriers and supporting them at different stages of their business lifecycle and dynamising the culture of entrepreneurship in Europe by nurturing the new generation of entrepreneurs, additionally reaching out to specific groups whose entrepreneurial potential is not being tapped to its fullest extent or who are not reached by traditional outreach for business support is also under their priorities. The Communication on the Action Plan was preceded by a public consultation in July 2012. The consultation did not target any specific group as all citizens and organisations were welcome to participate. Among other conclusions, the public consultation showed that access to finance constitutes one of the most significant constraints on the growth of SMEs and entrepreneurship in Europe. (European Commission 2012).

These shifts in thinking towards prioritising entrepreneurship and SMEs are also heavily reflected in the reformed Cohesion Policy framework, and in particular in the smart specialisation approach to policy prioritisation. In the case of actions and interventions arising specifically from Cohesion Policy, all entrepreneurship and SME-related policies operate under the Thematic Objective of the Cohesion Policy Operational Programmes 2014–2020 entitled ‘Enhancing the competitiveness of Small and Medium Enterprises (SMEs)’. These Cohesion Policy actions are all linked with the ex ante conditionality: Specific actions that have been carried to underpin the promotion of entrepreneurship taking into account the ‘Small Business Act’Footnote 2 for Europe (SBA). The investment priorities connected with this Thematic Objective under the European Regional Development Fund (ERDF) are promoting entrepreneurship and supporting the capacity of European SMEs. The broad rationale behind these thematic priorities and ex ante conditionalities is that competitiveness and growth of SMEs and the starting steps of new companies are often hampered by a poor business environment that does not consider their financial, administrative and other specific needs. Without improvements in these fields, the investments devoted to SMEs would risk not delivering on their expected impacts. In particular, policy actions are to be aimed at: promoting entrepreneurship by facilitating the economic exploitation of ideas; fostering the creation of new firms; supporting the capacity of SMEs to grow in regional, national and international markets; and helping SMEs to engage in innovation processes. Among the specific actions that the European Commission envisages are also measures to reduce the time and cost involved in setting-up a business or developing monitoring and evaluation mechanisms to assess the implementation of the SBA.

3 The rationale and logic of intervention for result-oriented policies

Originating as a conceptual framework for thinking about growth priorities in an aspatial setting smart specialisation has now become a key policy prioritisation framework in EU regional development policy. The focus of the policy is on promoting entrepreneurial search processes in local and regional economies. Yet, this is not, and never has been, about sectoral specialisation, but rather about carefully choosing priorities which are best suited to moving the region from its current development trajectory to a stronger trajectory via the enhancement of the local entrepreneurial climate. However, in order to effect this it is smart specialisation also requires that policy makers, working in tandem with as many other local stakeholders and institutions as possible, articulate a theory of expected change in their local and regional context which motivates the prioritised policy interventions, and such an theory of change must build precisely on the type of entrepreneurial ecosystems-type of thinking. In other words, policies need to be chosen and designed with the existing portfolio of assets in mind and with explicit goals as to what is intended to be achieved by the polices. Yet, as already mentioned above, the multi-level context in which many aspects of entrepreneurship and SME policies operate often makes their monitoring and evaluation somewhat harder than for other types of policies, and these issues need to be built into the policy design right from the outset.

In order to commence the policy prioritisation process smart specialisation therefore requires a detailed analysis of the current regional economic and industrial structure on the basis of the best available evidence currently available. For this we need baseline or profiling indicators. No evidence will be complete or ideally constructed but working with the best evidence and indicators available is essential for smart specialisation. If activities, technologies, inter-institutional linkages, sectors, or a mix of these are to be prioritised as part of a smart specialisation agenda, then there have to be clear arguments as to why these are being prioritised and these depend on the theory of expected change which is being articulated. Smart specialisation clearly helps to establish these priorities, but once they have been established then there also needs to be a clear logic for assessing the progress of the chosen policy interventions.

In order to provide the requisite clarity of intended policy objectives and to facilitate the better design and delivery of interventions and actions, policies which are amenable to monitoring and evaluation exercises are increasingly advocated. One of the key components of such policies is that they permit the use of outcome indicators or results indicators. There is a wide-ranging literature (Rodrik 2004, 2007; World Bank 2010) which argues that developing a result-oriented policy setting is generally perceived as being an important topic in any type of industrial policy or regional development policy, and within this policy portfolio entrepreneurship and innovation-related policies are increasingly seen as being essential for all aspects of growth (McCann and Ortega-Argilés 2013a, b, c). The use of outcome/results indicators allied with monitoring and evaluation exercises helps to make explicitly clear the intentions and objectives of the policy, because only with such clarity of intentions and objectives can indicators be chosen and policies monitored and evaluated. Yet, while in the field of public policy analysis the idea of outcome-oriented or result-oriented policies is generally regarded as being a sensible and meaningful way of thinking about policy design, in reality it is surprising how few policies are really result-oriented in terms of both design and delivery. Many policy interventions even in advanced economies have little explicitly measurable objectives in-built in their design and very few are therefore amenable to comprehensive monitoring and evaluation exercises. Many policies appear to have multiple—and often too many—goals, while others have stated objectives—such as raising GDP—which are realistically too far away from the individual policy actions or interventions to be meaningful. Instead, what are needed are a small number of clearly stated objectives and intended outcomes which are realistically close enough to the policy actions to be connected to those same actions, and which are also directly amenable to tracking via the use of indicators. Otherwise, it will be impossible to identify whether the apparently observed outcomes of the policy are actually due to the policy actions.

There is now a growing literature on outcome-oriented policy making (McCann 2015), the requisite properties of outcome indicators and results indicators (Barca and McCann 2011a, b, c), along with the features of good evaluation and monitoring exercises (Davies et al. 2000; Pawson 2006; Technopolis Group and MIOIR 2012; Link and Vonortas 2013; Gault 2013; CCA 2013; World Bank 2002; European Commission 2013b; Hempel and Fiala 2011).Footnote 3 The principles emerging from this literature must necessarily be applied directly to the smart specialisation agenda and tailored to the local context and the specific chosen policy priorities. In the case of EU regional policy interventions the evaluation approach to be employed is discussed in detail by the European Commission Evalsed Footnote 4 guidebook along with other detailed policy evaluation guidance documents.Footnote 5

In terms of assessing the progress of the policy via monitoring and evaluation, as we see in Fig. 1 the logic of intervention to be adopted is:

Fig. 1
figure 1

Logic of intervention

Full size image

In this logic of intervention framework, the inputs are the financial resources employed in the policy interventions, the outputs are the directly measurable actions whose intention it is to produce results, and the results/outcomes are the changes in behaviour which the policy is intended to influence. The results/outcomes indicators are designed to capture the changes in the intended results/outcomes, and the impact of the policy is the change in the results/outcome indicator which can credibly be ascribed to the policy intervention such that the movement towards the desired outcomes can be confidently related to the policy.

There are various different uses of these types of terminology and Table 2 provides examples of how these types of terms are used in the case of various innovation and R&D-related programmes, and Fig. 1 and Table 2 provide a more detailed and nuanced diagrammatic schema of the logic of intervention in innovation-related interventions. For clarity and consistency, however, here we use the terminology exactly as it is employed by the European Commission in the specifications and regulations for Cohesion Policy.

Using this framework, smart specialisation makes clear the ex ante policy prioritisation principles which underpins the logic of the overall strategy design. Ex ante evaluation is key to assessing whether the proposed actions are relevant and coherent and whether the expected impacts are realistic. Ex ante evaluation is important for designing indicators as well as the procedures for subsequent monitoring and evaluation activities. Monitoring is used to observe the ongoing behaviour of the results/outcome indicator as the policy progresses and evaluation is the ex post activity by which the impact is assessed. Program and project evaluation approaches concerning to innovation-related activities typically use a combination of a realist case study perspective alongside an ex post counter-factual analysis based on logical positivist principles, as outlined in Table 3.

Table 3 Suggested measurement methodologies by innovation program type
Full size table

However, evaluation cannot realistically be undertaken unless targets also exist. This is because evaluation can only take place in a framework in which the expected policy impacts are clearly specified. As such, a result indicator will always have an associated baseline value that is related with the value of the indicator before the policy intervention, linked with a number or a description of the situation and a target that is the intended value or the quantification or desired development trend after intervention in a particular year or period.Footnote 6 Hence, considerations of how a policy is to be evaluatedFootnote 7 should therefore be incorporated into policy formulation when new ideas are being developed and the indicators are being chosen. Moreover, in order to coherently link the logic of intervention to the monitoring and evaluation activities we need different types of indicators. Indeed, the most robust and sophisticated indicator-based monitoring and evaluation systems are those which incorporate both quantitative and qualitative evaluation methodologies which are intended to complement each other and to respond to different issues and provide insights. Qualitative and case study techniques (Vanclay 2012) allow for a detailed understanding of how the expected links within the theory of change operated and performed, while quantitative indicators more readily permit ex post and counter-factual type evaluation approaches (Scarpa 2012). Qualitative evaluation methodologies engage participants in the policy learning, offer a deeper understanding of processes leading to impacts, can assess against a wide range of evaluation criteria and allow to pick up unintended consequences. However, qualitative evaluation methodologies also have disadvantages such as respondents and interviewers may be biased or poorly informed, rarely provide a clear answer, tend to ‘describe’ rather than ‘evaluate’, have the risk to including ‘unrepresentative’ groups and present difficulties in judging efficiency and effectiveness or establishing cause and effect. On the other hand, quantitative evaluation techniques also have advantages and disadvantages associated to the situation: among their advantages, they provide clear answers on impact or can be independently verified; among the disadvantages: they have a higher associated costs related to data collection and technical demands; lack information on context and mechanisms behind policy impacts; absence of pure control group; possible false impression of precision; narrow focus on effectiveness and efficiency and are difficult to use on indirect interventions that seek to influence the business environment. Realist approaches to evaluation (Davies et al. 2000; Pawson 2006; Link and Vonortas 2013; OECD 2007; Stockmann 2011; Sedlacko and Martinuzzi 2012) aim to combine these different techniques in order to produce a portfolio of evidence including outcome indicators (Abreu 2012) which ideally largely point in the same direction.

What is most important here is that for smart specialisation innovation and entrepreneurship policies to be result-oriented, it must be the case that the logic of intervention, the theory of expected change, the indicators to be employed during the life cycle of the policy programmes, the data to be constructed, and the design of the policy, are all closely interrelated issues which cannot be divorced from each other. Table 1 outlines some of the key principles in terms of the links between the project design and its impact evaluation potential via the use of indicators.

McCann and Ortega-Argilés (2015b) present a more detailed outline of the types of evaluation methodologies typically employed in different settings and their particular and disadvantages, including: different types of policy monitoring and evaluation techniques and tools which are employed in different contexts; examples of systems of indicators used in various cases for capturing the effects of innovation and entrepreneurship-related policies and examples of good policy monitoring and evaluation practices employed by different European regions. Importantly for our purposes, what becomes clear is that there is no ‘one-size-fits-all’ blueprint or template for the use of results indicators and for result-oriented policy evaluation. Rather, these results orientation and policy monitoring and evaluation aspects have to be built into the policy design right from the beginning, exactly as the originators of the smart specialisation concept understood (David et al. 2009). Adopting a result-oriented approach to policy making therefore imposes an analytical discipline on all aspects of the policy process which allows for agreement between different parties, actors and institutions on the basis of intentions, analysis and expectations, and works against a purely political logic to the policy process. The clarity of analysis and expectations introduced into the policy process also facilitates a policy transparency and accountability which is associated with an openness to measurement, monitoring and evaluating, and as far as possible the development of a culture of policy learning and institutional capability (Sedlacko and Martinuzzi 2012).

4 The evaluation of SME and entrepreneurship policies

Translating these principles, lessons and insights to the case of policies which prioritise entrepreneurship and SMEs operating under the Cohesion Policy rubric, also requires that we consider certain specific features relevant to entrepreneurship. As we have already mentioned, the evaluation of SME and entrepreneurship policies can be a complicated exercise due to the broad scope of policy actions in different mainstream government policies such as tax, education or social policies like immigration or unemployment benefits, among others. These policies have a clear effect on entrepreneurship and SME development but rarely are taken into consideration in the evaluation of the impact of SME and entrepreneurship policies. Additionally, we can also add the fact that SME owners, since they have a business to run, often regard themselves as having little time to engage with the government in providing data to secure the monitoring and evaluation of their activities. Finally, any evaluation should ensure that all types of SMEs are taken into consideration (gazelles, spin-offs, self-employees, micro-enterprises,Footnote 8 start-ups, etc.).

Having said that, the monitoring and evaluation of entrepreneurship and SME policies is extremely important not only to identify whether the programme represents a cost-effective use of public funds but also to foster policy learning and institutional capacity improvements regarding the better design of future programmes. As such, within EU Cohesion Policy the shift towards both an enhanced focus on entrepreneurship and SMEs and also towards an outcome-oriented and result-oriented approach signalled by smart specialisation also reflects similar earlier shifts in thinking in other arenas. In particular, several international organisations have already been developing framework for evaluating and testing entrepreneurship and SME policies, including the MILES framework from the World Bank (2007), the ‘SME Test’ of the European Commission, the ‘COTE’ framework of the OECD (2007) and the MILES framework from the World Bank.

4.1 The MILES framework (World Bank 2007)

The MILES framework for the development of entrepreneurship and SME policy (World Bank 2007) considers the overarching higher-levels goals of such policies and the acronym stands for the themes of: Macroeconomic and political stability; Investment climate, institutions and infrastructure; Labour market regulation and institutions; Education and skills; Social protection. Macroeconomic and political stability refers to the entrepreneurs’ need for a sound macroeconomic framework in which to expand their business and create new jobs. A good investment climate, institutions and infrastructure with predictable and low costs of regulation and doing business are essential in order to allow firms to expand and create jobs. Sound labour market regulations and institutions are crucial for enhancing long-term employer-employee working relationships. High-quality education and skills providers and infrastructure are essential for enhancing the labour markets and a strong and balanced social protection scheme protects the income of workers from shocks to employment.

Within these overarching high-level themes and goals, the MILES framework also identified more specific issues especially pertinent to entrepreneurship and SME policies.Footnote 9 Firstly, one of the key concerns of entrepreneurs and SME owners is connected with financial constraints. Among the mechanisms to accelerate the access to finance the use of loans seems to be more effective than grants in supporting innovation, and also non-financial ‘soft’ support such as business advice has been found also effective in business performance. As such, a combination of financial and non-financial support in one package seems to have a positive effect in the impact of the policy. At the same time, the most successful policy measures were the ones that target not just capital market failures but also information market failures. For medium-sized enterprises, innovation support, networking and innovation consortia have generally proved to be effective at increasing long-term growth and productivity. For small and micro-enterprises, in particular, basic business advice may be the single most cost-effective form of support. For SMEs of all size, this suggests that tailored packages which mix appropriate financial with non-financial elements represent the most effective policy support systems.

The ability of entrepreneurs and SMEs to access the right types of support and the ability of the policies to access the right types of entrepreneurs and SMEs depends in many cases on specific local knowledge and face-to-face interactions and this is especially so in situations where local and regional economic contexts differ markedly. The provision of these types of appropriately tailored policy support systems is likely to involve different actors and institutions in different combinations in different contexts. This type of multi-level governance approach is very important in the case of entrepreneurship and SMEs policies because horizontal and framework policies focusing primarily on producing good general economic framework conditions are unlikely of themselves to be sufficient to produce a more entrepreneurial society. As such, policy evaluation approaches need to be developed that permit policy makers with SME and entrepreneurship responsibilities to be able to engage more fully in cross-government discussions on priority setting. Not only does evaluation need to become more central to the policy-making process, but monitoring and evaluation need to be central to the policy design and delivery processes, rather than being undertaken solely as a historic accounting exercise to determine whether public money has been spent correctly.

4.2 The ‘SME Test’ of the European Commission

In line with the overarching MILES framework thinking, the ‘think small first’ principle of the European Commission requires that SMEs’ interests are taken into account at the very early stages of EU policy making in order to make legislation more SME friendly. The Commission Impact Assessment Guidelines support the application of the ‘SME Test’ whereby EC ministries and services should assess the impact of any forthcoming legislation and administrative initiatives on SMEs and take the results of this analysis into account when designing proposals, including the use of alternative mechanisms and more flexible approaches (European Commission 2009). In each case these assessment activities should include a detailed analysis of the current market conditions of SMEs, a cost-benefit type of analysis and various counter-factual assessments.

The ‘SME test’ comprises four main steps (European Commission 2009). The first step involves a consultation with SMEs/SME representative organisations. The test establishes a minimum consultation period including among other activities a Small Business Act follow-up meeting with stakeholders. The European Commission has developed a number of tools which help to get the opinion of businesses, and these include the Enterprise Europe Network and the Network of SME Envoys. Examples of good practices for the consultation of stakeholders include round table discussions with stakeholders, specific committees, on-line consultations, or public and industry forums (European Commission 2009).

The second step involves a preliminary assessment of the businesses which are likely to be affected. In this step the government should establish whether SMEs are among the affected population and the characteristics of the businesses and sectors likely to be affected should be identified. The relevant sources of information to be explored include: the number of businesses and their size; the proportion of employment concerned in the different categories of enterprises which may be affected; the weight or presence of the different types of SMEs in the sectors; and the links with other sectors and possible effects of subcontracting (European Commission 2009).

The third step involves a measurement of the impact of the proposed policy on SMEs. At this stage an exhaustive cost-benefit analysis should be performed. The analysis should examine the distribution of the potential costs including financial costs, substantive costs of adoption of standards and regulation, and administrative costs and also of the benefits such as the improvement of working conditions, any increases in competition, or accessibility to more qualified staff. It would also be important in many cases to run a comparative analysis between the costs and benefits of SMEs and large firms (European Commission 2009).

The fourth step involves the use of alternative options or mitigating measures, if appropriate. Such mitigating measures can include: the size-related exemptions from certain accounting requirements; a temporary reduction in or exemptions from some aspects of legislation; reduced fees; simplified reporting obligations for SMEs; the provision of specific information campaigns or user guides, and provision of training and dedicated helpdesks or offices (European Commission 2009).

4.3 The application of the COTE framework by the OECD (OECD 2007)

In line with the overarching themes of the MILES framework and the specific issues raised by the SME Test, the OECD (2007) argue that all entrepreneurship and SME policies should contain aspects that (OECD 2007) include: being a strong voice for small business at the heart of government and ensuring that government is aware of the needs of small businesses; striving for a regulatory framework which minimises the burdens on business; developing and maintaining a world class business support service to enhance the competitiveness and profitability of small businesses; and the championing of the importance of entrepreneurship across society, and particularly among under-represented and disadvantaged groups. However, in order to help such policies realise these higher-level goals, the OECD (2007) developed what is known as the COTE framework for assessing the effectiveness of specific policy actions or interventions in the field of entrepreneurship and SMEs.

According to this framework, the main overarching features required by entrepreneurship policy as reflected in the C-O-T-E acronym are: Clarity and Coherence regarding the Objectives of the policy, its specified Targets and Evaluation processes. In order to ensure that these features are always evident against this broad policy backdrop of higher-level objectives, the OECD (2007) identify seven heading against which the specific performance of any particular set of entrepreneurship policy actions or interventions can be assessed. These are: Rationale, Additionality, Appropriateness, Superiority, Systemic Efficiency, Own Efficiency and Adaptive Efficiency. Of these seven criteria, in the end the most important criterion is that of additionality, and it is this which essentially defines as the true impact of the policy scheme or programme.

The additionality of a policy on a firm or firms reflects the participating firm’s activities that would not have taken place without the programme. While this is not always easy to quantify, as there are well-known selection issues inherent in such schemes (Scarpa 2011), it is likely to be reflected in empirical measures such as additional output, employment, sales, innovations, or export activity that can be specifically attributed to the existence of the programme (OECD 2007). However, in the case of entrepreneurship and SMEs, this exercise may often be very difficult for the following reasons (OECD 2007). Firstly, it is not always clear what changes might have occurred in the firms as a result of participation. Some programmes might be expected to lead to a greater likelihood of firm survival, other growth in sales, profits or employment, others to the greater likelihood of innovating or selling into overseas markets. Secondly, participation in the programme will precede improvement and some programmes will have a more immediate impact than others. Thirdly, isolating which outcomes are related to which effects (managerial skills and experience, sector, location, macro-economic conditions etc.) may be very challenging. Yet, notwithstanding these various challenges, monitoring and evaluation are still essential for delivering outcome-oriented and result-oriented policies and however, much data are generated by the policy programmes, such processes can only be as good as the articulated theory of expected change on which the policy was originally designed. Indeed, it is this theory of expected change which itself helps to structure the evaluation exercises. Such evaluation almost always exercises require a mix of quantitative and qualitative evaluation exercises and in the specific case of policies focused on enhancing entrepreneurship and SMEs the ‘Six Steps to Heaven’Footnote 10 procedure has been defined to support the impact assessment of SME policies taking into consideration the potential problems that have been discussed above. This approach to evaluation requires considering: Step 1—the take-up of schemes in terms of counting the number of participants; Step 2—the recipients’ opinions and the assessment of the clients’ satisfaction; Step 3—the recipients’ views of the differences made by the policy assistance; Step 4—a comparison of the performance of the assisted firms with ‘typical’ firms; Step 5 a comparison with matched firms assessing ‘treated’ against ‘non-treated’ firms; and Step 6—the taking account of selection bias including self-selection as well as policy selection approaches. This is an approach that is mainly relevant to quantitative and ex post evaluations rather than to qualitative and ex ante evaluation, and the steps are ordered according to the sophistication of the procedure. The Six Steps procedure considers steps 1–3 as monitoring and steps 4–6 as evaluation procedures. The difference between the monitoring and evaluation activities is that the former relies primarily upon the views of the recipients of the policy, whereas the evaluation activities seeks to contrast these views or actions with those of non-recipients in order to present the counter-factual case against which the additionality effect can be identified.

To ensure that smart specialisation helps to deliver result-oriented and outcome-oriented policies, each of these entrepreneurship and SME-related themes and features need to be adapted and translated to both the local and regional context and also to the specific objectives of the policies. In different regions and localities, smart specialisation suggests that different priority objectives will be pursued, depending on the perceived bottlenecks and missing links, and in the case of entrepreneurship policies different indicators can be used for different objectives, related for example to social inclusion, gender, entrepreneurship education or youth enterprise (European Commission 2013a; Hempel and Fiala 2011).

5 Summary and conclusions

As a policy prioritisation framework smart specialisation signals a clear shift in EU regional policy making in that the fostering of entrepreneurship is now central to the policy priorities as is the outcome or result-orientation of the policies. Indeed, the outcome-oriented logic and result-oriented logic of the approach sets smart specialisation squarely within the new generation of policy thinking in which making the intentions and objectives of the policy as clear as possible from the outset helps to overcome institutional resistance, rent seeking and policy capture. At the same time, given the fact that SME and entrepreneurs are an important driver of the regional socio-economic system, they should be involved in the process of setting-up, implementation and evaluation of smart specialisation policies. Public authorities and policy designers and stakeholders must find suitable ways to ensure that the views of leading entrepreneurs and SME associations are not only taken into account but that these individuals and organisations become central to the whole process. In some regions the focus will tend to be on new firm start-ups, in other regions on growing the existing new firms, in others it will be on issues such as supply chain developments. Whatever is the priority it is clear that for a result-oriented policy the indicators used must well capture the levels of engagement, mobilisation and dynamism of SMEs in the entrepreneurial search processes. As Jaffe (2015) argues, when it comes to evaluating the effects of public interventions, and especially where knowledge-related and innovation-related issues are at stake, not everything can be even approximately captured by metrics. As such, a mix of quantitative and qualitative indicators is not only the best approach, but without such an approach a quantitative approach alone will produce biased results, as will a qualitative-only approach. Here, we have argued on the basis of the literature plus numerous examples of best practice from around the world that the current state of the art points exactly to this methodological mixture as being the best approach for both the monitoring and evaluation of smart specialisation interventions.

Another aspect of EU smart specialisation agenda is that it is also increasingly being incorporated into urban policy initiatives. The new EU Urban Agenda (McCann 2015) affords greater priority to those cities wishing to experiment and innovate with new technologies and urban design systems aimed at providing better responses to the societal challenges associated with enhancing wellbeing and quality of life, improving environmental quality and energy conservation, adapting to ageing and demographic change, as well as fostering entrepreneurship and innovation. These Smart Cities-types of programmes are now well established within the EU Urban Agenda. The smart specialisation policy prioritisation principles relating to the fostering of entrepreneurial and innovative activities are ideally suited for helping to best establish and design appropriate urban policies and programmes, both in wealthier or less prosperous cities (Caragliu and Del Bo 2015). Indeed, the applicability of the common policy prioritisation and evaluation principles in different contexts is a key feature of the smart specialisation concept.

Finally, while on some levels the popularity of the smart specialisation way of thinking in EU policy debates may be considered to be somewhat surprising (Foray 2015; Foray et al. 2011) to some observers, in terms of political economy there is also a logic to this take-up. In order to be effective, a policy such as regional policy necessarily involves multiple partners operating at different spatial scales and different governance levels. Finding ways to build complementarities between different policy arenas is essential and in the case of regional policy there are many arguments which suggest that it is at the local and regional levels where such complementarities can best be built (OECD 2011). Yet, mobilising different stakeholders in order to build such complementarities is a complex challenge and requires a consideration of the various incentives mechanisms operating. Therefore, in order to overcome institutional opposition and coordination problems and to better align development-related incentives, it is necessary to develop a concept or a narrative which can engage directly with a range of different actors, institutions or constituencies (Rodrik 2014). Smart specialisation has the potential to do exactly this because it derives from the insights, understanding and emerging empirical evidence from a variety of different fields spanning entrepreneurship, innovation, science policy, regional development, and economic geography (OECD 2013), as well as being results and outcome-oriented. Such a broadly based consensus on which the smart specialisation agenda builds offers the possibility to develop an overarching framework on which policy prioritisation decisions can be based in a variety of different settings, and indeed, one of the most attractive aspects of the smart specialisation approach is that it can be made to be very practical and workable (Fraunhofer ISI 2013; OECD 2013).