An assessment view to evaluate whether Spatial Data Infrastructures meet their goals

https://doi.org/10.1016/j.compenvurbsys.2010.09.004Get rights and content

Abstract

The motives for constructing Spatial Data Infrastructures (SDIs) are often based on their anticipated benefits for society, economy, and environment. According to those widely articulated but rarely proven benefits, SDI coordinators have been defining more specific objectives to be achieved by their SDIs. However, there is a limited number of assessment approaches that are able to demonstrate whether SDIs indeed realize the intended goals. In this article we develop, apply and evaluate an assessment view for evaluating the extent to which SDIs realize their goals. The assessment view has been developed stepwise using the Multi-view SDI assessment framework as a guideline. The application of the proposed view in the Dutch SDI demonstrates its potential. In addition, the evaluation of the proposed view by the potential users confirms to a certain extent its usability. The results also show that the ease of determining assessment indicators depends on the precision with which the SDI goals are formulated.

Research highlights

► As a result of the research, a SDI goal-oriented assessment view has been proposed. ► The practical applicability of the proposed assessment view has been demonstrated by its implementation in the Dutch SDI case. ► The presented goal-oriented SDI assessment approach offers a specific and rational assessment which helps to evaluate SDIs for accountability purposes. ► The design of the assessment view is generic, so it can be used to measure the extent of goals realization of any infrastructures with clearly defined and agreeable goals and where all of the stakeholders can be identified and approached.

Introduction

Since the beginning of the 1990s, many local governments, countries, and regions have been building Spatial Data Infrastructures (SDIs) (Crompvoets and Bregt, 2007, Crompvoets et al., 2004, Masser, 1999, Masser, 2005, Masser, 2007, Onsrud, 1998, Onsrud, 2007, Rajabifard, 2002, Rajabifard et al., 2003). The aim of SDIs is facilitate the exchange and sharing of spatial data between stakeholders in the spatial data community (Crompvoets, Rajabifard, van Loenen, & Delgado Fernandez, 2008). SDI is also about coordinating spatial data assets. The establishment and management of SDIs is in many cases coordinated by mapping agencies or political or administrative organizations. These organizations integrate standards, data access facilities, policies and technologies to enable the spatial data exchange within SDIs. The proponents of SDIs articulate many dimensions of goals and benefits that SDIs can realize and bring to the society, for example: economic (e.g., reduced costs of data production), technical (e.g., improved development of applications), social (e.g., better management and decision making) and environmental (e.g., the integration of spatial information and knowledge from different sectors for solving environmental issues). SDIs also have the potential to spatially enable governments, i.e. providing better service to decision-makers, politicians and societies by using spatial concepts and technology (Bennett, 2009, Masser, 2007, Masser et al., 2007, Rajabifard et al., 2003, Rajabifard, 2007). The goals for an SDI can also be more specific. The SDI policies written for a specific jurisdiction (e.g., province, country or region) define specifically what the SDI being designed and developed aims to achieve. For example, the specific goal of the INSPIRE directive is to create an SDI that would assist policy makers in their activities affecting the environment (European Commission., 2007). Given this wide array of intended but rarely proven SDI benefits and goals, it is natural for policy makers, government representatives and the public to be interested in the assessment studies measuring the benefits of SDIs and the level of realization of the goals.

The Dutch SDI implementation provides an example of governmental demand for monitoring SDI goals. Since 2008, the Dutch SDI is being constructed by implementing the vision and strategic plan called GIDEON (VROM., 2008). GIDEON establishes four goals that need to be realized by 2011. In 2009, the Ministry of Housing, Spatial Planning and the Environment (VROM), which is responsible for GIDEON implementation, requested monitoring of the extent to which the four GIDEON goals have been realized. However, a “ready-to-use” approach to monitor the realization of SDI goals, which could also be applied to monitor the realization of the GIDEON goals, does not exist.

An extensive body of literature on SDI assessment (or its components) already exists. Many SDI assessment views have already been proposed (Crompvoets et al., 2008, Delgado Fernandez et al., 2005, Georgiadou et al., 2006, Kok and van Loenen, 2005, Lance et al., 2006, Masser, 1999, Onsrud, 1998, Rodriguez-Pabon, 2005, SADL, 2005). The majority of them were developed within the SDI scientific community. The character of these studies was rather intuitive and curiosity-driven. Their aim was to explore and build knowledge about the performance and benefits of SDI. These studies were natural in the early stage of SDI development when knowledge about SDI was limited. However, in recent years, a shift from an intuitive to more rational SDI assessments can be observed (Bregt, Grus, Crompvoets, Castelein, & Meerkerk, 2008). The increasing demand for rational assessments requires generic assessment approaches that measure, for example, the extent to which SDIs programs meet their objectives (Lance et al., 2009, Lance et al., 2006). There is also a growing awareness across governments and communities of practitioners that much more attention needs to be paid to assessing the social and economic impacts of SDIs. These impacts need to be articulated when a significant number of such infrastructures have already been established (Craglia & Nowak, 2006). Up until now, SDI impact studies have mainly had an ex-ante character, i.e., focused on predicted SDI impacts and benefits (Craglia, 2003, Craglia et al., 2008, Dufourmont, 2004). Ex-post studies of SDI benefits and impacts are still rare (Lance et al., 2006) and only a few theoretical considerations and best practices in this matter can be mentioned (see e.g. Castelein et al., 2010, Craglia and Campagna, 2010, European Commission, 2009, Genovese et al., 2009, Geoconnections, 2008).

As a response to the growing need for SDI assessment, a Multi-view SDI assessment framework has been proposed (Crompvoets et al., 2008, Grus et al., 2007). The core of the Multi-view SDI Assessment Framework is represented by the multiple assessment views that can be classified in terms of (1) assessing different SDI aspects e.g., organizational, technical, etc., and (2) assessing SDIs for different purposes, i.e., knowledge, development and accountability. The theoretical reasoning for the Multi-view SDI Assessment Framework originates from the principles of assessing complex systems where the use of multiple assessment methods simultaneously provides more holistic and less biased assessment result of complex phenomena. As SDIs can be viewed as Complex Adaptive Systems (CAS), the principles of assessing CAS can also be applied in assessing SDIs (Grus, Crompvoets, & Bregt, 2010). The Multi-view SDI Assessment Framework by combining multiple SDI assessment views, which have been developed for different purposes and for different users, can provide a comprehensive and realistic SDI assessment to satisfy many SDI stakeholders (Grus, Crompvoets, Bregt, van Loenen, & Delgado Fernandez, 2008). However, there is a growing demand, especially from policy makers and politicians, for more rational SDI assessments which could provide evidence if the SDI intended goals has or has not been met. According to the authors of this article, there is currently no SDI assessment approach that could answer this specific demand for SDI goals monitoring.

In this paper, the authors develop and present a generic assessment view to assess the extent to which SDIs realize their goals. The view is implemented in the Dutch SDI and evaluated by the potential users.

This article is structured as follows. Section 2 presents the methodology used to develop the assessment view. Section 3 presents the developed SDI goal-oriented assessment view. Section 4 discusses the assessment view implementation in the Dutch case study. Section 5 presents and discusses the results of users’ evaluation of the developed assessment view. Section 6 closes the paper with conclusions and recommendations for the further research.

Section snippets

Methodology

To develop a SDI goal-oriented assessment view, the authors first reviewed the existing evaluation models. Hansen (2005) presented a typology and classification of several evaluation models. They differ in the questions that they aim to answer and the evaluation criteria they use (see Table 1). From the overview of these evaluation models and the questions they aim to answer it is evident that the “goal-attainment model” model best fits the objective of developing a goal-oriented SDI assessment

A goal-oriented assessment view

Fig. 2 presents a schematic picture of the proposed assessment view for assessing the extent to which SDI realizes its goals. The view is divided into four phases, according to which it is developed.

Phase one focuses on the purpose of the assessment. According to Hansen (2005), the purpose of the “goal-attainment model” is to answer the question: to what degree have the goals been realized? This assessment purpose can be classified as accountability (Chelimsky, 1997, Grus et al., 2007).

Phase

Case study

To demonstrate how the developed view is implemented in a real-world case, it was applied to measure the extent to which the goals of the Dutch Spatial Data Infrastructure (GIDEON) are realized. This section is divided into two parts. The first part describes the GIDEON strategy and the plan to develop a Dutch SDI. The second part discusses the implementation of the developed view in the GIDEON case.

Evaluation of the goal-oriented assessment view

To evaluate the proposed assessment view the authors used the meta-evaluation standard criteria for evaluating assessment frameworks (Shepard, 1977, Stufflebeam, 1974). The evaluation standard comprises three groups of criteria: technical adequacy, utility, and efficiency (see Appendix B). The evaluation was conducted by asking the group of respondents to fill out a questionnaire containing standard meta-evaluation statements (see Appendix C). The statements were formulated in such a way that

Conclusions

The objective of this article was to develop a goal-oriented assessment view approach for assessing the realization of SDI goals. The methodology was proposed to realize this objective. The proposed assessment view was developed according to the four phases of the Multi-view SDI assessment framework. As a result, a SDI goal-oriented assessment view has been proposed.

The practical applicability of the proposed assessment view has been demonstrated by its implementation in the Dutch SDI case. All

Acknowledgements

We would like to acknowledge and thank the participants of two workshops on monitoring the GIDEON goals held in Wageningen on October 21, 2008 and October 14, 2009, for their active participation and contribution to the presented assessment view design. We would like to thank the anonymous reviewers of this paper for their valuable remarks and suggestions that helped to improve the paper. We also thank the Dutch ‘Space for Geo-Information’ (RGI) innovation program for providing the necessary

References (46)

  • G. Giff et al.

    Performance Indicators as a tool to support Spatial Data Infrastructures assessment

    Computers, Environment and Urban Systems

    (2008)
  • B. Kok et al.

    How to assess the success of National Spatial Data Infrastructures?

    Computers, Environment and Urban Systems

    (2005)
  • Bennett, R. M., & Rajabifard, A. (2009). Spatially enabling government: A snapshot from Victoria. In Proceedings of a...
  • Bregt, A. K., Castelein, W., Pluimers, Y., & van Leeuwen, W. S. (2009). Geo-sector in kaart. Marktmonitor Nederlandse...
  • A.K. Bregt et al.

    Changing demands for Spatial Data Infrastructure assessment: Experience from The Netherlands

  • T.W. Castelein et al.

    The economic value of the Dutch geo-information sector

    International Journal of Spatial Data Infrastructures Research

    (2010)
  • E. Chelimsky

    The coming transformations in evaluation

  • Craglia, M. (2003). Contribution towards the extended impact assessment of INSPIRE. Environment Agency for England and...
  • M. Craglia et al.

    Advanced regional SDIs in Europe: Comparative cost-benefit evaluation and impact assessment perspectives

    International Journal of Spatial Data Infrastructures Research

    (2010)
  • M. Craglia et al.

    The socio-economic impact of the spatial data infrastructure of catalonia

    (2008)
  • M. Craglia et al.

    Report of the international workshop on spatial data infrastructures cost-benefit/return on investment. 12–13 January 2006

    (2006)
  • J. Crompvoets et al.

    National spatial data clearinghouses, 2000–2005

  • J. Crompvoets et al.

    Assessing the worldwide developments of national spatial data clearinghouses

    International Journal of Geographical Information Science

    (2004)
  • J. Crompvoets et al.

    A multi-view framework to assess spatial data infrastructures

    (2008)
  • Delgado Fernandez, T., Lance, K., Buck, M., & Onsrud, H. J. (2005). Assessing SDI readiness index. In Proceedings from...
  • Dufourmont, H. (2004). Results Task Force XIA, Eurostat, Luxembourg....
  • European Commission. (2007). Directive 2007/2/EC of the European Parliament and of the Council of 14 March 2007...
  • European Commission. (2009). Commission decision of 5 June 2009 implementing Directive 2007/2/EC of the European...
  • E. Genovese et al.

    Evaluating the socio-economic impact of Geographic Information: A classification of the literature

    International Journal of Spatial Data Infrastructures Research

    (2009)
  • Geoconnections. (2008). GeoConnections Annual Report 2006–2007....
  • Geonovum. (2009). Tussenrapportage voortgang GIDEON. Amersfoort: Geonovum. <http://www.geonovum.nl/dossiers/gideon>...
  • Y. Georgiadou et al.

    SDI and e-Governance: A quest for appropriate evaluation approaches

    URISA Journal

    (2006)
  • P.Y. Georgiadou et al.

    Studying the use of geo-information in government: A conceptual framework

    Computers, Environment and Urban Systems

    (2010)
  • Cited by (0)

    View full text