Qualitative monitoring of information infrastructures: A case study

Since the first images of hunting grounds were painted on cave walls, mankind has recorded information vital for human survival. The Greeks drew maps to help them understand the Earth’s geography, and for the Romans, maps were vital to the administration of their vast empire. In the 16th century, Mercator devised a flat representation of the globe which helped navigators sail to the four corners of the earth, and 300 years later John Snow drew his cholera map of London which showed the relationship between location and the incidence of disease.

We have recorded geographical information for millennia, but the pace of development in the digital age has outstripped everything that has gone before. ‘Geographic information systems’ (GIS), that is computerised systems for managing, integrating, analysing and displaying location-specific or ‘spatial’ data, now support many functions on which our lives depend. From the despatch of emergency services to the location of utilities, from the management of pollution incidents to projections about climate change, and from environmental monitoring to population censuses, these systems have become vital to the effective functioning of communities around the world.

Because of the sheer volume of data and its complexity in terms of things like ownership and location, as well as management and transferability, governments have sought to create infrastructures to facilitate access to, and management of, spatial data since the early 1990s.

‘Spatial data infrastructures’ (SDIs) bring together the technology, policies, standards, resources, and other activities which are needed to acquire, process, distribute, use, maintain, and preserve spatial data in a manner which maximises overall usage.

Ian Masser and Joep Crompvoets of the Public Governance Institute in Leuven, Belgium, are experts in the field. Their most recent work looks at how a European research and implementation programme provides a case study for qualitative monitoring of information infrastructures.

INSPIRE
In 2007 the European Council and the European Parliament agreed a Directive to establish an ‘Infrastructure for Spatial Information in the European Community’. Known as INSPIRE, the project to implement the agreement aims to develop common standards for describing, collecting and sharing spatial data to support environmental policies in all Member States. Given that environmental incidents do not respect state borders, it is hoped that such coordination will help governments to work together and take more informed and timely decisions, for example following natural disasters such as major flooding or pollution incidents.

Since the first images of hunting grounds were painted on cave walls, mankind has recorded information vital for human survival.

INSPIRE is an ambitious project to create a large and complex multinational SDI in the 28 Member States by 2020. Monitoring is an important part of the project. Countries have to report on quantitative performance indicators annually and provide qualitative feedback in three-yearly written reports.

The qualitative reports are compiled according to a template which covers five measures: coordination and quality assurance; functioning and coordination of the infrastructure; usage of the infrastructure for spatial information; data sharing; and the costs and benefits of implementation. Two sets of reports were published in 2010 and 2013. Because these reports have all been translated into English, Masser and Crompvoets argue that they present a unique data set which is particularly suitable for evaluation.

2010 reports
Masser and Crompvoets found that the 2010 reports varied significantly in detail. Length does not imply quality, but, for example, Portugal submitted an 88-page report whereas Poland, the UK and Ireland submitted reports around a dozen pages long. All countries reported that they were in the process of creating a structure to coordinate implementation of INSPIRE and identify stakeholders. In most cases the work was being led by the government department responsible for the environment or the country’s mapping agency.

Countries differed in whether work included regional or local governments, though this largely depended on whether spatial data management was devolved to regional level. Countries also differed in whether private sector representatives were included in the national coordinating bodies. There was little comment about usage and data sharing, as many countries said it was too early to provide useful information. Similarly, though most countries provided some information about costs and benefits, few commented in detail.

2013 reports
The 2013 reports were longer than those submitted in the first round, with an average 39 pages compared to 26 previously. This time the shortest reports were from Luxembourg and Ireland, with 15 and 20 pages respectively. As in 2010, responses to questions about coordination and functioning were more detailed than those about data usage, data sharing and the costs and benefits of implementation. Data sharing was commented on most by Belgium and Spain and least by the Czech Republic, Hungary, Italy, Malta and Slovenia. Belgium, Greece and Sweden had the most to say about costs and benefits.

There were changes from countries’ earlier positions. For example, environmental ministries were now responsible for implementing INSPIRE in most countries. However, countries still showed considerable variation in whether regional and other stakeholders were also involved. Most Member States reported that they had created some form of national web portal or ‘geoportal’ for users to access geographic information. Some countries also reported a dramatic growth in spatial data infrastructure usage. For example, services in Catalonia in Spain were reported to be accessed more than a billion times a year between 2010 and 2012.

Though some countries were developing open data structures for government information to promote data sharing, most reported limited progress on this topic. Similarly, though the 2013 reports included more information on costs and benefits than in 2010, most Member States still found this difficult to assess, given that implementation is ongoing.

Issues arising
Masser and Crompvoets believe that the outlook for INSPIRE is positive and argue that their analysis of the INSPIRE reports shows that key issues can be usefully evaluated by qualitative monitoring.

INSPIRE has been an important factor in building up a European wide community that has been instrumental in translating SDI concepts into practice.

They said: “The overall impression given by the country reports submitted to the EC in 2013 is that implementation of the INSPIRE Directive is well underway and that the necessary steps are being taken by Member States to create an operational European wide environmental spatial data infrastructures (SDI).” These findings were subsequently confirmed in the Commission’s own mid-term evaluation of the INSPIRE Directive.

Masser and Crompvoets believe there is some room for improvement in implementation of INSPIRE. For example, they found that while most countries report significant increase in the use of data and services, little overall progress has been made in promoting data sharing. They are also concerned about the diversity in different countries’ approach to the involvement of public authorities and third parties. For example, while some see implementation of a European SDI as a task for central government alone, Finland has involved 350 experts from 120 organisations in its deliberations.

The impact of language on programme implementation, “the politics of representation”, is an area of potential future research identified by Masser and Crompvoets. English is the dominant language for the programme and the first two rounds of reports were all translated into English. However, translation is costly and time-consuming, and it did not make all reports accessible to participants who only speak their native language. The 2016 reports were not translated into English because of pressures on time.
Masser and Crompvoets recommend a socio-technical approach to help the programme’s future implementation, that is an approach which looks at the interaction between people and technology. They said: “This has been an important factor in the building up of a European-wide SDI community which has been instrumental in translating SDI concepts into practice.”

Effective evaluation
In a world that tends to judge scientific research by hard facts that easily translate into soundbites, qualitative monitoring is less frequently used to evaluate projects than quantitative assessment.

While noting that self-evaluation of INSPIRE’s country reports has some drawbacks in that it risks presenting an over-optimistic view of national circumstances, Masser and Crompvoets believe that the programme shows how effective qualitative monitoring can be.

The researchers conclude: “The great strength of the self-evaluation approach is that it places the onus for reporting on those organisations that are directly engaged in the implementation process. These groups are in a position to draw upon the combined body of knowledge and experience of the national communities in preparing their reports.”