Synthesizing and standardizing criteria for the evaluation of sustainability indicators in the water sector

Abstract

Indicators are one of the tools available in planning and management projects that aid in the decision-making process and the monitoring of those decisions on the path to sustainable use and management of water and natural resources. However, the quality and trustworthiness of the indicators depend on the constant improvement in the means to assess and design criteria sets. The identification of criteria to evaluate indicators and its subsequence selection are not an ordinary task. The research identified a proliferation of unconsolidated criteria in use in the sustainability and water resource management domains. In response, a process of synthesis and consolidation was undertaken in order to reduce the level of redundancies and to identify possible candidates for “core criteria” that are identified as being a relevant part of most evaluation frameworks. A representative collection of sources from the specialized literature was screened for evaluation criteria. Altogether, 74 sources were assessed, comprising 346 mentions of criteria applied for indicator assessment. A detailed synthesis was performed to organize the criteria and identify possible redundancies. The analysis allowed a reduction from the 346 initial criteria to 60 unique criteria. The study offers a standard title and description for each criterion, contributing to improve clarity and avoid ambiguity. The criteria were also ranked to identify which criteria were in more systemic use. Of the 60 criteria found, the 12 most cited were identified as possible core criteria for framework development. Also, in order to facilitate the design of indicator sets, all 60 criteria were divided into two approaches (scientific/top-down or end-use/bottom-up). This study identified significant redundancies and a lack of standardization in the use of criteria, and it also ranked criteria to facilitate multi-method framework development. Thus, it is essential that indicator designers not only consider criteria that have some level of standardization to be able to compare and communicate with other agencies and communities but also consider how to utilize core criteria in the design of indicator sets.

Appendices

Appendix 1: Scientific criteria to assess indicators

Citations*	Name and definition of the criteria	Sources
19	Scientific foundation—“The extent to which an indicator is based on currently sound and internationally accepted theoretical, conceptual, technical, and scientific standards and principles (adapted from UNEP 2006)	Niemeijer and de Groot (2008) (4x)**, WWAP (2006), Bockstaller and Girardin (2003), Bockstaller et al. (2008), Aveline et al. (2009), Cloquell-Ballester et al. (2006), Parris and Kates (2003), Clark and Dickson (1999), UNEP (2006), SNZ (2002), UN (2007), BNIA (2006), WHO (2002), OECD (2003), FAO (1999), World Bank (2000)
18	Reliability—“The extent to which an experiment, test, or measuring procedure yields the same results on repeated trials” (Webster’s Dictionary)	Niemeijer and de Groot (2008) (2x)**, Bringhenti et al. (2011), Cloquell-Ballester et al. (2006), UNEP (2006), SNZ (2002), ITFM (1995), BNIA (2006), Kurka and Blackwood (2013), Rovere et al. (2010), Graymore et al. (2009), Buchholz et al. (2009), Singh et al. (2009), Wang et al. (2009), Bélanger et al. (2012), Segnestam (2002), OECD (2003), World Bank (2000), Gudmundsson (2010)
15	Measurability—“The extent to which the proposed measurement procedures to obtain the indicator adopts standardized methods” (adapted from Cloquell-Ballester et al. 2006)	Niemeijer and de Groot (2008) (4x)**, WWAP (2006), IISD (2008), Cloquell-Ballester et al. (2006), Prescott-Allen (2001), USEPA (2000), UNEP (2006), SNZ (2002), ITFM (1995), BNIA (2006), WHO (2002), World Bank (2000)
13	Sensitivity—“The extent to which a small change in the factor measured should result in a measurable change in the indicator” (adapted from WWAP 2006)	Niemeijer and de Groot (2008), WWAP (2006), Aveline et al. (2009), Bockstaller et al.(2008), IISD (2008), Cloquell-Ballester et al. (2006), SNZ (2002), ITFM (1995), Bélanger et al. (2012), WHO (2002), OECD (2003), World Bank. (2000), Gudmundsson (2010)
10	Accuracy—“The extend to which the result of a measurement, or of an indicator conforms to the correct value” (adapted Oxford Dictionary of English 2014)	Bockstaller and Girardin (2003), Bockstaller et al. (2008), Aveline et al. (2009), ITFM (1995), Bringhenti et al. (2011), Cloquell-Ballester et al. (2006), Meul et al. (2009), OECD (2003), FAO (1999), Gudmundsson (2010)
9	Specificity—“Clearly and unambiguously defined” (Niemeijer and de Groot 2008)	Niemeijer and de Groot (2008) (4x)**, WWAP (2006), UNEP (2006), SNZ (2002), Segnestam (2002), World Bank. (2000)
8	Time-bound—“Measure changes on an appropriate temporal scale” (SNZ 2002)	Niemeijer and de Groot (2008) (4x)**, IISD (2008), US EPA (2000), SNZ (2002), ITFM (1995)
8	Representativeness—”Related to a specific question or issue of concern and representative of the conditions in question” (WHO (2002))	Niemeijer and de Groot (2008), WWAP (2006), Bringhenti et al. (2011), ITFM (1995), WHO (2002), Prescott-Allen 2001, SNZ (2002), World Bank (2000)
6	Data quality—“The data used to establish the indicator are adequately documented and of known quality” (adapted from OECD 2003)	US EPA (2000), Segnestam (2002), OECD (2003), FAO (1999), World Bank (2000), Cloquell-Ballester et al. (2006)
5	Space-bound—“Adopt an appropriate geographical scope” (IISD 2008)	Niemeijer and de Groot (2008), IISD (2008), US EPA (2000), SNZ (2002), ITFM (1995)
5	Anticipatory—“Provides an early warning of changes” (ITFM 1995)	Niemeijer and de Groot (2008), WWAP (2006), ITFM (1995), WHO (2002), World Bank (2000)
5	Spatial and temporal scales of applicability—“Provide information at the appropriate spatial and temporal scales” (Niemeijer and de Groot 2008)	Niemeijer and de Groot (2008) (2x)**, Segnestam (2002), ITFM (1995), World Bank (2000)
4	Robustness—“Be relatively insensitive to expected sources of interference” (Niemeijer and de Groot 2008)	Niemeijer and de Groot (2008), WWAP (2006), WHO (2002), FAO (1999)
2	Predictability—“Respond in a predictable manner to changes and stresses” (Niemeijer and de Groot 2008)	Niemeijer and de Groot (2008), SNZ (2002)
2	Universality—“Applicable to many areas situations and scales” (Niemeijer and de Groot 2008)	Niemeijer and de Groot (2008), WWAP (2006)
2	Discriminatory—“Ability to discriminate differences separating extraneous variability (US EPA, 2000)	US EPA (2000), ITFM (1995)
2	Uncertainty—“Detailed with regards to uncertainties and limitations” (WWAP (2006))	Niemeijer and de Groot (2008), WWAP (2006)
1	Portability—“Be repeatable and reproducible in different contexts”	Niemeijer and de Groot (2008)
1	“Specific for a certain stress or effect”	WWAP (2006)
1	“General importance—Beer on a fundaments process or widespread change”	Niemeijer and de Groot (2008)
1	“Formulation—The mathematical formulation of the indicator is suitable with regard to the concept which is to be quantified”	Cloquell-Ballester et al. (2006)
1	“Transformable—intelligent”	WWAP (2006)
1	“Estimation of measurement error—must be estimated and reported”	US EPA (2000)
1	“Integrates effects/exposure—Integrates effects or exposure over time and space”	ITFM (1995)
1	“Focus on causes not symptoms”	BNIA (2006)

1. Total: 141 mentions/24 scientific criteria
2. *Number of mentions of the criterion under analyses
3. ** Number of citations on the meta-review performed by Niemeijer and de Groot (2008)

Appendix 2: End-use criteria to assess indicators

Citations*	Name and definition of the criteria	Sources
31	Data availability—“The extent which the data required for the indicator is easy or possible to get at a reasonable cost” (adapted from Merriam-Webster Dictionary of English 2014 and OECD 2003)	Niemeijer and de Groot (2008) (3x)**, WWAP (2006), Bringhenti et al. (2011), BNIA (2006), Cloquell-Ballester et al. (2006), Prescott-Allen 2001, UNEP (2006), SNZ (2002), Segnestam (2002), ITFM (1995), UN (2007), Kurka and Blackwood (2013), Lattimore et al. (2009), Vera and Langlois (2007), Fraser et al. (2006), Olsthoorn et al. 2001, Rovere et al. (2010), Shmelev and Rodríguez-Labajos (2009), Graymore et al. (2009), Buchholz et al. (2009), Singh et al. (2009), Wang et al. (2009), Butler et al. (2003), Bélanger et al. (2012), WHO (2002), OECD (2003), FAO (1999), World Bank (2000), Gudmundsson (2010)
25	Relevance—“The extent which an indicator is related or connected to the matter in hand” (adapted from Merriam-Webster Dictionary of English 2014)	Niemeijer and de Groot (2008) (4x)**, Cloquell-Ballester et al. (2006), Parris and Kates (2003), Clark and Dickson (1999), US EPA (2000), UNEP (2006), SNZ (2002), ITFM (1995), UN (2007), BNIA (2006), Kurka and Blackwood (2013), Lattimore et al. (2009), Rovere et al. (2010): Graymore et al. (2009), Gilmour et al. (2007), Buchholz et al. (2009), Wang et al. (2009), Doukas et al. (2007), Baker et al. (2002), Butler et al. (2003), WHO (2002), World Bank (2000)
21	Comprehensibility—“The extent which the indicator is able to be understood by the target audience” (adapted from Oxford Dictionary of English 2014)	Niemeijer and de Groot (2008) (2x)**, WWAP (2006), Aveline et al. (2009), Bockstaller et al.(2008), IISD (2008), Cloquell-Ballester et al. (2006), UNEP (2006), SNZ (2002), ITFM (1995), UN (2007), BNIA (2006), Kurka and Blackwood (2013), Fraser et al. (2006), Singh et al. (2009), Butler et al. (2003), Bélanger et al. (2012), WHO (2002), OECD (2003), FAO (1999), Gudmundsson (2010)
13	Usefulness—“User-driven to be relevant to target audience” (Niemeijer and de Groot 2008)	Niemeijer and de Groot (2008), WWAP (2006), Bockstaller and Girardin (2003), Bockstaller et al. (2008), Aveline et al. (2009), IISD (2008), Meul et al. (2009), Segnestam (2002), BNIA (2006), Bélanger et al. 2012, WHO (2002), FAO (1999), World Bank (2000)
10	Target-oriented—“Have a threshold and/or target against which to compare the indicator” (adapted from OECD 2003)	Niemeijer and de Groot (2008), WWAP (2006), IISD (2008), US EPA (2000), UNEP (2006), SNZ (2002), ITFM (1995), BNIA (2006), OECD (2003), Gudmundsson (2010)
10	Operational simplicity “Simple to measure, manage and analyze” (Niemeijer and de Groot 2008)	Niemeijer and de Groot (2008) (2x)**, Aveline et al. (2009), Bockstaller et al. (2008), SNZ (2002), ITFM (1995), UN (2007), FAO (1999), World Bank (2000), Gudmundsson (2010)
10	Compatibility—“Be compatible with indicators developed and used in other regions” (Niemeijer and de Groot 2008)	Niemeijer and de Groot (2008) (2x)**, WWAP (2006), UNEP (2006), UN (2007), Kurka and Blackwood (2013), Rovere et al. (2010), Wang et al. (2009), Doukas et al. (2007), OECD (2003)
9	Linkage to management action—“Provide information to support a management decision or to quantify the success of past decisions” (US EPA, 2000)	Niemeijer and de Groot (2008) (3x)**, WWAP (2006), US EPA (2000), UNEP (2006), SNZ (2002), FAO (1999), Gudmundsson (2010)
9	Retrospectivity—“Able to show trends over time” (OECD 2003)	Niemeijer and de Groot (2008) (2x)**, WWAP (2006), SNZ (2002), IISD (2008), ITFM (1995), OECD (2003), FAO (1999), FAO (2000)
9	Resource demand—“Logistical requirements (personnel, equipment, training) are reasonable” (US EPA 2000)	Niemeijer and de Groot (2008) (5x)**, IISD (2008), US EPA (2000), UNEP (2006), UN (2007)
7	Sustainability—“Consider the underlying social, economic and environmental system as a whole, including issues related to governance and the interactions among its components” (IISD 2008)	Niemeijer and de Groot (2008), IISD (2008), SNZ (2002), UN (2007), BNIA (2006), Bélanger et al. 2012, WHO (2002)
7	Cost-effectiveness—“Benefits of the information provided by the indicators should outweigh the cost of usage” (Niemeijer and de Groot 2008)	Niemeijer and de Groot (2008), Cloquell-Ballester et al. (2006), US EPA (2000), Segnestam (2002), ITFM (1995), FAO (1999), World Bank (2000)
5	Participatory—“Developed with the palpation of a broad range of stakeholders to ensure the indicators: encompass community visions and values, and promote ownership” (UNEP 2006)	IISD (2008), Parris and Kates (2003), Clark and Dickson (1999), WWAP (2006), UNEP (2006)
5	Causal links—“Cause–effect chain has to be known to enable tackling of the problem” (WWAP 2006)	WWAP (2006), IISD (2008), SNZ (2002), OECD (2003), World Bank (2000)
5	Individuality—“Are the indicators independent enough or do they duplicate other C and I?’ (Kurka and Blackwood 2013)	Kurka and Blackwood (2013), Rovere et al. (2010), Graymore et al. (2009), Wang et al. (2009), Doukas et al. (2007)
3	Transparency—“ensure it is accessible to the public; explain the underlined choices assumptions and uncertainties; disclose data sources and methods and disclose all sources of funding and potential conflicts of interest.” (IISD 2008)	IISD (2008), Gudmundsson (2010), UNEP (2006)
3	Flexibility—“Are flexible, so new information can lead to adjustments in the indicator” (UNEP 2006)	Aveline et al. (2009), Bockstaller et al. (2008), UNEP (2006)
2	Linked to models, forecasting and information systems	WWAP (2006), OECD (2003)
2	Conceptual framework—“Be developed within an agreed-upon conceptual and operational framework” (WWAP 2006)	IISD (2008), WWAP (2006)
2	National in scope—“Be either national in scope or applicable to regional environmental issues of national significance” (OECD 2003)	UN (2007), OECD (2003)
2	Ecological function—“Conceptually linked to ecological function of concern” (USEPA 2000)	US EPA (2000), Segnestam (2002)
2	Pedagogy—Educational am “helps to make the factors understandable to the pubic” (Aveline et al. 2009)	Aveline et al. (2009), Bockstaller et al. (2008)
1	Quantified—“Information should be quantified in such a way that it is significant apparent”	Niemeijer and de Groot (2008)
1	Policy Changes—“Recording ether changes in the means recommended by policy or changes in the development impact attributable to policy “	WWAP (2006)
1	Guiding vision—be guided by the goal of delivering well being within the capacity of the biosphere to sustain it for future generations	IISD (2008)
1	Boundaries—Takes into consideration risks uncertainties and activities that can have an impact across boundaries	IISD (2008)
1	Continuous learning and improvement	IISD (2008)
1	Ethical concerns—An indicator must comply with fundamental human rights and must require only data that are consistent with moral beliefs or values of the population	Gudmundsson (2010)
1	Information management—requirements for data analysis storage, processing documentation and retrieval are feasible	US EPA (2000)
1	Quality assurance—degree of validity of the steps in collation and computation of data aiming to assure the quality of the indicator	US EPA (2000)
1	Program coverage—Program uses a suite of indicators that encompass major components of the ecosystem over the range of environmental conditions that can be expected	ITFM (1995)
1	Relate to the whole community	BNIA (2006)
1	Focus on resources and assets (framed in a positive way/focus on problems or assets)	BNIA (2006)
1	Adapted to the objectives—Does the indicator meet the objectives?	Bélanger et al. (2012)
1	Formal (legal) foundation	FAO (1999)

1. Total: 205 mentions/25 end-use criteria
2. *Number of mentions of the criterion under analyses
3. **Number of citations on the meta-review performed by Niemeijer and de Groot (2008)