Measuring the rule of law: A brief comparison | Dr. Pim Albers

 DR. PIM ALBERS

  Senior Visiting Fellow at UCLan Cyprus
  Independent Justice Sector Reform Expert

 

 

Introduction

When reviewing the literature about the main principles of the rule of law it must be noticed that there is a wide array of definitions available. For example, the German notion of ‘Rechtsstaat’ is primarily focused on the vertical relation between the state or government and their citizens, whilst in the common law tradition rule of law principles includes horizontal relations among citizens as well. In the most ideal situation, a comprehensive rule of law concept should include both. In addition to this, rule of law concepts should also contain the principle that laws should rule the conduct of citizens and organize/regulate the powers of government officials (government powers should be exercised through laws and officials must operate in accordance with these laws).

Besides the vertical and horizontal relationships in the Rule of Law literature a differentiation is being made between ‘thin’ and ‘thick’ concepts of the Rule of Law. Simply speaking, the thin concept on the Rule of Law is focused on the notion that the process of obtaining a justice outcome has to meet certain criteria [1]. For example, it should be public, independent and fair. This approach is not looking at the substance of laws but is more concentrated at institutions and their capacities to deliver appropriate justice. Though in the thick notion on the Rule of Law it is not sufficient to have correct procedures to deliver justice. Fair justice can be realized in this notion through the existence of rights and duties laid down in international treaties and national laws protecting human, political and social rights. In the Rule of Law Frameworks presented in this blog, both ‘thin’ and ‘thick’ concepts are being reflected. Some of the Frameworks focus more on institutions, whilst other Frameworks underlined the need for the presence of laws protecting the (human) rights of citizens.

With regards what should be included in a rule of law analytical framework differs not only from the concepts that are being followed, but on the methodologies that are being used to measure the level of rule of law in a country or a region as well. Certain institutions rely on subjective data collected through the use of perception surveys among households or experts (World Bank Governance Indicators and the Rule of Law Index of the World Justice Project), on (legal) evaluations carried out by international and European organizations, NGOs, academia (Venice  Commission Rule of Law checklist and the UN Rule of Law Indicators), or are using a combination of self-assessments, evaluations carried out by a European Institution and other relevant sources of information (Rule of Law mechanism European Union). In this blog I will discuss a selective number of Rule of Law frameworks, their methodologies for measuring the rule of law and their strengths and weaknesses.

Rule of Law Frameworks

The Worldwide Governance Indicators (WGI) developed by the World Bank is not primarily focused on measuring the Rule of Law, but on promoting good governance in countries. Therefore, certain aspects that are included in other Rule of Law frameworks are presented in separate Governance indicators, such as political stability and absence off violence/terrorism, regulatory quality and control of corruption.

Governance in the WBI study is defined as (a) the process by which governments are selected, monitored and replaced, (b) the capacity of the government to effectively formulate and implement sound policies and (c) respect of citizens and the state for institutions that govern economic and social interaction among them. Where indicators such as political stability, government effectiveness and regulatory quality are placed under the categories (a) and (b), the rule of law is situated under category (c). According to WBI rule of law is defined as: the perceptions of the extent to which agents have confidence in and abide by the rules of society, and in particular the quality of contract enforcement, property rights, the police and the courts, as well as the likelihood of crime and violence[2].

The Rule of Law Index of the World Justice Project (WJP) applies a larger concept of the Rule of Law which includes principles of good governance, that are excluded in the rule of law concept of the World Bank. In the WJP a high level of rule of law is based on four universal principles:

  • Accountability: the government as well as private actors are accountable under the law
  • Just laws: the laws are clear, publicized and stable, as well as they protect fundamental rights
  • Open government: the processes by which laws are enacted, administered and enforced are accessible, fair and efficient.
  • Accessible justice: justice is delivered timely by competent, ethical and independent representatives who are accessible, have adequate resources and reflect the makeup of communities they serve.

To measure the rule of law, based on these four principles, eight factors have been identified: constraints on government powers, absence of corruption, open government, fundamental rights, order and security, regulatory enforcement, civil justice and criminal justice. According to the viewpoint of the WJP, if countries are scoring high on these factors, this will positively contribute to economic growth, greater peace, less inequality, improved health and better education.

The United Nations Rule of Law Indicators is also based on the idea of good governance. However, the set of indicators is limited to institutions focused on the criminal law system (the police and other law enforcement agencies, the judiciary and prisons). The civil and administrative law system is excluded from this framework. To measure the rule of law of these three institutions the UN rule of law indicators

are concentrated on: performance (institutions provide effective and efficient services), integrity/transparency and accountability, treatment of members of vulnerable groups (how criminal justice institutions treat minorities, victims, juveniles, etc.) and capacity (institutions have sufficient human and material resources to perform their functions)[3].

The last two rule of law frameworks that will be discussed are the Rule of Law Framework of the Venice Commission and the EU Rule of Law monitoring mechanism. The starting point for the Venice Commission’s definition of the Rule of Law is a concept defined by Bingham[4]. In this definition the essential elements of the rule of law entails: “all persons and authorities within the state, whether public or private, should be bound by and entitled to the benefit of laws publicly made, taking effect (generally) in the future and publicly administered in the courts”. Following this definition, the Venice Commission indicated six elements of the rule of law: (1) legality (supremacy of the law), (2) legal certainty (laws must be accessible and foreseeable, rules are clear, etc.), (3) prohibition of arbitrariness, (4) access to justice before independent and impartial courts, (5) respect for human rights and (6) non-discrimination and equality before the law.

Where the WBI rule of law notion is mainly concentrated on governance, the UN rule of law indicators on criminal justice and institutions, the rule of law index covers a combination of governance principles, civil and criminal justice, the Venice Commission’s notion is strongly focused on key aspects of law and protection of human rights. What about the European Union perspective on the Rule of Law?

The Rule of Law principles of the EU[5] are primarily based on article 2 of the Treaty of the European Union and the EU Charter for Fundamental Rights. One of the preconditions to become member of the European Union is respect for the rule of law (under Article 49 TEU). Based on the decisions of the EU Court of Justice, the European Court of Human Rights, and inspired on the rule of law principles of the Venice Commission the rule of law monitoring mechanism applied in 2020 contained four pillars: justice systems, anti-corruption, media-pluralism and institutional issues related to checks and balances. The selection of the pillars is based on actual developments on EU member states and therefore have a different perspective compared to the other rule of law frameworks.

In the summary table below the main components of the Rule of Law Frameworks are being summarized for the sake of comparison. Also, coming back to the ‘thin’ and ‘thick concepts of the Rule of Law it is evident that the WBI Governance indicators, the Rule of Law Index and the UN Rule of Law indicators are mainly based on the ‘thin’ notion, since these Frameworks are primarily focused on procedures and institutions, whilst the Rule of Law checklist of the Venice Commission is predominantly oriented at the ‘thick’ notion of the Rule of Law. The EU Rule of Law monitoring mechanism though is a mixture between ‘thin’ and ‘thick’ notions of the Rule of Law. 

WB Global Governance indicators

Rule of Law Index WJB

Rule of Law checklist Venice Commission

UN Rule of Law indicators

EU Rule of Law monitoring mechanism 2020

Confidence in abide the rules of society

Constraints on government powers

legality

performance

Justice systems (independence, efficiency and quality)

quality of contract enforcement

Absence of corruption

Legal certainty

Integrity, transparency and accountability

Anti-corruption (capacity, prevention and repression)

Property rights

Open Government

Prohibition of arbitrariness

Treatment of members of vulnerable groups

Media-pluralism (transparency media, protection journalists)

Functioning of the police

Fundamental rights

Access to justice

capacity

Checks and balances institutions (process for preparing and enacting laws, independent authorities, access and judicial review, framework for civil society)

Functioning of courts

Order and Security

Respect for human rights

 

 

Likelihood of crime and violence

Regulatory enforcement

Non-discrimination and equality before the law

 

 

 

Civil Justice

 

 

 

 

Criminal Justice

 

 

 

Table 1 Rule of Law frameworks compared

 

Measurement of the Rule of Law

The various rule of law frameworks is using different methods of data collection and analysis of the results. For example, the World Bank Governance indicators is applying several external data sets ranging from survey results of the World Justice Project, Transparency International, Freedom house, World Economic Forum, etc[6]. As a result of this, the level of rule of law in countries and regions is based on intersubjective information. One of the criticisms on the WBI approach is that the methodology for measuring the rule of law is too complex. Furthermore, there is an absence of the underlying theory which defines good governance and the rule of law and it has a lack of comparability (since how the Rule of Law is scored in one country cannot be easily compared with the score on the Rule of Law in other countries). This is especially visible when the level of the rule of law is visualized through a world map. The countries in green represents a high level of rule of law, in yellow an intermediate level and in red the countries with a low level of rule of law. Since the results are based on subjective data, one can question if you can compare the perceived level of the rule of law in one country in the same manner with other countries.

In the World Justice Project a set of five questionnaires are being used to measure per country the level of the rule of law by questioning 300 local experts and general households. The results of the data have been tested on statistical reliability by one of the statistical units of the Joint Research Center of the European Commission. Similar to the WBI Rule of Law indicators, the Rule of Law Index is centered on subjective information (i.e., perceptions based on surveys among households and experts). In addition to this it must be noted that the results of the rule of law index provides a (subjective) ranking ranging from ‘poor’ rule of law countries or regions till ‘good’ rule of law countries. The Rule of Law Checklist of the Venice Commission does not prescribe a specific methodology for collecting data. The checklist makes only references to external sources of information that can be used for measurement divided into ‘hard law’ and ‘soft law’ standards. One of the major differences with WBI and the Rule of Law Index is that the Rule of Law checklist is focused on an assessment in one country, and not to compare the results with other countries. Same is the case for the UN Rule of Law Indicators. Despite the fact that the indicators are limited to criminal justice and their justice institutions, the handbook for measurement leaves it up to the evaluators which methods can be used for measurement. Options are the application of surveys, interviews, literature studies, etc.

The last methodology that will be discussed in the blog concerns the EU Rule of Law mechanism. The content of the annual rule of law reports is based on different methods of data collection. For collecting the necessary information, the commission has appointed a network of national contact points on the rule of law, who will be responsible for coordination and to provide written contributions to the report. As a part of this process the contact points must make in their contribution references to external sources of information, such as reports from the OECD, Council of Europe and United Nations Institutions (e.g., EU Justice Scoreboard, CEPEJ reports, Eurobarometer, reports from the World Bank). In addition to this, the Commission will conduct country visits to the EU member states in order to have a dialogue about the content of the rule of law reports. Furthermore, input on the rule of law reports are being provided through institutional stakeholders, such as the European Network of Councils for the Judiciary and the European Association of Judges. Despite the fact that the EU Rule of Law mechanism is based on different methodologies for data collection and is using a combination of subjective and objective data, the reports are mainly of qualitative nature. Furthermore, to avoid political sensitivity the mechanism avoids a ranking of EU member states, which is being applied by the World Justice Project and the Worldwide Governance Indicators program.

Conclusions

There are different Rule of Law frameworks available. The use of these frameworks is often limited to the specific purpose of measuring the rule of law. Some of the frameworks are focused on ranking of countries, whilst others are country specific or limited to a certain area of justice (i.e., criminal justice). Also, the methodologies applied varies from framework to framework. Certain frameworks are using only surveys, others a combination of various sources of information. To improve a high quality of the measurement of the rule of law further research is required, especially in order to define a common rule of law framework and uniform, reliable and integer methods of data collection.


1. See for example: Erwin van Veen (2017), A shotgun marriage: political contestation and the rule of law in fragile societies, Clingendael
2. Kaufmann, Kraay and Mastruzzi (2010), The Worldwide Governance Indicators: methodology and analytical issues, (policy research working paper World Bank 5430)
3. The indicators are further grouped in 25 baskets, containing rated and unrated indicators, such as: effectiveness and efficiency, public confidence, integrity and accountability, transparency, material resources, human resources, administrative and management capacity, access to justice, security/safety/order, prisoner health/welfare/rehabilitation.
4. Tom Bingham (2011), The Rule of Law, Penguin books (ISBN 9780141034539)
5. European Commission (2014), Communication from the Commission to the European Parliament and the Council: a new EU framework to strengthen the Rule of Law (COM(2014) 158 final.
6. World Bank, Rule of Law Dataset: https://info.worldbank.org/governance/wgi/Home/Documents