Measuring the benefits of open contracting: Case studies on Mexico, Paraguay, and Slovakia

Adam, I., Fazekas, M. & Tóth, B. (2020). Measuring the benefits of open contracting: Case studies on Mexico, Paraguay, and Slovakia. GTI-WP/2020:01, Budapest: Government Transparency Institute.

Enhancing the transparency of government in general and of public procurement processes in particular has been increasingly on the agenda of governments, civil societies and businesses as evidenced by initiatives such as the Open Government Partnership which has seen 70 OGP members making 189 open contracting commitments by March 2019^[1]. In spite of such major policy developments, we still lack the specific and rigorous evidence on the impact of transparency interventions and which types of transparency matter for which actors to support desirable societal outcomes such as high quality public services, procedural justice or public sector integrity (Bauhr et al, 2019).

In order to address this evidence gap, this research supported by a consortium of not-for -profit organisations (Open Contracting Partnership, HIVOS, The B Team) interested in improving procurement policy and impact, explores the short-term effects of public procurement transparency reforms on corruption risks, institutional efficiency, competition and prices by comparing procurement outcomes before and after the change in transparency regulations. It establishes a robust methodology to explore what impacts can be measured and also explores the channels through which transparency is indeed impactful – e.g. who are the stakeholders and enablers that are the actual drivers of change using increased transparency. We analyse three countries: Mexico, Paraguay and Slovakia. They were selected based on the implementation of recent open contracting reforms and the availability of procurement data for both before and after the reforms. By implication, each of these cases represent a data rich environment to start with. The transparency interventions selected for each country predominantly led to more data in a more accessible format to be published by the government for the general public, including civil society, businesses, but also government agencies themselves (Table 1). Each of these interventions were accompanied by some publicity, workshops, and trainings, hence we consider the analysis to estimate the effects of change in data publication largely on its own holding demand for data, user skills, and related environmental conditions constant.

Table 1: Intervention overview

 From a theoretical perspective, transparency interventions’ beneficial effects are conditional on two sets of factors: (i) the nature of the transparency intervention, including which public procurement phase it targets, the scope of change (i.e. the quantity and depth of new information) and whether new information is actionable for key stakeholders such as bidding firms; and (ii) the nature of demand for transparency, that is the existence of data users who are willing and able to act upon the information published. Following our theoretical framework, we also expect transparency effects to unfold over time with many effects arising through the help of stakeholders – journalists picking up stories based on newly available data, buyers learning about procurement markets or bidder finding new opportunities more efficiently.

Methodology

Using public procurement databases, our analysis compares very similar contracts awarded before vs. after the transparency interventions. Under certain assumptions, this provides an estimation of the reforms’ short-term causal effect on procurement outcomes. In order to find as similar as possible comparison groups in very diverse procurement datasets before and after the intervention, we matched contracts according to essential characteristics such as sector or contract value. To inform the quantitative analysis and fully understand the interventions in question we have also conducted desk research and in-depth interviews with key stakeholders.

A key methodological challenge was to find the right time-frame and comparable contracts before and after the transparency interventions. On the one hand, identifying causal links based on quasi-experimental settings has several prerequisites. One of them is that we should not compare too long time periods because it would risk comparing dissimilar contracts under different macro environments or other, non-transparency-related interventions may take place which impact procurement outcomes. For example, when we take a whole year of contracts after a transparency reform taking place in May, our estimation may be biased if a procurement regulatory reform happened 5 months later, such as a new reporting threshold introduced. On the other hand, we expect smaller short-term effects due to the nature of systemic transparency interventions, which take time to build and institutions change practices only slowly involving many stakeholders in a complex, highly technical area such as public procurement.

The main channels through which increased transparency affects procurement outcomes require time, for example NGOs using data for project monitoring, journalists using easily available information, public buyers understanding their markets better. We balanced these considerations for and against applying a longer time-frame by selecting one year before and after the transparency intervention which limits the incidence of confounding factors while also taking care of seasonality biases. This was a feasible strategy in Slovakia and Paraguay but due to data availability constraints, we had to use a quarterly time window in Mexico.

Moreover, our comparisons of contracts before and after the transparency intervention allow for identifying the causal effects only if the contracts from the two groups are similar in all relevant characteristics impacting outcomes. Hence, we matched contacts in the before and after groups according to key characteristics such as contract value, sector, or buyer type.

Data limitations

In all 3 countries, data quality has several important problems even after combining data from multiple sources and applying a range of data cleaning procedures. For example, in Mexico, around 20% of the contracts do not have information on the number of bids received; but the missing rate is similarly high for other variables: 23% for product codes and 29% for submission period length. In Paraguay, buyers are not legally required to record all bids they have received. According to the interviews, this leads to an under-estimation of the number of bids which can bias our effect size calculations if there is a systematic change in under-reporting from the before to after periods. In Slovakia, the data collection process is prone to error due to the several Call for tenders and Contract award publication form types that were also changed throughout the year (i.e. the same information is reported in very many different and changing formats).

Main findings

Overall, no policy relevant short-term impact, that is both statistically significant and of substantive size, of transparency interventions was identified in the 3 countries’ public procurement datasets.

While some individual effects are statistically significant in each of the 3 countries, in neither of the cases do they reveal a consistent picture of systemic impact (i.e. robust to alternative sub-samples or to the use of different indicators tapping into the same concepts). We see some  early signs of a potential impact in selected cases where a longer time window or more investment into user take-up may lead to robust, sustained, systemic change. These deserve further investigation.

Selected results are highlighted country by country below.

Mexico

• The Mexican dataset was more limited than data on the two other countries because our estimations could only be based on a very short time-window: a quarter year before and after the intervention of the open contracting portal and its transition to OCDS. Hence, the results may under or overestimate the true short term effects. This ambiguity is further amplified by the fact that federal elections took place around the same time as the intervention. Unfortunately, even if we apply the most appropriate empirical strategy for estimating the effects with before-after analysis with matching, our results are only tentative and capture very short-term effects.
• Given these qualifications, we find a 4 to 9 percentage points increase in the share of single-bidder contracts – depending on whether direct procedures are included in the sample. However, the average number of received bids increases by 0.5 bids when direct contracts are included in the sample, while we see an increase of 4.2 bids without direct contracts considered. This suggests that the intervention increased the level of competition in tenders that were already competitive, but it increased the share of high-corruption risk, single-bidder contracts in the very short-run.
• The share of non-open procedures^[2] decreases by 0 to 2 percentage points depending on whether we include direct awards in the sample. This is a somewhat more robust result as missing data is not affecting our estimations.
• Submission period length decreases by 0.5 to 0.6 day on average. Given that the average submission period length in Mexico falls in the 12-16 days band, in substantive terms this effects is rather marginal.
• Buyer’s average decision period length decreases by 1 to 2.7 days depending on whether direct awards are included in the sample. Given that average decision period lengths vary in the range of 7-20 days, these effects are considered substantive.
• Nonetheless, according to the interviews, the intervention seems to have generated awareness and interest in open contracting leading to increased usage of public procurement data by media and civil society.

Paraguay

• Among the 3 countries, Paraguay has the highest quality dataset allowing for the most robust estimation in our sample, while the likely bias in recording bidders due to buyers not being obliged to publish all bids on the Contrataciones website means that estimations related to bidder numbers should be treated with caution.
• The short-term (1-year) estimations regarding bidder numbers show substantially small and statistically weak deterioration after the launch of the new public procurement transparency portal. The share of non-competitive tenders – the ones receiving exactly the same number of bids as many companies they awarded – increased slightly from 48% to 50%, while the level of competition – the ratio of number of bids and the awarded companies per tender – decreased by 4%. These weak, albeit counterintuitive, results only hold if reporting discipline remained unchanged throughout our two years observation period. While we cannot quantitatively verify it, if anything, reporting discipline is likely to have improved hence making our estimates conservative.
• The only competition-related indicator which is not biased by the likely lack of recording losing bidders is the share of recurring winners. For this variable, we find a 5% increase, that is a deterioration, which is both substantive and statistically significant. While this result may appear counterintuitive, it is consistent with theories predicting complex, efficiency-enhancing technologies exacerbating market concentration. However, more research is needed to better understand the reasons behind this identified effect.
• The share of non-open procedures and decision period length both remained unchanged.
• The average submission period length got longer by 8 days which may be due to increased scrutiny thanks to better availability of data. Our background interview with the national procurement agency DNCP suggests that this might be driven by the introduction of an electronic complaints system in 2015 which might have pushed buyers into extending advertisement periods due to bidders’ complaints about the tendering processes.
• Nonetheless, according to our interviews, the launch of the Contrataciones portal has made contracting data more reusable and understandable for the public to some degree.

Slovakia

• In the Slovakian case, a number of other regulatory changes took place shortly after the transparency intervention of mandatory comprehensive online publishing of procurement documents. These other regulatory changes could also have an effect on the procurement market outcomes we analyse, for example the scope of public buyers was expanded and value thresholds for mandatory publication were changed shortly after the transparency intervention took place.
• To explore alternative options, we used two different samples for estimating the differences in our indicators from before and after the intervention:
  • A broad sample which only filters out atypical contracts; and
  • a narrow sample which filters out contracts that are potentially related to newly regulated buyers and that were below the original publication threshold. This considerably increases the consistency of the before-after samples at the expense of reducing sample sizes.
• Based on the broad sample, we found no significant difference in the share of single bidder contracts, however, the number of received bids per contract increased by one bid on average. Also, the use of non-open procedures decreased significantly. Relative prices (final price divided by the initially estimated price) decreased by 1.9 percentage points.
• Based on the narrow sample, we found a decrease of 19 percentage points in the share of single bidder contracts, and the number of bids per contract increased by two on average. However, the share of non-open procedures and relative prices did not change significantly.
• Underlining our concerns about the multitude of policy changes happening in a short period of time, interviews confirmed that parts of the observed impacts are probably due to changes in government staff, a host of new regulations, and an overall shift in p

Policy conclusions and lessons learnt

The lessons learnt throughout this research exercise are policy-relevant both in terms of methodology and substance.

Methodologically, data quality remains a challenge even in countries with good quality data by global standards. Importantly, public procurement data needs to be of high quality throughout the whole comparison period both in terms of its scope, the availability of data fields and the truthfulness of the recorded information. However, as a second best alternative, the data has to at least remain consistent, that is of similar scope and quality for both before and after the transparency intervention which is often problematic as transparency interventions tend to impact on publication formats, practices and effort, hence data quality.

Transparency reforms improving the scope and quality of public procurement data are highly valuable on their own as open data is as good as the data going into it; however, evaluating such reforms will remain a challenge as the reform influences the data used to measure outcomes as well as potentially the outcomes themselves. Our interventions were selected specifically to keep data quality constant. The OCDS publications made public procurement data available in a standardized format in Mexico and Paraguay, but due to regulatory deficiencies the data quality remained problematic in spite of the interventions. For example, if collecting information on the number of received bids is not mandatory, it is hard to gauge market competition.

Countering challenges of interpretation and the identification of causal impacts, subsequent research could adopt a mixed methods research strategy adding further data sources to the administrative data on public procurement. It might make sense to combine procurement data with interview evidence, document reviews and survey data. Surveying data users – for example, public buyers – could reveal whether and how open data is used, and how it could be made more useful.

Substantially, our research has revealed that increasing the amount and accessibility of data publication in public procurement is unlikely to lead to short term improvements in procurement outcomes in countries with considerable data transparency at the outset. While data quality and scope limitations imposed constraints on the effect sizes detectable, the identified small, and inconsistent changes in key outcome variables, such as bidder number or decision period length, suggest that there are no systemic, large-scale impacts. Moreover, intervening changes happening during the period after the transparency intervention in Slovakia (e.g. government change, e-auction expansion), further warn us about interpreting the significant and sizeable quantitative effects as causal impacts. Overall, it remains to be seen, using alternative methods and data sources, if there are small-scale impacts on the short term or effects for particular sub-samples (e.g. for high capacity buyers) which may build up over time in the right supporting environment; or whether more and better open data in combination with a substantial investment in promotion, skills, practice change, and data use would produce the effects predicted by theory.

Putting these findings in the light of prior research on transparency, especially transparency in public procurement, it furthers our understanding that it is not the mere availability of more data which matters rather the timely and easy availability of the right information for the right actor. For example, data on bidding opportunities matters most to bidding firms who are both motivated and able to act on it (Bauhr et al, 2019); or the reliable provision of contract performance data to local civil society groups who are also supported by law enforcement agencies (Lagunes, 2017); or where there are specific feedback and mediation channels available.

[1] https://www.opengovpartnership.org/wp-content/uploads/2019/05/Open-Contracting-Fact-Sheet-May-2019.pdf

^[2] We simply categorized everything besides the explicitly open procedures as non-open.

Full text PDF