Evaluating public participation in policymaking. Learning from narratives.

Roberto Falanga and João Ferrão, Instituto de Ciências Sociais, Universidade de Lisboa

Why this post?

The search for new mechanisms of public participation in policymaking has become a case in point in the international debate. In the last two decades, the wide range of participatory initiatives implemented on different scales has provided a vibrant mixture of aims, methods, and public policy domains. Against the growing impact of public participation in political life, some scholars have pinpointed a variety of risks related to the diffusion of buzzwords (Cornwall, 2008) and the “discursive crisis” casting a shadow over differences between opposing political views (Dagnino, 2004). While the larger narrative acts as a burden on the boundaries of what constitutes public participation and what it should be for, political representatives continue to experiment with forms of the engagement of people, with little commitment to providing evidence of their success. As such, the debate on public participation has focused largely on issues of techniques and methods, risking the depoliticising of the understanding of goals and instruments for social inclusion. Accordingly, not only has little attention has been paid to evaluation methods by scholars, but also limited evidence of evaluation has been reported by sponsors. Against this problematic scenario, our argument is that evaluation can help with “repoliticising” the debate about public participation and its effects on technocratic approaches and hidden political agendas. Towards this aim, public participation should be considered as a descriptor of mechanisms (how to do it), democracy values (why to do it), and arguably as a primary factor in the policymaking process. Considering the complexity of the concept of participation, together with its uses and misuses, this post contributes to the debate by discussing the shortcomings of an evaluation carried out on a case in Portugal.

Why evaluate public participation?

The evaluation of public participation in policymaking can provide information about the political promotion of participatory principles and the actual implementation of the participatory mechanisms. As such, evaluation can feed the policy process with knowledge about the different forms of expertise required for participatory processes, as well as the different forms of knowledge produced during the process, be they political, technical, or social. Despite several good reasons for running evaluations on this kind of process, literature about the topic is scarce and evidence of structured methodologies is even less prevalent. According to Rosener (1981), the scarcity of reliable evaluation methodologies is due to the lack of general agreement about what participation should be for and the very complex nature of the phenomenon in the field. The scholar, in line with the problems more recently raised about the transformation of participation into a buzzword (Cornwall, 2008) as well as the discourse crisis about its uses (Dagnino, 2004), also argues: “The use of evaluation research methodology in the design of participation programs will mean the exposure of value biases and “hidden agendas” […]. Ambiguity often serves to protect public administrations and citizens alike from being accountable, thus “going public” with explanations of participation assumptions will prove useful. It will mean that those mandating participation programs will need to clarify their expectations; public administrators will need to be honest about their intentions; and citizens will need to be reasonable in their demands.” (Rosener, 1978, p. 462).

Why this case study?

The coming together of different agents is crucial for public participation, as well as for its evaluation. The narratives produced by the agents involved in the evaluation of the project “Portugal Participa: caminhos para a inovação societal” (“Portugal Participates: routes for societal innovation”) (hereafter: project) in Portugal posits some critical challenges to this debate. The project was implemented between 2015 and 2016, led by the national NGO “In Loco” together with the University of Coimbra, and the Gulbenkian Foundation. The project aimed at promoting participatory principles and mechanisms both on a national scale (conferences, meetings, and the creation of the network of municipalities implementing participatory processes) and on the local scale, by supervising four pivotal processes in four Portuguese municipalities: the participatory forum for health promotion in Cascais; participatory budgeting in Funchal; the participatory revision of the master plan in Odemira; and the participatory regeneration of the Campanhã area in Porto. In light of the aim of the evaluation, the two scales were assessed using four related approaches: underlying logic evaluation; process evaluation; impact evaluation; and cost evaluation. The approaches explored specific issues and created specific criteria against which the project was evaluated. Specific methods (data collection; online survey; semi-structured interviews; counterfactual focus groups) were applied with five typologies of agents (political representatives; civil servants; NGOs; civil society; sponsors). The agents were selected in order to ensure the inclusion of all the relevant actors intervening in the four local processes and in the implementation of the project on the national scale. A total number of 23 agents took part in the evaluation: 4 political representatives (one representative for each municipality); 9 civil servants; 5 NGOs (one city without representation); 2 citizens (two out of four municipalities were not represented by citizens); and 3 sponsors (one leading organisation, one partner, and one funding institution). Their involvement should be considered as informative in that their opinions were considered part of the evaluation, though not decisive for its results (cf. Murray, 2002).

Why analyse narratives?

The goal of the content analysis of the narratives produced by the agents was to provide evidence about the reasons for the importance of the evaluation of public participation both when supervising the implementation of this kind of processes, and in deepening our understanding of participatory phenomena. The analysis was run on the narratives produced via: (i) individual interviews with both local agents and national sponsors; (ii) counterfactual analysis of focus groups with local agents; and (iii) pre-post surveys with local agents and national sponsors. Before discussing the analysis, two characteristics of this work should be clarified. The first concerns the coming together of outcomes retrieved from the application of different methods implemented at different stages of the evaluation. Related to this, it is also important to underline that the data were selected to provide evidence from narratives on public participation (the evaluation included three additional areas: societal innovation; institutional partnership; and local development).

How to analyse narratives?

At the operational level, the analysis was run in order to first disaggregate findings according to the typology of agents and, secondly, aggregate the findings according to four emerging themes: government; governance; policymaking; and society. The first goal sought to identify common positions for each group of agents, by synthesising the main topics resulting from the analysis. Although this procedure does not take into account significant differences among the agents, the summing up of their positions is necessary for the exploratory understanding of the ways different agents produce different narratives. The aggregation of the different positions expressed by the agents was subsequently operated within the themes identified through the content analysis. Each theme is actually a specific “object” affected by public participation, and eventually leads to a broader understanding of the phenomenon. As such, the identification of lowest common denominators for each theme sought to sum up the variety within. The findings are organised as follows: tab.1 shows the key concepts retrieved from the narratives of the agents, while tab.2 shows the lowest common denominators of each emerging theme and provides a short example of the agent narratives.

 Table 1 - Agents

  Table 2 – Themes

What did we learn from this case study?

These results show that the five typologies of agents put emphasis on different topics related to the implementation of public participation. The second table gathers the lowest common denominators of the four themes identified in the content analysis. Likewise, each theme portrays specific quests for public participation and, accordingly, the examples of narratives provided aim to remark upon these differences. Finally, this exploratory study confirms that more debate and evidence on the evaluation of public participation is needed, as it also represents an exclusive source of information.

Bibliography

• Cornwall, A. “Unpacking ‘Participation’: models, meanings and practices” Community Development Journal. 43 (3) (2008): 269–283.
• Dagnino, E. “Construção democrática, neoliberalismo e participação: os dilemas da confluência perversa” Política e Sociedade, 5 (2004): 139-164
• Rosener, J.B. “User-oriented Evaluation. A New Way to View Citizen Participation.” Journal of Applied Behavioral Science, 17 (1981): 583-596.
• Rosener, J.B. “Citizen Participation: Can We Measure Its Effectiveness?” Public Administration Review, 38 (5) (1978): 457-463.
• Murray, R. “Citizens’ Control of Evaluations. Formulating and Assessing Alternatives.” Evaluation, 8 (1) (2002): 81-100.