Can we use citizen feedback from Community Radio and USSD for insights on measuring basic service delivery in urban and rural Kassala, East Sudan? UNDP Sudan’s pilot experience to measure SDG 16.6.2 indicator explores about the prospects in a post conflict development setting in part one of this blog series.

.

Accelerating Goal to Sustainable Development

Accelerating structural transformations for sustainable development is one of the three main UNDP development goals for next four years. Aligned with this corporate goal, the improvement in governance sector focusing on strengthening local capacities for planning and effective service delivery is the foremost development priority for 2018 -2021 UNDP Sudan’s country programme. This focus area has direct contribution to building effective, accountable and transparent institutions under the Sustainable Development Goal 16, an overarching goal that encompasses three components - peace, rule of law, and good governance. For Sudan, prevalence of peace has been identified as an accelerating goal[1] which necessitates stronger government institutions amongst other aspects.  Likewise, Sudan’s government formed in 2016, through its national dialogue recommendations, strongly calls for enhancing governance institutions to better serve the citizens.

United Nations Inter Agency Expert Group (IAEG) on SDG Indicators has identified UNDP as the custodian agency for two indicators under SDG 16. The first, under Target 16.6 is - Proportion of population satisfied with their last experience in public service delivery (16.6.2). Measurement of this indicator is highly significant as it directly relates to a country’s ability to deliver essential services reflective of sound expenditure, planning, arrangements and resource management systems in place, ideally at all levels of governance (federal, state and local) ensuring better outcomes for its citizens’ prosperity. Measurement of this indicator however is not common in developing countries and hence not systematically established in many of their national statistics systems. IAEG-SDG has hence classified the 16.6.2 indicator under Tier Three (III) category for which currently there are no standard methodology and standards, creating a demand and opportunity for innovative approaches to measure the indicator as per country context.

Lessons from the past for UNDP Sudan

Local governance development is one of the key areas supported by UNDP in many programme countries. In Eastern Sudan, UNDP’s Local Governance Development and Public Expenditure Management (LGDPEM) project has invested five years (2010-2015) and substantive resources to optimize State Government’s capacity for resource planning and public service delivery, reverse negative trends in socio-economic indicators and strengthen overall governance and public expenditure management systems. The 2016 LG4SD[2] evaluation study of the project attempts to establish dependency between two variables – local services delivery and budgeting to the performance of local service providers based on a research conducted amongst 252 government officials at state and local levels. The study establishes three factors for improvement in service provision in East Sudan - a) administrative systems and procedures used by local governments; b) enhancement in strategic planning processes; and c) local communities’ involvement in development planning. However, it does not take into account citizen - beneficiary views to provide a picture of the impact directly on ground leaving a gap in its findings.

Looking at the present - Why community feedback is important to Kassala government?

According to UN DESA - one of the core dimension to monitor governance progress is public participation. A key aspect in this dimension is the use of smart tools and community networks that ensure inclusive participation from citizens. The LG4SD study had identified local community involvement in development planning as a success factor for effective service delivery in Kassala state. Considering this aspect emphasized by both UNDESA and LG4SD study; and the new state government priority to boost community participation in development, such as through “Nafeer” (Free Participation in Arabic) programme which aims to enhance sustainability of their development interventions through local ownership; the General Directorate of Economic Planning and Development of Kassala State Ministry of Finance, Economy and Labour Force (MoFELF) was keen to obtain a picture of their service delivery status at locality level directly from the people of Kassala. They were in need of a new but cost-effective solution to overcome the sample gap experienced in LG4SD study on the ground impact of their investment and capacity building interventions made through both state and external resources as well as with regards to the pressure created on local services due to influx of refugees and migrants from neighbouring countries of Eritrea and Ethiopia. They wanted an updated understanding to validate accuracy of their resources distribution, and delivery of various services and investment not only on the basis of geographic access but also on a sector by sector functional performance.

In late 2017, UNDP collaborated with Kassala MoFELF on a common goal to explore methodologies that capture community feedback that are quick, cost effective, allowed active participation of citizens regardless of their literacy levels, gender or ethnicity and which would potentially enable service provider (government institutions) response early and on a long-term basis.

Proposed solution - Locally adaptable and innovative citizen feedback system and partnerships for SDG 16.6.2

Through the funding support of UNDP Innovation Facility for cross regional initiatives on measuring SDG Tier III indicators, UNDP and the Ministry ideated a mixed method approach that combined locally popular means and technology to measure the public service delivery through collection of residents’ feedback. Taking reference from Uganda community radio data innovation work, and the high penetration of mobile phone use in Sudan (estimated above 70%) the proposed mixed method attempts to capitalize on the wide access the two mediums – radio and mobile phones have in reaching larger and targeted population in the state. To test this mixed approach, a pilot was co designed (Figure 1) with five different stakeholders in Kassala: MoFELF, Line Ministries providing Basic Services – Health, Education and Water Supply, Kassala Broadcasting Corporation – Radio Division, Zain Telecom and UNDP as well as research institutions outside Sudan – Qatar Computing Research Institute (QCRI) and Data-Pop Alliance for data collection design and analysis support.  UNDP Oslo Governance Centre advised on the development of guiding questionnaires adaptable to radio and SMS/USSD format.

.

Kassala State Radio station implemented the programme between 1 May 2018 to 19 May 2018 where nine radio live programmes of one hour each were conducted in four local languages (Arabic, Hadandawa, Beni Amer and Nubba) in the format of a discussion with service provider experts, targeting community members from four localities in the state - Rural Kassala, Urban Kassala, Rural Aroma, and Halfa El Gedid (Map 2). The live broadcast preceded a week-long promotion asking community members from target localities to participate by calling on the radio telephone line during the discussion programme and sharing their feedback. Each week the programme focused on a single service sector starting with health, followed by education and finally water supply.

The radio dataset in both audio and transcribed format are currently under analysis to check whether meaningful and representative insights could be derived from radio airwaves. UNDP has collaborated with QCRI to conduct this audio analysis part. Meanwhile, in this blog we share initial observations of the radio feedback implementation phase.

What have we observed so far? 

For an hour-long show, the participation rate was found to be quite high as there were no delays or breaks in calls received from the moment the show started. This was observed in all nine live shows. Out of the total 380 calls received over the nine shows, 9% were from states other than Kassala including the border states with South Sudan - Blue Nile, White Nile as well as Gedaref (Map 1). The states of Blue Nile and White Nile receive refugees and IDPs from South Sudan which may imply that the influx of vulnerable people is creating additional demand and pressures to existing basic service provision and hence are not adequate. Similar is the case for Gedaref, a transit state for Ethiopian migrants and refugees.

Similarly, within Kassala state, there was high participation from localities outside targeted ones. In fact, Khashm Ghirba locality surpassed the calls from three other target localities (Fig. 2). On an average, women formed 20% of callers for each sector with higher participation in health sector feedback (Fig. 3). 

Comparing the locality level frequency of calls for sectors (Fig. 4, Map 2) it is clear that education followed by water supply ignited higher interest for community members of target localities Kassala and Halfa El Gedid in comparison to other target localities Aroma and Rural Kassala, while from non-targeted localities, participation from Hamash Koreb locality remained lowest across sectors and the highest was observed from Khashm Girba. This could be the influence of high radio listening rates (94.6% of the state population) as well as awareness level of citizens in these localities or it could indeed be reflecting the extent of their needs. To confirm this assumption, we need to await results of the audio analysis and SMS/USSD based feedback phase. 

.

Next steps and looking forward to the future

The pilot has completed its Phase 1. The next phase includes radio data analysis aimed at automating the process of indicators extraction applying radio data transcript processing tools to generate structured data for further analysis. With support from QCRI, a number of strategies and open source tools for Natural Language Processing will be explored such as information extraction, text segmentation, tokenization, and text similarities for Arabic and other languages on needs basis. Based on the findings, updated questionnaires addressing information gaps from Phase 1 will be deployed through USSD feedback system in target localities by Zain Telecom. Measures per sector from both radio and USSD mediums will be later composited and explored for computation of SDG 16.6.2 indicator applying Vietnam’s PAPI model deriving insights on the status of basic service delivery for Kasssala and target localities. Our next blog will feature the radio data analysis results and deployment experience of USSD based survey.

Our pilot is proposed as a proof of concept of a mixed method business model and therefore does not delve deeper into service delivery gaps as yet. It attempts to test whether the new mediums, new approach and new partnerships is effective in obtaining the voices of diverse citizens who are remotely located within a state that receives high volume of refugees and migrants. This pilot is an effort to demonstrate the use of innovative community feedback system as a mechanism by the state government to encourage social accountability over donor accountability that could act as a driver to at least ensure minimum quality basic service delivery for the state of Kassala which may potentially lapse into instability due to pressure on local services. If successfully piloted, this approach has the scope to not only measure SDG 16.6.2 indicator but enable improvements in service delivery; obtain insights on other public service areas; increase social accountability of service institutions enhancing relationship and trust with its people thereby contributing to stronger institution and local governance building to support development and maintain peace in a post conflict but still at-risk setting.