In 2020 a Mokoro team conducted the annual outcome monitoring for WFP Kenya’s Sustainable Food Systems Programme under restrictions imposed by the Covid-19 pandemic. In a previous article, Mokoro’s Survey Coordinator, Ernest Midega, presented some of the innovations introduced to ensure remote data collection from 1,460 households across the arid counties was possible. In 2021, the team once again were forced to adapt to limitations imposed by the pandemic in order to ensure data collection as part of the annual outcome monitoring and mid-term evaluation of the programme. In this piece, Farida Hassan and Zoe Driscoll, outline some important changes made to the methodology to ensure that in-depth beneficiary perspectives were collected as part of the qualitative component of the evaluation’s primary data collection.   

At the beginning of 2021, Mokoro kicked off the preparation for the Mid Term Evaluation (MTE) of WFP Kenya’s Sustainable Food Systems Programme, optimistic that safe data collection could take place with in-person data collection. However, in the weeks leading up to data collection, a new wave of the Covid-19 pandemic hit Kenya and restrictions impacted on every sector of the economy, as well as the plans for conducting the MTE of the programme. The Mokoro team considered various options including postponing the MTE. After a series of discussions between the Mokoro and WFP teams, it was agreed that data collection for the MTE should take place remotely, as was the case during the annual outcome monitoring in 2020. To ensure validity of the data, as well as robustness of the MTE, various methods were adopted, including: a desk review of documentation and data, telephone-based interviews with national and county level stakeholders, a remote household survey for collection of quantitative data conducted from a central call-centre, and an online survey for the County Government Staff who are the key implementers of the programme.

An interesting addition to the MTE was seeking, albeit remotely, the voice of the beneficiaries of the various interventions across the arid and semi-arid (ASAL) counties of Kenya. This was considered important as the MTE is a critical phase of the project, providing the project implementers important evidence to inform and improve implementation of the programme going forward. For this purpose, the Mokoro team designed, tested, and deployed tools that were used to carryout in-depth qualitative interviews with beneficiaries on some of the hard to measure aspects such as perceptions on effect of the programme on their personal, household and community wellbeing including their sense of food security, socio-economic status and general resilience to climate related shocks.

To obtain a good sample, the beneficiaries were clustered by county according to the interventions/activities that they were engaged in. The clusters comprised beneficiaries receiving food or cash transfers; those participating in Village Saving and Loans Associations (VSLAs) and youth agri-business groups; those beneficiaries engaged in different livelihood activities, including beekeeping, vegetable production, production of orange-flesh sweet potato, value addition, pasture production and fisheries, among others; and those trained in wider value-addition and market-linkage activities such as milling and fortification, food safety, and retail. From each cluster, a representative sample across the ASAL counties was selected to participate in the interviews held over the phone.

In engaging the beneficiaries, the Mokoro team were able to remotely obtain rich qualitative data on areas such as perception on personal sense of well-being, level of resilience or vulnerability and social capital necessary for weathering shocks. In some instances, the beneficiaries shared videos, pictures and voice messages using the WhatsApp platform in their bid to paint the picture of the field level context regarding progress made through the programme. The collected qualitative data also provided insights from the beneficiaries’ point of view of the effectiveness of the sustainable food systems programme in addressing their needs including making recommendations on improving the overall design of the programme for a better performance.

The experience of collecting data remotely was a positive one. The team found that one-to-one interviews over the phone gave beneficiaries a safe space to speak openly about the programme, compared to in-person focus group discussions. Beneficiaries were open to giving their time and elaborating on their experiences.  It is also easy to fall into being taken to the programme success stories when conducting field-work in-person, as well as to selecting those communities which are easily accessible from the county capitals. In this case, the team were also able to ensure that a completely random sample of beneficiaries were selected to participate in the discussions, as is often not the case conducting focus groups with beneficiaries brought together by programme implementers and community leaders. The sample also covered a wide sample across different counties, enabling comparisons and differences of programme implementation in different county contexts to be drawn out. However, there are notable limitations. Most important is that the team were restricted by the availability and access to mobile phones by beneficiaries. Despite good records kept by WFP, the team found that many of the beneficiary’s mobile phones numbers were no longer in use and it is known there are many additional beneficiaries who do not have their own mobile phone.

Land preparation by a farmer in Wajir County ahead of the long rains in late March 2021

During data analysis and report compilation, the qualitative data obtained from the beneficiaries was combined with findings from the quantitative household survey, feedback from stakeholders and desk reviews to provide a realistic evaluation of the programme in terms of progress made so far in enhancing food security amongst target households in the ASALs. Through the voice of the beneficiaries, the Mokoro team was also able to determine the extent to which the beneficiaries appreciated the activities and applied the various initiatives promoted under the programme. The qualitative evidence provided by these in-depth interviews provided invaluable richness to the evidence and enabled triangulation with the household survey.

The experience of remotely engaging the beneficiaries during the MTE revealed that the challenge of access to good qualitative data due to limitation of conducting physical field visits, can be overcome through reaching out to the voice of the beneficiaries. In addition, platforms such as WhatsApp can be instrumental in facilitating the sharing of real time information in the form of videos, pictures and voice messages necessary for enhancing not only the understanding but also in building the context of the field situation.

Download this article.