A step from the ordinary
Data collection during the Covid-19 pandemic in Kenya
2 March 2021
- 0 Comments
Since 2019 Mokoro have been carrying out the long-term monitoring and evaluation of WFP Kenya’s Sustainable Food Systems Programme (2018-2023) which is being implemented in 14 arid and semi-arid areas of Kenya. In 2020 the annual outcome monitoring was hampered by the restrictions of the Covid-19 pandemic. This article, written by Mokoro’s Survey Coordinator, Ernest Midega, outlines some of the adaptations that the team made to ensure data collection in 2020.
“Do we still continue with the survey or not? And if we are to continue, how do we do this and how do we ensure quality without compromising the safety of the survey personnel and the respondents?” These were the questions we constantly had to ask ourselves in March of 2020 as we were planning for the WFP Strategic Objective 2 (SO2) outcome monitoring. Covid-19, with all the restrictions and the preventive measures, had thrown us completely off track with our planning for the outcome monitoring survey. I mostly took the backseat as decisions were made on whether or not we could continue with the study. However, immediately we received a ‘go’ from WFP on the survey, I was thrown right into the driver’s seat. I had the responsibility of not only making sure that the survey is able to take place, but also in ensuring that it is carried out in line with Covid-19 protocols and procedures. In all this there was a crucial focus, ‘the data collected MUST be of quality’. After long deliberations and modelling of different scenarios and possible outcomes, as well as lengthy consultations with other data collection experts who had previous experience with data collection in conditions of limited accessibility, we settled on remote data collection through phone interviews.
We had to make careful consideration of the amount of time a respondent would be comfortable to be on call, as opposed to a routine field survey, without being fatigued. We also considered the practical sampling that could be carried out and the respondents reached, as we were only able to secure a full list with phone numbers of WFP Kenya’s cash beneficiaries and not in-kind food beneficiaries, and this was very critical for sampling. It was finally agreed that the interviews would last for up to 15 minutes per phone call as opposed to the normal 2 hours interviews, and we would reach a total of 1,460 compared to the usual sample of over 8,000 respondents.
I am experienced and quite confident with regular field surveys where we have no restrictions in mobilizing and reaching respondents. In regular field surveys, quality is never a problem to me as I have all the mechanisms, skills and tricks to ensure utmost data quality. However, in this case, I was in unfamiliar territory. The power of technology like GPS locators (that allow the survey supervisor to monitor the actual data collection locations), to verify the location of enumerators when entering data, and sound knits (sounds automatically recorded by the data collection device on intervals) to enhance data quality could no longer be used. Clues from respondents picked up during in-person interviews, as well as other quality assurance procedures used for these interviews, would no longer be applicable but still we had to deliver data that is not only ’good enough’, but data that is the ‘best quality’.
My colleagues and I held more than ten planning meetings over Zoom focusing just on two things: how to prevent any survey team member catching Covid-19, and collecting quality data. Mobilizing the respondents was the least of our worries since WFP had taken this burden off us and had done a good job in mobilization using a bulk short messaging system to the prospective respondents. Not all the meetings were successful and at times I would end the Zoom call and think to myself, “Will this really work?” However, there was only ever one response, “It has to work no matter what!”
We finally settled on a plan! We would set up a call centre, to ensure consistent data collection and overcome mobile connection challenges, and have all the survey personnel make calls to the respondents from the call centre after training and pre-testing. We had to worry about their travel to the call centre without contracting the infection, how to prevent and manage any case of infection at the call centre, and how to manage post-activity cases. We settled on the following;
- All the survey personnel would be hosted at the call centre without leaving for the entire period of the survey and with no visitors.
- All the survey staff would take utmost caution as they travel to the call centre.
- They would arrive to a holding venue and take a Covid-19 testing prior to being transferred to the call centre.
- We would put in place very elaborate Covid-19 preventive measures at the centre including routine disinfection, scheduled temperature checks and elaborate social distancing procedures.
- We liaised with the government’s Covid-19 task force as an emergency back-up plan and had a standby van for emergency facilitation in cases of need. In addition, the public health official volunteered to provide a briefing on Covid-19 to the survey participants.
- All the personnel had to take a post-exercise Covid-19 test as a precautionary measure to ensure that we are not ‘exporting’ Covid-19 to the villages.
In this process, I had one advantage. We had deliberately agreed to recruit survey enumerators and supervisors we had previously worked with, who had experience of following rules and regulations of engagement during a survey exercise and strictly following them. I was quite confident that this would be the case with this survey and I am glad I did not spend time ‘monitoring behaviour’.
My colleague and I are experts in data quality assurance and we are always innovative about this. Since we had everyone confined to a location with constant internet connection, we had the advantage of real time submission of data to the server and we could watch and sieve the data as it came in. The challenge was finding a way to sample interviews and listen in as the enumerators conducted the interviews to ensure quality. We came up with a plan that enabled us to sample and listen to calls through the call centres. We also ensured that the enumerators and their supervisors have routine quality assurance check–ins with each other; we configured the data collection devices to auto-submit the data to the server; and we developed data quality checks syntaxes to routinely check the quality of the data that was collected.
Most of the locations targeted by the survey are very remote and there are challenges with cellular network access. Though we made all the necessary effort to pre-inform the respondent about the anticipated survey call, including through bulk SMS, we still experienced challenges where respondents were not available not only as a result of the mobile phone network but also some of the respondents lacking phones and thus sharing their handset with others. Though the bulk SMS played a bigger role in mobilization of the respondents, the enumerators were able to use their own personal networks to bring as many sampled beneficiaries as possible to the phone of someone who has use of one.
Through this survey challenge, I have learnt, and we have learnt as a group. It is evident that remote data collection is here with us, we do not know what the next crisis will look like, when it will hit and how it will affect normal living. However, one thing is for sure, programmes and projects have to be monitored and evaluated with sound and robust methodologies, no matter the crisis. We need to start thinking of and embracing more remote training of data collectors. We have to think of better and more effective remote respondent mobilization techniques. We have to continue to review and strengthen remote data collection techniques and capabilities. Finally, we have to still ensure robustness and quality of data collected through any collection method used.
Working in confinement with very restricted one-on-one association with other colleagues for more than 10 days is not easy, both mentally and physically. The survey team was however in good spirits and ensured that the exercise was a success despite being locked up in a hotel for 2 weeks!