Methodology
A mixed methods approach involving both quantitative and qualitative research methods was used in the monitoring process. The quantitative method involved a survey of the experience of a proportion of beneficiaries in the NCTP and verification of data generated in the programme, while the qualitative method comprised a review of documents, audit of financial records and the review of processes and data in the upstream[1] and downstream[2] sectors of the NCTP.
The study area comprised 20 locations (19 States and the FCT)[3]. This represents the total number of States that benefitted from the May/June 2019 payment round of the NCTP. These locations are spread across the six (6) geo-political zones of the country.
Location selection
A total of 54 LGAs were selected with at least 2 LGAs per State from 2 different senatorial districts to provide a varied view. Following the application of an inclusion and exclusion criteria,[4]a multistage cluster sampling procedure was utilized to select the 54 LGAs visited in the exercise. Communities visited were then selected via convenience sampling utilizing the inclusion and exclusion criteria for location sampling listed below:
Table 1: Inclusion and exclusion criteria for location selection
|
Inclusion Criteria |
Exclusion criteria |
|
|
|
|
|
|
|
|
Beneficiary survey sample size
Total Population (beneficiary survey): The total population were all beneficiaries who received payment in the May/June 2019 payment round in all eligible States (N=329, 963)
Target population (beneficiary survey): 12% (39,596 beneficiaries) of the total population of beneficiaries who received payment in the May/June 2019 payment round (N=329, 963) were targeted with allowance made for non-response by taking 10% of the target sample and summing this with the original sample to get the final target sample (43,555).
Valid sample size (beneficiary survey): The final questionnaires retrieved from the field and found useful for data analysis was 43,152 (n=43,152). This represents a response rate of about 93%. This final (valid) sample size (n=43,152) comprised 13% of the total population and varied between 11% and 22% for States visited. (See annex for beneficiary sample size distribution per state
Key informant interviews – sample size
A total of 89 officials from the National level, 20 States and 54 LGAs were interviewed in the data verification exercise. Their distribution is as follows:
- Payment officers at national level (2)
- Enrollment officers at national level (1)
- MIS officers (1)
- Payment operator representatives for all States paid in the May/June round (10)
- State Cash Transfer Officers (20)
- LGA facilitators (55)
Documents reviewed – sample size
260 (reports, bank statements, correspondence, invoices, and summary sheets)
- Data collection
The beneficiary survey tool (questionnaire)[5] was administered directly to the registered beneficiaries by trained enumerators while a data verification key informant interview guide was administered to select government officials. A total of 1 auditor, 658 enumerators, 80Supervisorsand 74 data entry clerks were engaged for the exercise.
The survey data collection exercise lasted for one week (5 days), and data collected was encoded by the data entry clerks and saved in a Microsoft Excel template developed for the purpose of data analysis and interpretation.
Survey Questionnaire
A questionnaire was developed for the beneficiary survey. The instrument had five (5) sections:
- Section A addressed the background information of the beneficiary (i.e. personal characteristics of the respondents); and a review of the beneficiary ID
- Section B focused on issues relating to programme targeting and enrolment;
- Section C addressed cash disbursement issues in the programme;
- Section D examined the challenges experienced by beneficiaries
- Section E looked at grievance redress mechanisms of the NCTP.
Data verification key interview guide
A data verification guide was developed for the data verification exercise, and use to collate national, state, LGA and community level data from responsible government officials at the sampled sites. The data verification exercise key informant interview guide was to collate the following data:
-
-
-
- Total beneficiaries enrolled
- Total beneficiaries paid
- Total amount paid
-
-
Both instruments were peer reviewed by all participating CSOs in a workshop that lasted 2 days in Abuja, and data verification technical experts. Corrections and suggestions were incorporated into the final instrument.
Definition and Interpretation of the Verification Factor
For a specific reporting level, the verification factor is the ratio of the verified count (which the monitoring team recounts from source documents at the reporting level) to the reported count (from the summary report at National) for a specific reporting period. It is usually expressed as a percentage, and is mathematically represented as:
Verification Factor = (Verified count at selected Site)/ (Reported count at selected Site) ×100
Interpretation of the Verification Factor
Verification factors greater than 100 percent indicate under-reporting (i.e., the source shows a higher actual count than the numbers of the summary reports for the reporting level), while verification factors less than 100 percent indicate over-reporting. A variance of less than 10 percent in either direction may be considered a minor issue. While systematically high levels of over-reporting or under-reporting that are not due to errors can lead to questions on the authenticity of the data reporting system.
Data analysis was preceded by a data management process which involved cross-checking the final soft data for missing and / or irregular values. This entailed physical observation of the data, as well as running frequency distribution to have broader view of the data spread, which was compared to the data instrument to ensure coded responses aligned with the instrument coding. Analysis of the data was done with the aid of a statistical application (SPSS version 25).
Qualitative components of the data instrument were coded as obtained from the interview process. Common themes (points) across the various responses were created and quantified for the purpose of frequency distribution.
The following quality control measures were applied to ensure data quality
- Team composition
- Only trained team members selected
- Team members identified from survey LGAs
- Team members could speak local languages
- Team Supervision
- Presence of state level and national level supervisors
- Designated command center team to attend to all queries
- Survey Team Operations
- Standard operating procedures (SOPs) for consistency of operations
Daily review of field data and team performance
- Only the registered caregiver or alternate were interviewed
- Review of field data
- Daily validation checks
- Use of dashboard to monitor and support field teams
- Incomplete data was excluded from the final analysis
- Validation of findings with the NCTO and National Social Safety-Nets Coordinating Office (NASSCO)[6] representatives with respect to findings and recommendations
Verbal informed consent was taken from all respondents using a standard script and respondents had the option to refuse participation.
[1]Upstream actors are the Central Bank of Nigeria (CBN) and the National Cash Transfer Office (NCTO)
[2] Downstream actors are the Payment Service Operators (PSPs), the State Cash Transfer Offices (SCTOs) and relevant Local Government Areas.
[3]See Image 1 for the list of states monitored
[4]See Table 1 for inclusion and exclusion criteria and list of LGAs.
[5]See figure 9 (Data verification tool) in annex
[6] NASSCO is the office responsible for the overall coordination of Nigeria’s social investment programme and builds the country’s social register from which beneficiaries of the NCTP are mined. The data on beneficiaries of the programme were therefore verified with NASSCO as part of the monitoring process.
THE MANTRA MODEL
/sites/default/files/MANTRA%20MODEL.pdf
