However, the interest from Nigerian health officials in learning lessons for improving MCH outcomes remained high. A favourable policy environment within which the study is conducted will ensure the successful uptake of results into policy and practice. Evaluation. This is a protocol for a realist evaluation of a collective leadership intervention. The project uses an input-process-output-outcome continuum (see Fig. J Health Serv Res Policy. A general model for an ITS regression analysis (which will be adapted to allow for comparison between SURE-P/MCH, SURE-P/MCH + CCT and control clusters) is: where Y In realist evaluation the researcher or evaluator’s task is to gather the data – i.e. This realist technique acknowledges that intervention programmes and policy changes do not necessarily work for everyone, since people are different and are embedded in different contexts. One of the tasks of a realist evaluation is therefore to make the theories within a program explicit, by developing clear hypotheses about how, and for whom, 2nd ed. METHODS AND ANALYSIS: The study will be conducted in four phases. 1), i.e. Meeting the aims through realist evaluation. The eventual decision was to continue with the study, using the original methodology, though with the addition of assessment of sustainability of achieved changes and effects of on-going lobbying and advocacy efforts on entrenching the MCH on the political agenda in Nigeria. Effectiveness of a programme is thus not dependent on the outcomes alone (cause–effect), rather there is a consideration of the theoretical mechanisms that are applied, and the socio-historical context in which the programmes were implemented. Introduction Significant evidence in the literature supports case management (CM) as an effective intervention to improve care for patients with complex healthcare needs. 2007;101(2):95–105. 2009;8(1):265. The qualitative and quantitative methods will be integrated throughout to answer the eight study research questions. Since December 2012, CCTs have been implemented in selected SURE-P/MCH sites in 9 of the 36 states of Nigeria (SURE-P/MCH + CCT). Correspondence to (PDF 555 kb), Ethical approvals from the University of Nigeria. financial incentives to pregnant women to register at a Primary Health Care (PHC) centre, receive health check-ups while pregnant, deliver at a health facility and take their baby for vaccinations. Health Res Policy Syst. The relationships between and amongst these Cs, Ms and Os will be explored as part of the data collection and analysis. Abuja: Federal Ministry of Health; 2013. That is, whenever a program is implemented, it is testing a theory about what ‘might cause change’, even though that theory may not be explicit. Design: A multimethod study design was undertaken using realist evaluation methodology. The study is guided by the research questions shown in Table 1, alongside the corresponding objectives. This realist evaluation protocol sets out the approach that will be used to identify and explain which aspects of the programme are working, for whom and in what circumstances to produce the outcomes intended. REVAMP project: ‘Determinants of effectiveness and sustainability of a novel community health workers programme in improving mother and child health in Nigeria’. This paper presents a protocol of a multimethod study using a realist evaluation approach to investigate the impact and effectiveness of IR in hospital wards on the organisation, delivery and experience of care from the perspective of patients, their family members and staff. The state of the evidence on programmes, activities, costs and impact on health outcomes of using community health workers. Question guides will be adapted to the different actor groups, commensurate to their backgrounds and roles in the design and implementation of the programme. This combination was intended to improve access to quality health services and ultimately reduce maternal and child morbidity and mortality. COMDIS-HSD. Realist evaluation reports need to be developed in line with the realist nature of the evaluation. The aim of this study is to inform strengthening, scaling up and ensuring sustainability of CHW programmes. During step 2, we shall use a mix of methods to validate hypothetical pathways. For further reference on the realist interview and realist data collection, Manzano’s (2016) paper provides an overview of interviewing within a realist evaluation. Pawson and Tilley (1997) developed the first realist evaluation approach, although other interpretations have been developed since. Other services might also be affected by resources (or their lack) such as drugs and theatre time that are channelled into the new programme. Whereas in realist review the primary data comes from documents (e.g. The suggested protocol uses a realist evaluation with multiple cases across the 3 country sites: Turkey, Iraq and Lebanon. Bhutta ZA, Memon ZA, Soofi S, Salat MS, Cousens S, Martines J. 2010;25(5):551–64. The data will be collected using three methods. Int J Technol Assess Health Care. While studies have explored the effects of supply- and demand-side interventions separately [15, 18, 19], the combined effects of the two—such as the added value of CCTs within health programmes—are rarely assessed within the same intervention. Cost Eff Resour Alloc. [4], Pawson & Tilley (1997) describe the procedure followed in the implementation of realist evaluation techniques in programme evaluation and emphasise that once hypotheses have been generated and data collected, the outcomes of the programme are explored, focusing on the groups that the programme benefitted and those who did not benefit. Ozawa S, Pongpirul K. 10 best resources on … mixed methods research in health systems. That is, whenever a program is implemented, it is testing a theory about what ‘might cause change’, even though that theory may not be explicit. 2008;6(1):20. This approach, developed by the Nuffield Centre, has been effective in many countries in improving the quality and effectiveness of the scaled-up programme [46]. During the data collection and analysis stages, the case studies are used to refine or further generate CMOCs ( Marchal, Van Belle, van Olmen, Hoeree, & Kegels, 2012 ). 2010;88:364–70. This will give us greater understanding of the health system context, including the links with relevant policies, practices and programmes (e.g. The NDP comprises a range of organisational and service delivery changes to support evidence-based practices and policies. skills and practices of CHWs and effectiveness and efficiency of Primary Health Care facilities) and outcomes (e.g. In this paper, we report the methodology for a 5-year study which aims to evaluate the context, processes, outcomes and longer-term sustainability of a Nigerian CHW scheme. Mirzoev T, Baral S, Karki D, Green A, Newell J. Community-based DOTS and family member DOTS for TB control in Nepal: costs and cost-effectiveness. Ethical approvals for this study were obtained from the University of Leeds (ref: SoMREC/14/097) and the University of Nigeria (ref: NHREC/05/02/2008B-FWA00002458-1RB00002323). consistency of SURE-P management with overall governance approaches; or support to CHWs within staff supervision and performance appraisal systems). During step 1, we will develop specific hypothetical pathways (i.e. Spill-over effects and unintended consequences of the programme will be tracked, quantified and also explored qualitatively. Preliminary evidence indicates that paying CCTs to pregnant mothers is linked to increase antenatal care visits and facility deliveries [2]. Springer Nature. https://doi.org/10.1186/s13012-016-0443-1, DOI: https://doi.org/10.1186/s13012-016-0443-1. How does capacity building of health managers work? This paper presents a protocol of a multimethod study using a realist evaluation approach to investigate the impact and effectiveness of IR in hospital wards on the organisation, delivery and experience of care from the perspective of patients, their family members and staff. 1) to explore how inputs affect processes and how processes lead to outputs and ultimately outcomes. This realist evaluation protocol sets out the approach that will be used to identify and explain which aspects of the programme are working, for whom and in what circumstances to produce the outcomes intended. CCTs also target Traditional Birth Attendants (TBAs) to incentivise them to accompany pregnant women to PHC facilities. PROPEL: implementation of an evidence based pelvic floor muscle training intervention for women with pelvic organ prolapse: a realist evaluation and outcomes study protocol BMC Health Serv Res . organisations and their roles) and micro level (e.g. 2004;45(1):84–94. FMoH. it Realist approaches to evaluation assume that nothing works everywhere or for everyone: context really does make a difference to programme outcomes. $$ {Y}_t = {\beta}_0+\kern0.5em {\beta}_1{T}_t+\kern0.5em {\beta}_2{I}_t+\kern0.5em {\beta}_3{T}_t{I}_t + {\beta}_4{X}_t + {\varepsilon}_t $$, http://creativecommons.org/licenses/by/4.0/, http://creativecommons.org/publicdomain/zero/1.0/, https://doi.org/10.1186/s13012-016-0443-1. The study methodology will include three steps (Fig. An Intervention Involving Traditional Birth Attendants and Perinatal and Maternal Mortality in Pakistan. N Engl J Med. Lehmann U, Gilson L. Actor interfaces and practices of power in a community health worker programme: a South African study of unintended policy outcomes. The multidisciplinary and mixed-method realist approach will facilitate such evaluation. Karki D, Mirzoev T, Green A, Newell J, Baral S. Costs of a successful public-private partnership for TB control in an urban setting in Nepal. Protocol for a mixed methods realist evaluation of regional District Health Board groupings in New Zealand Tim Stokes , 1 Carol Atmore , 1 Erin Penno , 1 Lauralie Richard , 1 Emma Wyeth , 2 Rosalina Richards , 3 Fiona Doolan-Noble , 1 Andrew R Gray , 4 Trudy Sullivan , 5 and Robin Gauld 6 London: Medical Research Council; 2008. However, to guide further development and implementation of CHW programmes in different contexts, it is necessary to better understand what makes CHW programmes successful in achieving desired outcomes and under what circumstances they succeed [11, 12]. The study protocol reported in this paper has been evaluated through open competitive peer-review process as part of the Joint MRC/ESRC/DFID/Wellcome Trust health systems research initiative call 1 and accepted for funding (grant ref: MR/M01472X/1). The respondents for IDIs, identified through purposive sampling, will include health facility managers, CHWs, PHC staff and health planners and programme managers at local, state and federal levels. 2011;6(1):11. The specific objectives of the study are to: Develop an in-depth understanding of the context and the process of implementation of the interventions, including relationships between health workforce and infrastructure and supplies, Identify, assess and compare the intervention outputs (e.g. intra-component and inter-component, respectively. A realist evaluation study protocol. Article  Gopalan SS, Mohanty S, Das A. Assessing community health workers’ performance motivation: a mixed-methods approach on India's Accredited Social Health Activists (ASHA) programme. 2005;331(7525):1107. Robert E, Ridde V, Marchal B, Fournier P. Protocol: a realist review of user fee exemption policies for health services in Africa. First, small-scale facility exit survey (about 300 respondents) will use structured questionnaire to explore user perceptions about the programme and their experiences in accessing MCH care. An example involving the first two indicators might be: ‘deployment of CHWs, combined with improvement in infrastructure and supplies when implemented within the health systems context of Nigeria, will help reduce maternal and neonatal mortalities to 320/100,000 and 7/1000 live births respectively’. BMC Public Health. Evaluation of complex interventions such as SURE-P and their longer-term impact on MCH outcomes requires a comprehensive understanding of intervention context, implementation, mechanisms and outcomes. middle-range theories) that link intervention inputs to processes, outputs and outcomes within the context of Anambra state. Second, two initial hypothetical pathways or initial working theories (IWTs) (see Additional file 6) were developed, focusing on SURE-P/MCH supply and demand components, respectively. As new programmes emerge or existing programmes scale up, comparing the effectiveness of CHW programmes between different contexts becomes important. 'Protocol based care' was envisioned by policy makers as a mechanism for delivering on the service improvement agenda in England. We will select three study clusters, corresponding to Local Government Areas (LGAs): one with SURE-P/MCH, one with SURE-P/MCH + CCT and one with no intervention. 2005;352(20):2091–9. alternative respondents). Ritchie J, Spencer L. Qualitative data analysis for applied policy research. FMoH. Pawson R, Tilley N. Realistic Evaluation. Statistical analysis of quantitative data from the health management information system (HMIS) and SURE-P monitoring and evaluation (M&E) system will enable us to determine the extent to which the interventions achieved improvements in MCH services. It is based on the epistemological foundations of critical realism.Based on specific theories, realist evaluation provides an alternative lens to empiricist evaluation techniques for the study and understanding of programmes and policies. All authors read and approved the final version of the manuscript. Second, gaps in the literature on the CHWs, combined with an increasing interest in applied research from policymakers and funders, create a favourable environment for the study provide a timely contribution to an on-going debate about effectiveness of complex CHW interventions. In RE, researchers develop middle-range theories that take account of how Context (at micro, meso and macro levels) influences intervention processes or Mechanisms (e.g. 2011;38(2):65–76. Results: A protocol for a mixed methods realist evaluation was developed to gain insights into the mechanisms that foster successful results in ICPs. Google Scholar. realist evaluation covering the same kind of project or programme. Maternal, Newborn and Child Survival. Evidence suggests that the use of Community Health Workers (CHWs) can be effective in broadening access to, and coverage of, health services and improving MCH outcomes in such countries. This includes ensuring regular communication between the partners and engagement with policymakers and practitioners; quality assurance through regular peer-review within and between the teams; appropriate mentoring and coaching support to more junior researchers and equal opportunities to both genders. An example of the former is: ‘training and deployment of CHWs combined with a working supplies system, and implemented in the context Nigerian health system, will achieve increase in skilled birth attendance by over 60 % and improve equitable provision of MCH care’. Resources Overview Realist Synthesis: An Introduction: This paper presents an introductory overview of realist synthesis as applied to the review of primary research on healthcare systems. Using IDIs, we will explore actors’ understandings and views about the intervention’s context and processes (or mechanisms), their expected and unexpected effects (outputs and outcomes) and effects of advocacy and lobbying in entrenching the MCH on the political agenda. The specific hypotheses will draw upon two data collection methods: (a) review of key documents (SURE-P implementation manual, relevant federal and state-level policies, e.g. is a time series trend variable; I This handbook is available upon request. The study results will also inform strengthening the different aspects of the Nigerian health system, e.g. In realist evaluation the researcher or evaluator’s task is to gather the data – i.e. Programme Implementation Manual. The development of interview question guides will be informed by the study conceptual framework and structured around the study research questions to explore the specific hypothetical pathways identified in step 1. The qualitative evaluation proposed will be based on the document analysis of yearly ICP progress reports, selected case studies and focus group interviews with stakeholders. We realise, however, that implementation of a new programme can have an impact that goes beyond the direct programme beneficiaries and will consider these effects qualitatively through conducting IDIs with key actors referred to earlier. The additional costs on the supply side are likely to be training midwives and CHWs, salaries, equipment and other supplies (Table 2). We will ‘embed’ the research into policy and practice, working with the federal, state and local actors. Interrupted time series designs in health technology assessment: lessons from two systematic reviews of behavior change strategies. However, if we reach data saturation earlier, these numbers may decrease. J Dev Econ. BMJ Open 2012, 2(2). Rather, becoming competent at realist evaluation involves acquiring the ability to think, reflect and interpret data in a way that is resonant with realist philosophy and principles. Implementation Sci 11, 83 (2015). The aim of the ITS analysis is to identify discontinuities in the time series associated with, and potentially caused by, the introduction of the SURE-P programme to health facilities. PubMed Central  The Health Policy Research Group, University of Nigeria has developed three models for getting research into policy and practice (GRIPP), which will also be applied in this study. it is a form of primary research. While the average number of PHC facilities in each Nigerian state is typically between 1000 and 4000, the SURE-P/MCH was implemented in clusters of 12 to 21 health centres within selected states. Each IWT identified specific Cs, Ms and Os. 2013;13(1):847. We shall deploy the convergent model to allow continuous integration and triangulation of quantitative and qualitative findings. The hypotheses can cover more than one indicator. This paper should be of interest to researchers who are interested in adapting and applying robust methodologies for assessing complex health system interventions. The project will be implemented according to standard governance practices at the University of Leeds and University of Nigeria. An example of the latter is: ‘financial incentives, combined with increased access to MCH care following deployment of CHWs and improved infrastructure will improve the uptake of antenatal care by pregnant mothers (by over 50 %)’. Leeds, U.K: COMDIS-Health Service Delivery; 2012. As with all primary research approaches, guidance on quality assurance and uniform reporting is an important step towards improving quality and consistency. In this paper, we report a protocol for realist evaluation study of DeteRminants of Effectiveness and sustainability of a noVel Community HeAlth Workers (CHWs) programMe in imProving maternal and child health in Nigeria (REVAMP project). 2007;7(1):84. equitable access to quality MCH services and attainment of MCH outcome targets) during and after withdrawal of targeted support to the programme, Develop an empirically based and theoretically grounded dynamic model of complex relations between the actors, context, implementation process, outputs and outcomes of the interventions during, and after withdrawal, of targeted support to the programme, Assess the role of different advocacy and lobbying efforts in entrenching MCH on the political agenda and strengthening the provision of MCH services, following the suspension of targeted support to SURE-P/MCH, Develop transferable best practices for scalability (expansion within a broadly similar context) and generalisability (expansion to different contexts) of the lessons learned. Wolff N. Randomised trials of socially complex interventions: promise or peril? All interviews will be audio-recorded (subject to informed consent), transcribed and translated into English where required. Central to the supply component was the recruitment, training and deployment of 2000 formal service providers (e.g. Assessment of ITS design against quality criteria. TM, BU, OO and RH jointly conceived the study; TM, EE, BE, BU, AM, OO, RH, NE, JN and TE developed the study proposal; TM led the writing of this paper with the contributions from EE, BE, BU, AM, OO, RH, NE, JH, JN and TE. 2010;10(1):43. On the demand side, there are costs of the cash transfers themselves (conditional on using services), transportation to the health facility and opportunity costs of facility use to the women and their carers. BMJ Open 2012, 2(5). t 2012;10(1):30. We will use social science methods to explore views of key actor groups. capabilities, values and interests of individuals) [38, 39]. This paper describes our longitudinal, mixed methods realist evaluation of HNA and care planning. realist evaluation protocol Reena Devi, 1,2 Julienne Meyer , 3 Jay Banerjee, 4,5 Claire Goodman, 6 John Raymond Fletcher Gladman, 7 T om Dening, 8 Neil Chadborn, 1 Specific methods of communicating research will include combinations of: Developing short and practical policy briefs to national and international policymakers and practitioners. Two supplementary materials are included from this handbook. Evidence from elsewhere [43] is that demand-side programmes can impact on other services, through over-crowding and excessive bed-occupancy. Ethical approvals for this study were obtained from the School of Medicine Research Ethics Committee at the Faculty of Medicine and Health at the University of Leeds (ref: SoMREC/14/097) and the Health Research Ethics Committee at the University of Nigeria Teaching Hospital, Enugu (ref: NHREC/05/02/2008B-FWA00002458-1RB00002323). These hypotheses will be developed in discussions with programme managers and implementers. Malar J. In: Bryman A, Burgess R, editors. It was identified in consultation with the Federal and State Ministry of Health (MOH) and the SURE-P national team lead. Google Scholar. is a dependent outputs (e.g. The impact of the programme on a range of key output and outcome indicators will be assessed through an interrupted time-series analysis (ITS) of monthly (the unit of analysis) quantitative data from HMIS and SURE-P M&E monthly programme reports. Rather than attempting to cost all services, we will use an incremental approach that examines the additional programme benefits relative to the additional SURE-P costs. 2nd ed. PubMed  Ricketts JA. assessment of context will inform best practices in PHC staff performance management; assessment of the added value of CCTs will inform further demand-side financing schemes. UNICEF, WHO. (PDF 110 kb). Bristol: Policy Press; 2009. Article  Developing and Evaluating Complex Interventions. Terms and Conditions, 1) provides an overarching hypothesis, and more specific hypothetical pathways will represent the middle-range theories (MRT) to help us explore the C-M-O configurations within the programme. Countdown to 2015. acceptability of the intervention by the communities and front-line service providers, appropriateness of the intervention design to the current context of PHC facilities in Anambra state and sustainability of changes achieved from the implementation of interventions in the longer term [37]. 2017 Dec 22;17(1):843. doi: 10.1186/s12913-017-2795-x. Achievement of improved maternal and child health (MCH) outcomes continues to be an issue of international priority, particularly for sub-Saharan African countries such as Nigeria. A detailed list of respondents will be developed within step 1, and snowballing will be used to identify any further informants. Country Implementation Plan for Prioritized Life-Saving Commodities for Women and Children. A cluster is made up of four PHC facilities and one General Hospital (GH). [1] It is based on the epistemological foundations of critical realism. Excerpt "A realist approach assumes that programs are “theories incarnate”. 2003;19(4):613–23. Therefore, instead of attributing changes in health outcomes to the SURE-P/MCH only, we will explore the contribution of the interventions to achievement of desired effects within the real context. The making of Jamaica’s ‘National Policy for Persons with Disabilities 2000’: macro, meso and micro factors. 1, the mixed methods approach will enable us to also analyse the intervention implementation outcomes, e.g. Abuja, Nigeria: Subsidy Reinvestment and Empowerment Programme (SURE-P); 2013. [2][3], Realist evaluation techniques recognise that there are many interwoven variables operative at different levels in society, thus this evaluation method suits complex social interventions, rather than traditional cause-effect, non-contextual methods of analysis. It describes five key ideas which frame realism and their implications for impact evaluation. The Methodology and analysis This paper disseminates the protocol for an 18-month ‘RESPOND’ project that aims to evaluate the system of collecting and responding to user feedback in Bangladesh. The specific hypotheses will be driven by the study research questions (see Table 1) and will relate to the key programme targets ([40] p. 17), Reduction of MMR by 59 % from 545/100,000 live births to 320/100,000 live births, Reduction of neonatal mortality rate by 22 % from 37/1000 live births to 29/1000 live births, Increased percentage of pregnant women receiving focused antenatal care (ANC) by 52% from 50 % coverage, Increased percentage of skilled birth attendance by 63 % (from 16 % baseline), Increased postnatal care attendance within 2 days of birth by 63 % (from 16 % baseline), Increased family planning attendance by 26 % (from 1 % baseline). Implement Sci. Analyzing Qualitative Data. The panels will use data from each facility in the three study clusters covering the period from at least 12 months before and 12 months after the intervention began in the two intervention clusters and equivalent periods for the control cluster.