Program Evaluation & Leadership
Room: Triple Crown I, II, & III
Tuesday, April 21, 2015, 2:00 to 3:30 PM
Moderator: Christopher Maylahn, MPH
Rachel Tabak, PhD
Assessing Capacity for Sustainability of Effective Programs and Policies in Local Health Departments
Co-Investigator(s): Kathleen Duggan, MPH, MS; Carson Smith, MPH, MPA; Kristelle Aisaka, BS; & Ross Brownson, PhD
Background: Sustainability has been defined as the existence of structures and processes that allow programs to leverage resources to effectively implement and maintain evidence-based public health (EBPH). Sustainability is important in local health departments (LHDs) to retain the benefits of effective programs. Research Objective: To explore the eight domains of Luke et al.’s Program Sustainability Framework, organizational capacity, program adaptation, program evaluation, communications, strategic planning, funding stability, political support, and partnerships, as they apply to high-and low-capacity LHDs. Data Sets and Sources: The case study sample was selected using an Administrative-Evidence Based Practice (A-EBP) score from a 2012 national survey and was linked to secondary data from the National Public Health Performance Standards Program. The online survey was delivered to a national sample of LHD practitioners, administrators, and directors (n=849, 57% response rate). Study Design: Qualitative case studies were conducted with 35 practitioners in six LHDs across the United States. Three LHDs that scored in the top quartile of both the A-EBPs and the National Public Health Performance Standards measures (defined as ‘high-capacity’) and three LHDs with scores in the bottom quartile of both measures (defined as ‘low-capacity’) were selected as case study sites. The interview guide explored LHD use of an evidenced-based decision making process, including A-EBPs and evidence-based programs and policies. Analysis: Standard qualitative methodology was used for data coding and analysis using NVIVO software. Thematic reports were produced that explored the eight domains of the Program Sustainability Framework. Principal Findings: In the organizational capacity domain, high-capacity LHDs reported better access to and support for resources, including adequate staff and staff training, compared to low-capacity LHDs. When high-capacity LHDs described program adoption, they discussed these more as opportunities to adapt and evaluate. Low-capacity LHDs struggled with programs requiring adaptation. Although high-capacity LHDs described integration of program evaluation processes into implementation and sustainability, low-capacity LHDs reported limited capacity for measurement specifically and evaluation generally. High-capacity LHDs also described higher quality communication compared to low-capacity LHDs; much of the difference was from above, particularly department leaders and program managers. High-capacity LHDs were more likely to report efforts to integrate EBPH into strategic planning and also reported having greater flexibility with their funding than low-capacity LHDs. In addition, high-capacity LHDs were more likely to describe having political support, while low-capacity LHDs reported that it was lacking. Partnerships were important to both groups and they described building partnerships as an important part of sustaining programming. Conclusions: Organizational capacity, program evaluation, program adaptation, communication, and funding stability are associated with whether a LHD is able to sustain programming. Increased top-down communication and program evaluation could help to build an internal agency culture that research suggests would be more resilient to external forces, such as funding instability or changing political environments. Implications for Public Health Practice: Modest investments in leadership support for improving organizational capacity and communication from the top of the organization down, integrating program evaluation into implementation, and greater flexibility in funding may enhance the sustainability of evidence- based programming in LHDs.
Ross Brownson, PhD
Mis-Implementation in Public Health Practice
Co-Investigator(s): Peg Allen, PhD; Rebekah Jacob, MPH; Jenine Harris, PhD; Kathleen Duggan, MPH, MS; Pam Hipp, MPH; & Paul Erwin, MD, DrPH
Background: Mis-implementation in public health practice refers to both the de-adoption of effective programs, policies, or other interventions that should continue and to the continuation of ineffective interventions that should end. Mis-implementation in public health is poorly characterized and will provide important information for decision makers. Research Objective: The objective was to describe the frequency and patterns in mis-implementation of programs in state and local health departments in the United States. Data Sets and Sources: Data are derived from two cross-sectional surveys of the state and local public health workforce as part of ongoing research projects. Study Design: A total of 944 public health practitioners was studied. The sample included state (n = 277) and local health department employees (n = 398) and key partners from other agencies (n = 269). Data were collected from October 2013 through June 2014 (analyzed in May through October 2014). Online survey questions focused on ending programs that should continue, continuing programs that should end, and reasons for endings. Analysis: Descriptive statistics were calculated for the survey’s three core questions, reported as percentages with 95% confidence intervals. The sample characteristics were derived from the survey data and from archival data for each health department using population size of jurisdiction, and local health department governance structure. Principal Findings: Among persons working in a state health department, 37% reported that programs often or always end that should have continued, compared with 42% of respondents in local health departments, and 38% of respondents working in other agencies. In contrast to ending programs that should have continued, 25% of state respondents reported programs often or always continuing when they should have ended, compared to 29% for local health departments and 25% of respondents working in other agencies. Certain reasons for program endings differ at the state versus local level (e.g., policy support, support from agency leadership), suggesting that actions to address mis-implementation are likely to vary. Conclusions. The current data suggest a need to focus on mis-implementation in public health practice in order to make the best use of scarce resources. It is likely that research on public health mis-implementation is in its first generation. This suggests a wide range of future research needs, ranging from broad categories of inquiry such as developing reliable and valid measures of mis-implementation, understanding variables that predict mis-implementation, describing mediators and moderators (e.g., state vs. local differences, understanding how health care reform efforts may affect mis-implementation of clinical services, variations by program area, the role of media attention), and developing qualitative case studies of successful and unsuccessful mis-implementation. Implications for Public Health Practice and Policy: Developing the evidence base for mis-implementation in public health practice will allow practitioners and policy makers to bolster efforts to continue effective programs and target ineffective programs for discontinuation, diverting these resources to promising programs that are not being fully implemented, evaluated, or scaled up. Such actions are likely to make the most effective use of resources and improve the health of the public.
Katie Gardner, BS
Examining Process and Outcomes of Nuisance Inspection and Abatement Conducted by Local Health Departments
Co-Investigator(s): Sara Tillie, BA & Scott Frank, MD, MS, BA
Background: The Ohio Public Health Futures Report produced by the Association of Ohio Health Commissioners represents the Governor’s Office of Health Transformation (OHT) effort to reform the local public health system. The development and adoption of “minimum package” of essential public health services, which included both core services and foundational capabilities that all local health departments (LHDs) should possess. Nuisance abatement is an essential, mandated, unfunded public health service. Under Ohio revised code[i] LHDs are charged with “the abatement and removal of nuisances.” However, there has been no formal effort to quantify, categorize, or standardize nuisance abatement efforts. Research Objective: 1) Investigate conduct and nature of nuisance inspection and abatement through direct observation; 2) Identify differences in nuisance inspection and abatement process and outcomes through variation of service delivery and Environmental Health Specialists (EHS) characteristics. Data Sets and Sources: Data used for this research includes abstraction of routinely collected nuisance abatement services data (n=509), stored in the Health Department Information System. Original data documenting the process of nuisance abatement was produced from observational protocol forms administered by trained observers at 6 diverse LHDs to assess needs associated with nuisance abatement services (n=167). A survey profile of participating EHS helps understand variation in process (n=27). Study Design: This comparative case study utilized mixed methods including survey, interview, direct observation, and data abstraction. Analysis: Univariate descriptive analysis of the observational and abstracted data was performed. Tests of association were utilized to detect differences based on LHD nuisance abatement process and EHS characteristics. Analysis was conducted using SPSS v.22. EHS characteristics will be used to focus on identifying process variation across LHDs, including demographics, experience, and attitudes regarding nuisance abatement. Research questions will focus on nuisance inspection type; source and object of complaint; and outcomes. Principal Findings: The top five nuisance complaints based on direct observation and abstraction of nuisance reports were: property (32%); animal (28%); air (18%); water (18%); and trash/garbage (17%). Complaints were most likely to come from residents (64%) followed by anonymous or unknown (22%) and government employees (14%); and most often were directed toward residential (63%); commercial (12%); and public (3%) property. EHS characteristics and attitudes were related to the process and outcome of nuisance inspection. Conflict occurred during 16% of inspections; punitive actions were threatened in 11%. Inspection outcomes included immediate remediation (58%), citation (15%), written orders (18%), and verbal warnings (11%). Further inspection was warranted 28% of the time. Conclusions: This study represents an effort to subject the process and outcomes of nuisance inspection and abatement to research scrutiny. Outcomes of this essential component of environmental public health service vary based on process, EHS characteristics, and EHS attitudes. Implications for Public Health Practice and Policy: Understanding the nature and process nuisance inspection and abatement will inform decision-makers regarding future environmental health practice. [i] Mandated and Permitted Services and Governance/Administrative Provisions in the Ohio Revised Code and Ohio Administrative Code. Appendix D in Public Health Futures: Considerations for a New Framework for Local Public Health in Ohio. http://aohc.affiniscape.com/associations/4594/files/PHF_FullReport_Final....
Carmen Nevarez, MD, MPH (presented by Craig Sewald)
An Intersectoral Leadership Learning Academy: Impact on Multi-sector Teams Engaged in Community Health Improvement Projects
Background: For the past three years, the CDC has funded the National Leadership Academy for the Public’s Health. The goal is to provide ongoing leadership training and support to leaders that work across sectors to address community health issues. The academy also aims to build and support the development of a community of multi-sector leadership learners, using online technology and social media tools that contribute to community health improvement goals and influence policy change. Research Objective: Assess the impact of team participation in the leadership academy on the development of intersectoral leadership skills and capacity, and its impact on the achievement of the team’s project goals. Data Sets and Sources: Data was derived from team applications; surveys of individual participants and teams at baseline, mid-term and end of participation; post-webinar and retreat evaluations; coach assessments of team progress at mid-term and end of the program ; post-participation interviews; and information provided by teams and coaches throughout the academy to monitor progress in leadership learning and on project goals;. Study Design: Teams were recruited through a variety of communication networks. Each team was required to have four members, one from a governmental public health agency at the state or local level, the others from other sectors of the community. The teams were required to have a track record of having worked together, and to have chosen a community health improvement project. The program included an on-site retreat, a series of webinars, a coach assigned to each team, a site visit and monthly calls with the coach, and a variety of leadership learning tools. Teams were required to submit data throughout the year to document progress in intersectoral leadership learning and on the team projects. Analysis: An evaluation consultant manages the evaluation of NLAPH. The analysis of qualitative and quantitative data is used to produce a report following each academy year documenting results from the evaluation. The reports have been used as the foundation for improving and refining the academy structure and curriculum. Principal Findings: The academy has been successful in developing the leadership skills that teams and team members need to address the challenges encountered in community health improvement efforts. In addition, for many teams, the academy has resulted in accelerated progress in achieving project goals. Data from the various cohorts have been useful in further refining the framework and content of the academy to ensure that it is truly addressing the leadership challenges teams are encountering in their intersectoral work. Conclusions and Implications: Teams engaged in multisector community health improvement projects can improve their leadership skills and accelerate the achievement of project goals by participating in a yearlong leadership learning experience focused on intersectoral leadership. Teams have been able to use skills and practices developed during the academy to benefit their ongoing community health improvement work.