Lessons learned in evaluating system interdependencies using qualitative methods.

This article shares lessons learned while evaluating the system interdependencies for a clinical and translational research centre (CTR). It explores the methodological challenge of discussing system concepts (e.g.

Evaluation Utility Metrics (EUMs) in Reflective Practice.

The article proposes three evaluation utility metrics to assist evaluators in evaluating the quality of their evaluation. After an overview of reflective practice in evaluation, the different ways in which evaluators can hold themselves accountable are discussed. It is argued that reflective practice requires evaluators to go beyond evaluation (i.

Using the Homeland Security Exercise and Evaluation Program (HSEEP) Building Block Approach to Implement System Evaluation Theory (SET).

This article shares lessons learned in applying system evaluation theory (SET) to evaluate a Clinical and Translational Research Center (CTR) funded by the National Institutes of Health. After describing how CTR support cores are intended to work interdependently as a system, the case is made for SET as the best fit for evaluating this evaluand. The article then details how the evaluation was also challenged to facilitate a CTR culture shift, helping support cores to move from working autonomously to working together and understanding how the cores' individual operating processes impact each other.

COVID-19: Exposing the need for emergency management to invest in systems thinking.

The COVID-19 pandemic exposed numerous challenges in the emergency management (EM) response system. The article contends that had EM deliberately and systematically engaged in systems thinking; it would have been better able to anticipate and respond to many of the challenges. Reasons for EM not fully embracing systems thinking are discussed, including the perception that it is complex and theoretical.

Lessons Learned in Evaluating the Infrastructure of a Centre for Translational Research (CTR).

This paper shares lessons learned while evaluating the implementation of a Clinical and Translational Research Center (CTR). To meet its overarching goals the CTR consists of numerous research support units (e.g.

Using spatial regression methods to evaluate rural emergency medical services (EMS).

Emergency Medical Services (EMS) are acute services provided outside of the hospital. EMS are crucial in rural environments where hospitals are often far away and difficult to access. Establishing EMS performance measures is critical in improving a rural community's access to these services and eliminating systemic inequalities.

Using systems evaluation theory to improve points of dispensing planning, training, and evaluation.

Recurring throughput problems served as a catalyst to search for a better approach to planning, training, and evaluating points of dispensing (POD) exercises. The paper begins with a discussion of the potential of systems thinking and systems theory to improve exercise planning, training, and evaluation. The paper then illustrates how systems concepts guided a POD design team in improving their planning, training, and evaluation strategy.

Being smart about writing SMART objectives.

This article challenges the conventional wisdom in mainstream evaluation regarding the process for developing specific, measurable, attainable, relevant, and time-bound (SMART) objectives. The article notes several advantages of mainstreaming the SMART method including program capacity building and being able to independently monitor progress toward process and outcome objectives. It is argued the one size fits all approach for writing SMART objectives is misleading.

The power of the context map: Designing realistic outcome evaluation strategies and other unanticipated benefits.

Developing a feasible evaluation plan is challenging when multiple activities, often sponsored by multiple agencies, work together toward a common goal. Often, resources are limited and not every agency's interest can be represented in the final evaluation plan. The article illustrates how the Antecedent Target Measurement (ATM) approach to logic modeling was adapted to meet this challenge.

Lessons learned in testing the feasibility of evaluating transfer of training to an operations setting.

Introduction: The purpose of the study was to explore the feasibility, identify challenges, and offer solutions to evaluating transfer of training to the operations setting.

Background: The assumption underlying public health emergency preparedness training is competencies and capabilities will transfer to the operations setting. However, there are no studies describing methods for evaluating the transfer of training.

Emergency preparedness training of tribal community health representatives.

This study describes the development and evaluation of online Public Health Emergency Preparedness (PHEP) training adapted to the learning styles and needs of tribal Community Health Representatives (CHRs). Working through a university-tribal community college partnership, the Arizona Center for Public Health Preparedness at the University of Arizona and Diné College of the Navajo Nation delivered a blended online and face-to-face public health preparedness certificate program based on core public health emergency preparedness competencies. This program was carefully adapted to meet the environmental and learning needs of the tribal CHRs.

Utilization of the Native American Talking Circle to teach incident command system to tribal community health representatives.

The public health workforce is diverse and encompasses a wide range of professions. For tribal communities, the Community Health Representative (CHR) is a public health paraprofessional whose role as a community health educator and health advocate has expanded to become an integral part of the health delivery system of most tribes. CHRs possess a unique set of skills and cultural awareness that make them an essential first responder on tribal land.

A practical framework for evaluating a culturally tailored adolescent substance abuse treatment programme in Molokai, Hawaii.

Objective: Successful substance abuse treatment requires many changes in behavior, attitude and skills. Culturally tailored approaches to substance abuse treatment have shown initial success, but are not yet accepted as best practice models. In order to document programme effectiveness of a new culturally tailored substance abuse treatment programme on the rural island of Molokai, Hawaii, the authors worked to develop a multi-level evaluation plan to measure behavior changes occurring after participation in activities targeting identified causes of substance abuse in the population of interest.

How using a logic model refined our program to ensure success.

When a program does not produce expected changes in the outcomes of interest, program staff must understand why before making programmatic decisions. One mechanism for doing so is the logic model. This article describes how using one such logic model (the ATM approach) was used to improve areas of program planning and implementation.

Self-efficacy for health-related behaviors among deaf adults.

The purpose of this quasi-experimental, pre-post-test study was to test the effectiveness of the Deaf Heart Health Intervention (DHHI) in increasing self-efficacy for health-related behaviors among culturally deaf adults. The DHHI targets modifiable risk factors for cardiovascular disease. A sample of 84 participants completed time-1 and time-2 data collection.

Identifying antecedent conditions responsible for the high rate of mining injuries in Zambia.

The incident rates of mining-related accidents and injuries in developing countries exceed those of developed nations. Interventions by international organizations routinely fail to produce appreciable long-term improvement. One major reason is the inability to identify and analyze the underlying factors responsible for creating unsafe working conditions.

Deaf community analysis for health education priorities.

Deaf persons' access to health-related information is limited by barriers to spoken or written language: they cannot overhear information; they have limited access to television, radio, and other channels for public information; and the average reading level of Deaf adults is at a 3rd to 4th grade level. However, literature searches revealed no published reports of community analysis focusing specifically on health education priorities for Deaf communities. A seven-step community analysis was conducted to learn the health education priorities in Arizona Deaf communities and to inform development of culturally relevant health education interventions in Deaf communities.

Osteoporosis prevention: knowledge and behavior in a southwestern community.

Prior to developing an osteoporosis prevention education program and social marketing campaign, we sought to (a) establish current status of osteoporosis-related knowledge and behavior among women aged 25-55 years in Maricopa County, Arizona, and (b) assess factors that segment the population by age and ethnicity. Two-hundred women were surveyed by telephone using random-digit dialing selection. Data demonstrated knowledge of need to consume adequate calcium, but mixed understanding of forms of exercise that help prevent osteoporosis.

Assessing the effectiveness of a community-based media campaign targeting physical inactivity.

The goal of this study was to develop, implement, and evaluate a community-based effort addressing the problem of physical inactivity. Using Prochaska's Transtheoretical Model as a guide, community members developed television and worksite media messages focusing on the benefits and barriers of physical activity and on increasing self-efficacy. The media campaign was effective in changing perceived barriers, perceived benefits, and self-efficacy surrounding physical activity and had an unexpected effect of changing behavior.