Feedback Instrument for Rescue Force Development (FIRE)

Abstract

The Feedback Instrument for Rescue Force Development (FIRE) is a 21-item questionnaire designed to measure and evaluate the quality of specialized training programs for rescue forces, with a particular focus on leadership development within the fire service. The instrument’s development was based on qualitative analyses identifying key characteristics of effective training in this domain. Items were rigorously tested for relevance and comprehensibility in initial studies conducted at the Institute of the North Rhine-Westphalia Fire Brigade (IdF NRW).

An Exploratory Factor Analysis (EFA) reduced the scale to 21 core items, identifying six factors: four related to learning processes (Lecturer Behavior, Requirement Level, Structure, and Group) and two related to learning outcomes (Competence Acquisition and Transfer). Subsequent validation using Confirmatory Factor Analysis (CFA) confirmed this factor structure with acceptable to good model fit. The FIRE scales demonstrate strong psychometric properties, exhibiting acceptable to good internal consistency (Cronbach’s Alpha $ge$ .73) and clear evidence of construct validity and criterion validity.

Keywords

Firefighter training, Training evaluation, Rescue services, Leadership development, Psychometric scale, Competence acquisition, Transfer, IdF NRW, CFA, EFA

Authors

Niklas Schulte, Stephanie Babiel, Marius Messinger, Meinald Thielsch

Purpose

The primary purpose of the FIRE instrument is to facilitate the systematic evaluation and quality assurance of training programs for active and aspiring leadership personnel within the rescue forces. It aims to provide instructors and organizational leaders with structured, multi-dimensional feedback from course participants regarding the effectiveness and quality of the instruction received.

By measuring both the perceived quality of the learning environment (process variables) and the resulting skills and preparedness (outcome variables), FIRE enables training organizations to identify specific strengths and weaknesses in their curriculum delivery. Although developed within the German fire service context, the instrument is explicitly designed to capture general characteristics of good training, making it transferable for use in evaluating training modules in other high-stakes organizations such as the police and military.

Construct

The FIRE scale measures the multi-faceted construct of Feedback for Rescue Force Development, integrating principles from Kirkpatrick’s Model (Reaction and Learning levels) and Rindermann’s Multidimensional Model of Teaching Success. The core questionnaire is structured around six empirically validated subscales:

• Lecturer Behavior: Evaluates the instructor’s effectiveness in motivating students, providing concise explanations, and offering useful feedback.
• Requirement Level: Assesses the perceived difficulty, pace, and volume of course material, indicating whether participants feel overwhelmed.
• Structure: Measures the clarity, organization, and coherence of the instructional content and course flow.
• Group: Captures the quality of peer interaction and collective support among course participants.
• Competence Acquisition: Focuses on the participant’s self-assessed gain in critical skills, such as recognizing hazards, making decisions under pressure, and understanding personal limitations.
• Transfer: Measures the perceived readiness and confidence to apply acquired knowledge and skills directly to future operational leadership roles in the field.

Validity

Validation efforts demonstrated strong evidence for the utility and accuracy of the FIRE instrument.

Content Validity: Content validity was ensured through an iterative development process starting with qualitative interviews of trainers and participants at the IdF NRW to define success-critical criteria for rescue force training. The resulting items were then critically reviewed and rated for completeness and relevance by subject matter experts and course attendees before final selection.

Construct Validity: Factorial validity was supported by both EFA and CFA, confirming that the six distinct dimensions accurately reflect the underlying theoretical structure. Convergent validity was established by correlating the FIRE subscales with established teaching evaluation instruments (HILVE and TRIL), showing consistently positive correlations across corresponding measures of teaching competence, structure, and learning outcomes. This differentiation between factors suggests a low impact of common method biases like Halo effects.

Criterion Validity: All FIRE subscales showed medium to large positive correlations with external measures of overall course satisfaction and the participants’ formal grading of the course. These correlations support the theory that high scores on the FIRE dimensions predict favorable overall evaluation and positive subjective outcomes of the training experience.

Reliability

The internal consistency of the FIRE instrument was assessed using data from Study IV (N=382). The reliability coefficients indicate that the scales are sufficiently consistent for use in training evaluation, with reliability classified as acceptable to good according to established criteria for evaluation instruments.

• The preferred reliability measure, McDonald’s Omega h ($omega_h$), ranged from .76 (Transfer) to .86 (Requirement Level).
• Cronbach’s Alpha ($alpha$) coefficients ranged from .73 (Lecturer Behavior) to .86 (Requirement Level).

The reliability indices confirm that the subscales provide stable and consistent measurements of their respective constructs within the target population of rescue force leaders.

Factor Analysis

The factor structure was determined through sequential factor analytic studies. Study III employed EFA (N=243) on initially large sets of process and outcome items. The analysis identified four factors related to learning processes (Lecturer Behavior, Requirement Level, Structure, Group) and two factors related to learning outcomes (Competence Acquisition, Transfer), leading to the final 21-item core questionnaire.

Study IV utilized CFA (N=382) to cross-validate the six-factor structure. The CFA models demonstrated acceptable to good fit indices (e.g., CFI $ge$ .96; RMSEA $le$ 0.07 for both process and outcome factors), confirming the dimensional structure identified in the EFA. These results validate the instrument’s ability to measure distinct latent variables related to training quality.

Instrument

Test Type: Self-report Questionnaire / Training Evaluation Instrument

Format: The FIRE core questionnaire consists of 21 items, typically administered immediately following the training module. It can be used in both paper-based and digital (online) formats. Optional items (22-25) measuring global judgments and open feedback are often included but do not form part of the psychometric scale structure.

Language Available: German (deutsch), English

Population Group: Active and aspiring leadership personnel (Group/Platoon leaders) within emergency services, including fire brigades, police, and military organizations.

Age Group: Adults (age range of development samples: 20 to 56 years).

Population Details: The scale was developed and validated on participants of leadership training courses at the Institute of the Fire Brigade North Rhine-Westphalia (IdF NRW). Samples predominantly consisted of men (95%+) from professional and volunteer fire services, reflecting the demographic characteristics of the population attending these specific leadership courses.

Test Methodology: Items 1-22 use a 7-point Likert scale (1 = stimme gar nicht zu [strongly disagree] to 7 = stimme vollkommen zu [strongly agree]), plus an option for “not meaningfully answerable.” Item 23 uses a 6-point school grade format (1 = very good, 6 = insufficient). Scoring involves calculating item means and then scale means, with items on the Requirement Level scale being negatively coded before aggregation to ensure that higher scores consistently reflect a more positive evaluation of the training.

Keywords

Evaluation, Fire brigade, Emergency services, Training quality, Leadership, Competence, Transfer, Lecturer behavior, Internal consistency, Psychometrics

Authors

Author ORCID Identifier: Not provided in source content.

Affiliation Email addresses: [email protected] (Niklas Schulte), [email protected] (Stephanie Babiel), [email protected] (Marius Messinger), [email protected] (Meinald Thielsch)

Correspondence Address: Niklas Schulte, M.Sc., Westfälische Wilhelms-Universität Münster, Institut für Psychologie.

Permissions & Fee and Test Year

The FIRE instrument was developed between 2015 and 2017 in collaboration between the IdF NRW and the University of Münster. The primary publication describing the instrument is Schulte & Thielsch (2019). The authors may provide access to related underlying development work (Babiel, 2016; Sievers, 2015) upon request.

References

Schulte, N., & Thielsch, M. T. (2019). Evaluation of firefighter leadership trainings. International Journal of Emergency Services, 8, 34-49.

Thielsch, M. T., Busjan, J. N., & Frerichs, K. (2018). Feedback-Instrument zur Rettungskräfte-Entwicklung – Prüfungen (FIRE-P) [Feedback instrument for rescue forces education – Examinations (FIRE-P)]. Zusammenstellung sozialwissenschaftlicher Items und Skalen.

Thielsch, M. T., Kläpker, M. & Streppel, D. (2019). Feedback-Instrument zur Rettungskräfte-Entwicklung – Basis (FIRE-B) [Feedback instrument for rescue forces education – Basic (FIRE-B)]. Zusammenstellung sozialwissenschaftlicher Items und Skalen.

Kirkpatrick, D. L. (1979). Techniques for evaluating training programs. Training and Development Journal, 33(6), 78–92.

Rindermann, H. (2001). Das Heidelberger Inventar zur Lehrveranstaltungsevaluation (HILVE). Empirische Pädagogik, 15(4), 481-512.

Items of the Feedback Instrument for Rescue Force Development (FIRE)

IMPORTANT: The following scale items must be preserved in their original language and must not be changed in any way.

Tabelle 1

FIRE-Items: Kernfragebogen