Feedback Instrument for Rescue Force Development – Emergency Training (FIRE-E)

Abstract

The Feedback Instrument for Rescue Force Development – Emergency Training (FIRE-E) is a specialized, four-item instrument designed to measure the perceived quality of training exercises among rescue forces in simulated acute settings. It functions as an additional module to the core Feedback Instrument for Rescue Force Development (FIRE), which is primarily used for the evaluation of leadership training courses for rescue forces.

Developed through a cooperation between the Westfälische Wilhelms-Universität Münster and the Institute of the North Rhine-Westphalia Fire Brigade (IdF NRW), the FIRE-E was validated using a sample of 375 firefighters participating in various training courses. Psychometric analysis confirmed a robust unifactorial structure via Confirmatory Factor Analysis, demonstrated excellent internal consistency (McDonald’s Ω = .87), and provided strong empirical evidence supporting both construct and criterion validity, confirming its utility as a reliable evaluation tool for practical emergency simulations.

Keywords

Fire service, rescue forces, emergency training, evaluation, practical exercises, leadership training, IdF NRW

Authors

Nils Schulte, Markus T. Thielsch

Purpose

The primary purpose of the FIRE-E scale is the systematic evaluation of the quality of practical emergency exercises (Einsatzübungen) utilized in the training and development of rescue personnel. It provides critical, subjective feedback from the participants’ perspective, enabling training providers to assess and improve their teaching methodologies concerning simulated acute scenarios and high-stakes decision-making processes.

The instrument is designed to capture quality characteristics relevant to the specific teaching method of emergency simulations, focusing on features that are independent of the exercise’s specific implementation. While typically appended as a supplementary module to the comprehensive FIRE core questionnaire, the FIRE-E can also be used independently for the evaluation of basic rescue training for personnel without leadership responsibilities (e.g., alongside the FIRE-B), or in adapted form for evaluating staff exercises.

Construct

The FIRE-E measures the participants’ subjective assessment of the quality of practical emergency exercises within rescue force training. This construct is vital, as the challenging nature of emergency work necessitates continuous and high-quality training. The scale captures four key dimensions of effective simulation quality:

• The perceived subjective learning gain.
• The utility and quality of feedback received from instructors.
• The ability to apply newly acquired theoretical knowledge practically.
• The appropriateness of the exercise’s difficulty level, ensuring participants are challenged but not overwhelmed.

The theoretical background posits that high-fidelity simulations are critical for developing the practical experience and domain knowledge required for effective leadership in acute situations. Furthermore, structured debriefing and instructor feedback are essential components of these exercises, facilitating reflection and initiating self-directed learning from simulated experience.

Validity

The validity of the FIRE-E was established through three structured studies, demonstrating strong evidence across content, construct, and criterion dimensions.

Content Validity: Initial item relevance and comprehensibility were assessed by both instructors and course participants (Study I, N=33). All items were rated as important or rather important by over 90% of the sample, confirming strong content validity regarding the essential aspects of practical training quality.

Construct Validity: Convergent validity was assessed by correlating FIRE-E scores with related factors of the FIRE core questionnaire. High correlations were found with Instructor Behavior (r = .60), Competence Acquisition (r = .53), and Preparation & Transfer (r = .62). Divergent validity was supported by a non-significant correlation with participants’ entrance examination grades (r = -.04) and a moderate correlation with bias variables like mood (r = .34), suggesting the scale measures specific training quality rather than general sentiment or prior academic performance.

Criterion Validity: The scale showed robust correlations with key outcome criteria. High FIRE-E scores correlated strongly with overall participant satisfaction (r = .72) and moderately with the positive grading of the seminar (r = .45). Furthermore, the scale showed large correlations with the perceived transfer of theoretical knowledge into practice (r = .64) and the extent to which participants learned from their own experiences during the exercises (r = .55).

Reliability

The internal consistency of the FIRE-E scale is considered high, supported by analysis across two independent validation samples (Study II and Study III).

Since a congeneric measurement model was confirmed, McDonald’s Omega (ω) was used as the appropriate coefficient for internal consistency. In both Study II (N = 155) and Study III (N = 375), the McDonald’s ω coefficient was calculated at .87. This value indicates good reliability and suggests that the four items consistently measure the intended construct of emergency exercise quality.

Factor Analysis

The factor structure of the FIRE-E was analyzed using a combination of Exploratory and Confirmatory Factor Analysis (CFA).

During preliminary EFA (Study II, N=155) of various teaching method items, the four FIRE-E items were temporarily excluded due to limited variance (floor effects) and the low number of remaining items after initial exclusions. However, based on theoretical alignment with other validated modules (like Plan Exercises), the four items were hypothesized to form a single factor.

This hypothesized unifactorial structure was rigorously tested using Confirmatory Factor Analysis (CFA) employing Maximum-Likelihood estimation (Study III, N=375). The CFA results demonstrated an excellent model fit (Χ²(2) = 3.40; CFI = .998; TLI = .994; RMSEA = .04; SRMR = .01). Furthermore, all standardized factor loadings exceeded .70 and were statistically significant (p < .001). These results strongly support the conceptualization of the FIRE-E as a single-factor scale measuring the overall quality of emergency training exercises.

Instrument

Test Type: Questionnaire/Rating Scale (Evaluation Instrument)

Format: Paper & Pencil or Online Questionnaire

Language Available: German (Deutsch)

Population Group: Rescue Forces (e.g., Firefighters, civil defense personnel, police, military)

Age Group: Adults (Validation sample ages ranged from 20 to 55 years)

Population Details: The scale was developed and validated primarily among participants (N=375 in Study III) attending group and platoon leader training courses at the Institute of the North Rhine-Westphalia Fire Brigade (IdF NRW). The sample was predominantly male (94.4%) and included individuals affiliated with volunteer and professional fire service.

Test Methodology: The FIRE-E consists of 4 positively polarized items measured on a 7-point Likert scale format. Response options range from 1 = <em>stimme gar nicht zu</em> (strongly disagree) to 7 = <em>stimme vollkommen zu</em> (strongly agree), with an additional option for <em>nicht beantwortbar</em> (not applicable). The score is calculated as the mean of the four item scores. Interpretation is straightforward: higher mean values indicate a more positive evaluation of the practical exercise quality. Processing time is estimated by the authors to be highly economical, requiring approximately 1 minute.

Keywords

Evaluation, emergency services, simulated training, psychometrics, internal consistency, Confirmatory Factor Analysis, rescue forces

Authors

Author ORCID Identifier: Not specified in source content.

Affiliation Email addresses: Not specified in source content.

Correspondence Address: Further information regarding the scale and related projects is available at the University of Münster website: https://www.uni-muenster.de/OWMS/bfo/projekte/fire/

Permissions & Fee and Test Year

The FIRE-E scale is a validated evaluation instrument developed through academic and organizational cooperation. The scale development and primary validation studies occurred between 2015 and 2017. The scale is available as a module for evaluation purposes. To ensure response integrity, anonymity must be guaranteed for participants during administration and evaluation, and aggregated results should only be reported when a sufficient number of completed questionnaires (e.g., minimum of eight) are available.

The scale was developed in collaboration with the Institute of the Fire Brigade of North Rhine-Westphalia (IdF NRW).

Reference’s

• Schulte, N. & Thielsch, M. T. (2019). Evaluation of Firefighter Leadership Trainings. International Journal of Emergency Services, 8, 34-49. https://doi.org/10.1108/IJES-03-2018-0020
• Thielsch, M. T. & Hadzihalilovic, D. (in press). Evaluation of Fire Service Command Unit Trainings. International Journal of Disaster Risk Science.

Items of the Feedback Instrument for Rescue Force Development – Emergency Training (FIRE-E)

IMPORTANT: The following scale items must be preserved in their original language and must not be changed in any way.

Antwortvorgaben

Siebenstufiges Antwortformat mit den Optionen 1 = <em>stimme gar nicht zu</em>, 2 = <em>stimme nicht zu</em>, 3 = <em>stimme eher nicht zu</em>, 4 = <em>neutral</em>, 5 = <em>stimme eher zu</em>, 6 = <em>stimme zu</em>, 7 = <em>stimme vollkommen zu</em>. Ein zusätzliches Feld bietet die Möglichkeit anzugeben, dass das jeweilige Item <em>nicht beantwortbar</em> sei.