Feedback Instrument for Rescue Force Development – Tabletop Exercises (FIRE-PL)

Abstract

The Feedback Instrument for Rescue Force Development – Tabletop Exercises (FIRE-PL) is a specialized, four-item supplementary module designed to measure the perceived quality and effectiveness of tabletop exercises used in the training of rescue forces. This instrument serves as an extension of the broader Feedback Instrument for Rescue Force Development (FIRE), a comprehensive questionnaire utilized for the evaluation of leadership training courses for emergency personnel.

The FIRE-PL was developed and validated in cooperation with the Institute of the North Rhine-Westphalia Fire Brigade (IdF NRW), based on data collected from 288 firefighters participating in training programs. A rigorous Confirmatory Factor Analysis (CFA) supported a robust unidimensional structure. The scale demonstrated high internal consistency (Cronbach’s Alpha ≥ .81; McDonald’s Omega ≥ .82) and provided clear indicators of construct validity and criterion validity, affirming its utility as an objective and economical tool for evaluating critical learning methods in emergency services leadership training.

Keywords

Fire service, rescue forces, tabletop exercise, evaluation, leadership training, questionnaire, psychometric scale, IdF NRW.

Authors

Stefan Röseler, M.Sc., Prof. Dr. Dipl.-Psych. Meinald Thielsch.

Purpose

The primary purpose of the FIRE-PL instrument is to systematically evaluate the quality of tabletop exercises (Planübungen) as a specific teaching method within the leadership training curriculum for rescue forces. While the overarching FIRE questionnaire assesses general dimensions of training quality (e.g., instructor behavior, competence acquisition), the FIRE-PL focuses specifically on the perceived efficacy of these simulation exercises, which are crucial for developing decision-making skills under stress.

This targeted evaluation helps instructors and training institutions, such as the IdF NRW, gather specific feedback regarding the learning gain, the quality of feedback provided by instructors, the applicability of acquired knowledge, and the appropriate difficulty level of the exercises, facilitating continuous improvement in specialized emergency training methods.

Construct

The psychological construct measured by the FIRE-PL scale is the Perceived Quality of Tabletop Exercises. This single, unidimensional construct captures the trainees’ subjective assessment of the effectiveness of the exercise format in promoting learning and skill application relevant to complex operational decision-making.

The four items comprising the scale were developed to address key components necessary for effective training: 1) subjective learning gain, 2) benefit from instructor feedback, 3) ability to apply newly acquired knowledge, and 4) appropriateness of the difficulty level. The scale is applicable not only in the fire service context but also potentially in other rescue organizations like civil protection or military training, given its focus on general quality characteristics of this teaching method.

Validity

The validation studies (Study I, II, and III) provided strong evidence supporting the scale’s validity across multiple criteria.

Content Validity: Initial review (Study I, N=33) by instructors and trainees at the IdF NRW confirmed the items’ comprehensibility and relevance. Experts deemed the aspects contained within the items relevant for learning success, leading to the conclusion that the scale possesses strong content validity.

Construct Validity: Convergent measures utilizing subscales from the core FIRE questionnaire demonstrated high, significant correlations with FIRE-PL scores. Specifically, high correlations were found with Instructor Behavior (r = .53), Competence Acquisition (r = .40), and Preparation & Transfer (r = .41), reflecting the importance of instructor guidance and the exercise’s role in knowledge transfer. Divergent measures showed appropriately lower correlations, such as Group atmosphere (r = .25) and participant mood (r = .20), confirming that the scale accurately isolates the quality of the tabletop exercise construct.

Criterion Validity: The FIRE-PL scores showed a large correlation with overall course satisfaction (r = .49, p < .001) and a moderate correlation with the seminar grade assigned by participants (r = .32, p < .001). This indicates that the perceived quality of the tabletop exercises is a significant predictor of overall positive training outcomes.

Reliability

The FIRE-PL demonstrated high internal consistency across the validation samples (Study II and III).

Since the measurement model was identified as congeneric in both studies, McDonald’s Omega (ω) was deemed the most adequate coefficient for assessing internal consistency. The calculated reliability coefficients were consistently strong: in Study II (N=155), Cronbach’s Alpha was .86 and McDonald’s Omega was .86. In Study III (N=288), Cronbach’s Alpha was .81 and McDonald’s Omega was .82. These values confirm the scale’s excellent reliability.

Factor Analysis

The underlying factor structure of the FIRE-PL was confirmed through exploratory and confirmatory analyses, supporting its intended unidimensionality.

Initially, an Exploratory Factor Analysis (EFA) in Study II (N=155) indicated that the four items related to tabletop exercises loaded strongly onto a single distinct factor. Subsequently, a Confirmatory Factor Analysis (CFA) was performed in Study III (N=288) to test the proposed one-factor structure. The CFA results yielded a good fit for the model (χ²(2) = 4.72, CFI = .99, TLI = .98, RMSEA = .07, SRMR = .02), based on standard fit indices. All standardized factor loadings were statistically significant (p < .001) and high (≥.60), definitively confirming the single-factor structure of the FIRE-PL scale.

Instrument

Test Type: Questionnaire/Psychometric Scale (Supplementary Module)

Format: Paper & Pencil or Online Questionnaire.

Language Available: German (deutsch).

Population Group: Rescue forces (primarily leadership and command personnel).

Age Group: Adults (Ages 21–55 in validation samples).

Population Details: Developed primarily for use among leadership trainees (Group and Platoon Leaders) at the Institute of the North Rhine-Westphalia Fire Brigade (IdF NRW). The validation sample (N=288) consisted mostly of career and volunteer firefighters.

Test Methodology: Self-report using a 7-point Likert scale (1 = stimme gar nicht zu [strongly disagree] to 7 = stimme vollkommen zu [strongly agree]). The scale is positively keyed. Estimated processing time is approximately 1 minute. Scoring involves calculating the unweighted mean of the four items.

Keywords

Evaluation of training, emergency management, leadership development, psychometrics, construct validity, Cronbach’s Alpha, McDonald’s Omega, IdF NRW.

Authors

Author ORCID Identifier: N/A

Affiliation Email addresses: [email protected]; [email protected]

Correspondence Address: Westfälische Wilhelms-Universität Münster, Institut für Psychologie.

Permissions & Fee and Test Year

The scale was developed and validated through a cooperation between the Westfälische Wilhelms-Universität Münster and the Institute of the North Rhine-Westphalia Fire Brigade (IdF NRW). The main validation study (Study III) was conducted in 2017. The instrument is documented academically, and information is available via the project website: https://www.uni-muenster.de/OWMS/bfo/projekte/fire/.

Reference’s

Schulte, N. & Thielsch, M. T. (2019). Evaluation of Firefighter Leadership Trainings. International Journal of Emergency Services, 8(1), 34-49. http://doi.org/10.1108/IJES-03-2018-0020

Thielsch & Weltzin (2013). Cited in scoring instructions regarding anonymity guidelines.

Rosseel, Y. (2012). lavaan: An R Package for Structural Equation Modeling. Journal of Statistical Software, 48(2), 1-36.

Revelle, W. (2021). psych: Procedures for Psychological, Psychometric, and Personality Research (R package version 2.1.3).

Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1-55.

Gläßer et al., 2002. Cited for Gesamtzufriedenheit scale.

Jäger, R. S. (2004). Cited for measurement of participant mood.

Röseler et al., 2020. Cited regarding practical vs. tabletop exercises.

Feuerwehr-Dienstvorschrift 100 (1999). Cited regarding leadership procedures.

Items of the Feedback Instrument for Rescue Force Development – Tabletop Exercises (FIRE-PL)

IMPORTANT: The following scale items must be preserved in their original language and must not be changed in any way.