Feedback Tool for Responder Development – Senior Staff (FIRE-CU)

Abstract

The Feedback Tool for Responder Development – Senior Staff (FIRE-CU) is a 22-item psychometric instrument designed to assess the quality and effectiveness of training programs aimed at senior emergency response personnel, specifically members of the fire brigade command staff (Command Unit, CU). Developed as an adaptation of the original FIRE scale (Schulte & Thielsch, 2019), the FIRE-CU is tailored to reflect the complex competencies required at the strategic leadership level.

The instrument measures six core dimensions: four representing the learning process (Instructor Behavior, Requirement Level, Structure, and Group Dynamics) and two representing learning outcomes (Acquisition of Competence and Transfer). Confirmatory Factor Analysis (CFA) validated this six-factor structure. The scales demonstrate robust psychometric properties, including high internal consistency (Cronbach’s Alpha ranging from .75 to .90) and strong evidence for content, construct, and criterion validity.

Keywords

Command staff, Crisis management, Command post exercise, Rescue forces, Training evaluation, Instructor Behavior, Competence Acquisition, Training effectiveness, High Reliability Teams

Authors

Meinald T. Thielsch, Dzenana Hadzihalilovic, Schulte

Purpose

The primary objective of the FIRE-CU is to facilitate the standardized and economical evaluation of training courses for senior command staff, ensuring that educational content and delivery methods meet the high demands of operational leadership in crisis situations. By collecting feedback from course participants immediately following the training, the instrument offers valuable insights for instructors and organizational leaders regarding areas needing improvement or commendation.

While originally developed in collaboration with the Institute of the Fire Department NRW (IdF NRW) for the German fire service, the scale items are intentionally formulated to be non-fire-service-specific. This generic structure permits its potential application in evaluating command staff training within other high-risk organizations, such as police, technical relief agencies (THW), and rescue services, where high reliability team training principles are comparable. Utilization outside the fire service context necessitates a review of content validity and accompanying validation of measurements within the new operational environment.

Construct

The FIRE-CU measures training effectiveness based on a model that separates evaluation into process and outcome levels, consistent with evaluation frameworks like Kirkpatrick’s Four-Level Model.

The six measured subscales are:

1. Instructor Behavior: Measures the quality of instruction, feedback, and motivation provided by the educators.
2. Requirement Level: Assesses the perceived difficulty, volume, and pace of the course material (higher scores here indicate potential participant overload, requiring inversion during scoring).
3. Structure: Evaluates the clarity, organization, and coherence of the training program overview.
4. Group Dynamics: Measures the quality of collaboration, mutual support, and cohesion among course participants.
5. Acquisition of Competence: Measures the self-assessed learning outcomes, focusing on critical skills for command staff, such as decision-making in crisis and information processing.
6. Transfer: Assesses the perceived ability of participants to apply the newly acquired knowledge and skills directly to their future operational role in the command staff.

Validity

The validity evidence for the FIRE-CU supports its use as a measure of training quality for senior command personnel.

Content Validity was ensured through expert interviews and pre-testing with course participants, guaranteeing that the adapted items adequately cover the specific requirements and learning objectives of command staff training.

Construct Validity was established using theoretically related constructs. Convergent analyses showed significant correlations between the FIRE-CU subscales and established evaluation instruments. For example, the Transfer scale correlated highly (r = .69, p < .001) with self-developed items assessing preparedness for the command post exercise. Divergent validity was supported by low correlations with demographic factors like age and mood, indicating the scale is relatively free from common evaluation biases.

Criterion Validity (concurrent) was demonstrated through correlations with global measures of training satisfaction and assigned school grades. Correlations between the FIRE-CU dimensions and overall satisfaction ranged from r = 0.19 (Requirement Level) to r = 0.44 (Instructor Behavior), confirming that the subscales successfully predict general positive evaluation of the course.

Reliability

The reliability analysis confirms the internal consistency of the FIRE-CU scales. Depending on the best-fitting measurement model (tau-equivalent or congeneric), reliability coefficients were calculated using Cronbach’s Alpha or McDonald’s Omega (ωH).

All six core scales achieved reliability coefficients considered acceptable or superior for program evaluation purposes. The Requirement Level scale showed the highest internal consistency (Cronbach’s Alpha = .90; ωH = .91), demonstrating excellent reliability. Scales measuring Instructor Behavior (α = .82; ωH = .82) and Group Dynamics (α = .88; ωH = .88) also showed high reliability. The lowest, yet still sufficient, reliability was recorded for the Structure scale (α = .75; ωH = .76).

Factor Analysis

The factor structure of the FIRE-CU was investigated using Confirmatory Factor Analysis (CFA) on data collected from N = 288 participants. The CFA aimed to confirm the theoretical distinction between the four process factors and the two outcome factors.

The four-factor model for the process variables (Instructor Behavior, Requirement Level, Structure, Group) achieved a good fit based on standard indices (RMSEA = 0.06, CFI = 0.96). The analysis for the two outcome factors (Acquisition of Competence and Transfer) initially indicated a suboptimal fit, but adjustments based on the theoretical interdependence of complex command competencies led to a well-fitting model (RMSEA = 0.06, CFI = 0.97). This confirms that the FIRE-CU effectively measures six distinct but related dimensions of training quality.

Instrument

Test Type: Feedback/Training Evaluation Instrument

Format: Self-report questionnaire consisting of 22 core items, typically administered via Paper & Pencil, although digital formats are also appropriate. Items are rated on a 7-point scale (1 = strongly disagree to 7 = strongly agree), with optional items for overall judgment (e.g., school grade) and open feedback.

Language Available: German (Item and documentation language); an English version is also available.

Population Group: Senior Staff members (Führungsstab/Command Unit) undergoing specialized leadership training for complex emergency management. This includes professional and volunteer members of fire services.

Age Group: Adults (validated sample ages 23 to 64).

Population Details: The validation sample consisted of 288 participants from command staff training courses at IdF NRW, predominantly male (96.18%). The total estimated time for completion is brief, ranging from 4 to 5 minutes.

Test Methodology: Scoring involves assigning points from 1 to 7 based on the agreement scale. Crucially, items belonging to the Requirement Level scale must be negatively coded before calculating subscale means to ensure that higher scores consistently indicate a positive training evaluation (i.e., appropriate difficulty level).

Keywords

Führungsstab, Command Unit, Evaluation, Training quality, Command staff, Learning outcomes, High-risk organizations, Psychometrics, Fire service

Authors

Author ORCID Identifier: N/A

Affiliation Email addresses: N/A

Correspondence Address: Meinald Thielsch, Prof. Dr. Dipl.-Psych., Universität Münster, Institut f (Refer to project website: https://www.uni-muenster.de/OWMS/bfo/projekte/fire/)

Permissions & Fee and Test Year

The FIRE-CU was developed in collaboration between the Organizational and Business Psychology department at the Westfälische Wilhelms-Universität Münster and the Institut der Feuerwehr NRW (IdF NRW). The primary validation study was conducted in 2018, with results published in 2020 (Thielsch & Hadzihalilovic). Information regarding the scale development and related instruments is available on the university’s project website. The core publication detailing the scale’s evaluation is available here: https://doi.org/10.1007/s13753-020-00279-6.

Reference’s

The following key references supported the development and validation of the FIRE-CU:

• Kirkpatrick, D. L. (1979). Techniques for evaluating training programs. Training and Development Journal, 33(6), 78–92.
• Schulte, A., & Thielsch, M. T. (2019). Evaluation of training for fire brigade group and platoon leaders: Development and validation of the FIRE questionnaire. International Journal of Disaster Risk Reduction, 37, 101150.
• Thielsch, M. T. & Hadzihalilovic, D. (2020). Evaluation of Fire Service Command Unit Trainings. International Journal of Disaster Risk Science, 11(3), 300-315. https://doi.org/10.1007/s13753-020-00279-6
• Thielsch, M. T., Busjan, J. N., & Frerichs, K. (2018). Feedback-Instrument-zur Rettungskräfte Entwicklung – Prüfungen (FIRE-P) [Feedback instrument for rescue force development – examinations (FIRE-P)]. Zusammenstellung sozialwissenschaftlicher Items und Skalen (ZIS). https://doi.org/10.6102/zis260.
• van den Bossche, P., Segers, M., & Gijselaers, W. (2006). The many faces of competence. Educational Research Review, 1(1), 3–18.

Items of the Feedback Tool for Responder Development – Senior Staff (FIRE-CU)

IMPORTANT: The following scale items must be preserved in their original language and must not be changed in any way.