Feedback Instrument for Rescue Force Development – trainer (FIRE-DO)

Abstract

The Feedback Instrument for Rescue Force Development – trainer (FIRE-DO) is a concise, 7-item questionnaire designed to evaluate the didactic competence of individual trainers or instructors involved in rescue force training. Developed as an optional supplementary module for the core Feedback Instrument for Rescue Force Development (FIRE), the FIRE-DO allows for a specific, individual assessment of teaching quality, complementing the broader course evaluation provided by the FIRE core instrument.

The scale was rigorously validated using a sample of 350 firefighters participating in leadership training courses at the Institute of the North Rhine-Westphalia Fire Brigade (IdF NRW). Psychometric analysis confirmed a robust one-factor structure, demonstrating very high internal consistency (McDonald’s Omega = .91), along with significant evidence supporting both construct validity and criterion validity. The instrument is utilized within organizations responsible for emergency services training to gather actionable feedback on instructor performance.

Keywords

fire service, rescue forces, trainer evaluation, didactic competence, leadership training, questionnaire, psychometric scale, professional development

Authors

B.Sc. Cordelia Kleinstück, M.Sc. Stefan Röseler, Prof. Dr. Dipl.-Psych. Meinald Thielsch

Purpose

The primary purpose of the FIRE-DO is the individual evaluation of instructors within the context of leadership training for rescue forces. While the original FIRE instrument assesses the overall quality of the training course and the teaching team collectively, the FIRE-DO specifically isolates and measures the perceived teaching effectiveness and didactic competence of a single trainer, as rated by the course participants.

This instrument serves an important function in professional development, providing trainers with crucial, individualized feedback to improve their instructional methods and content delivery. Given its high internal consistency, the FIRE-DO is suitable for individual diagnostic application and can be used to compare the performance of different instructors or track the development of a single instructor over multiple seminars. The scale is applicable not only in the fire service context but also potentially in other rescue and emergency organizations (e.g., civil protection, police, military).

Construct

The FIRE-DO measures the Didactic Competence of instructors within the framework of rescue force training, as perceived and assessed by the course participants. This construct encompasses various quality criteria essential for effective teaching, derived largely from established models of successful university teaching, such as Rindermann’s (2009) multidimensional model of teaching success, adapted for the high-demand environment of rescue services.

Key facets of Didactic Competence assessed by the 7 items include the instructor’s ability to clearly summarize complex information, provide useful feedback, motivate participants, demonstrate interest in participant success, maintain clear linguistic expression, possess comprehensive subject knowledge, and effectively integrate practical operational experience into the teaching material.

Validity

The validity of the FIRE-DO is supported by strong evidence across content, construct, and criterion measures. Content validity was established through expert review (IdF NRW instructors and participants) and the derivation of items from established scales for university teaching evaluation. Experts confirmed the items were generally relevant and understandable for the target context (Study I).

Construct validity was demonstrated through high correlations with convergent measures, including the “Instructor Behavior” subscale of the FIRE core questionnaire (r = .69***) and similar subscales from the Heidelberger Inventar zur Lehrveranstaltungs-Evaluation (HILVE I & HILVE II), such as “Teaching Competence” (r = .86***) and “Supervision” (r = .75***). Divergent validity was confirmed by a non-significant correlation with participants’ entrance exam scores (r = .09 n.s.), indicating that prior knowledge does not unduly influence the instructor rating.

Criterion validity was supported by moderate to high correlations with key outcome factors of the training course, such as “Competence Acquisition” (r = .44***) and “Preparation & Transfer” (r = .48***) from the FIRE core questionnaire. Furthermore, high FIRE-DO scores correlated strongly with participants’ overall satisfaction with the course (r = .59***) and their positive grading of the seminar (r = .40***).

Reliability

The reliability of the FIRE-DO scale was assessed using data from Study II (N = 350). The internal consistency was calculated using McDonald’s Omega, which is appropriate for congeneric measurement models. The resulting model was found to be significantly better fitting than an essentially tau-equivalent model (χ²(6) = 52.10, p < .001).

The calculated value of McDonald’s Omega was 0.91, indicating a very high level of internal consistency for the 7-item scale. This high reliability supports the use of the FIRE-DO for individual diagnostic assessment of instructor performance.

Factor Analysis

The underlying structure of the FIRE-DO was examined using Confirmatory Factor Analysis (CFA) based on the data from Study II (N = 350), utilizing the robust Maximum-Likelihood estimator (MLR) in R package lavaan. Given the previously confirmed unidimensionality of the related “Instructor Behavior” subscale in the FIRE core instrument, the researchers hypothesized a single-factor structure for the FIRE-DO items.

The initial one-dimensional model did not yield a satisfactory fit (χ²(14) = 64.83, CFI = .968, TLI = .952, RMSEA = .102, SRMR = .028), with the RMSEA indicating an inadequate fit. Modification indices suggested modeling a residual correlation between Item 3 (motivation) and Item 4 (interest in learner success), two items conceptually related to motivational processes. The resulting modified model demonstrated an acceptable fit to the data (χ²(13) = 47.40, CFI = .978, TLI = .965, RMSEA = .087, SRMR = .024), confirming the scale’s fundamental unidimensional structure.

Instrument

Test Type: Questionnaire (Psychometric Scale, Evaluation Instrument)

Format: 7-item self-report scale utilizing a 7-point Likert response format (1 = stimme gar nicht zu [strongly disagree], 7 = stimme vollkommen zu [strongly agree]). The scale can be administered in either Paper & Pencil or online formats.

Language Available: German (deutsch)

Population Group: Participants/trainees in rescue force leadership programs (e.g., firefighters, emergency services personnel).

Age Group: Adults (Validation sample ranged from 20 to 55 years).

Population Details: The validation sample (N=350) consisted primarily of male participants (94.6%) undergoing Group and Platoon Leader training at the IdF NRW. This demographic is considered representative of the population attending leadership courses at German fire schools. Participants had an average of 14.4 years of operational experience.

Test Methodology: The FIRE-DO is typically administered as an add-on module to the FIRE core questionnaire. It requires participants to name and evaluate a single instructor of their choice. The test is time-efficient, requiring an estimated processing time of only 2 minutes. Scoring involves calculating the mean of the seven positively polarized item scores, resulting in a single scale mean, where higher scores indicate greater perceived didactic competence.

Keywords

instructor evaluation, teaching effectiveness, fire brigade, emergency services, leadership training, psychological scale, training quality, IdF NRW, McDonald’s Omega

Authors

Author ORCID Identifier: Not provided in source content.

Affiliation Email addresses: [email protected], [email protected], [email protected]

Correspondence Address: Westfälische Wilhelms-Universität Münster, Institut für Psychologie.

Permissions & Fee and Test Year

The status of development is categorized as “tested” (erprobt). The primary validation study (Study II) took place in 2017. Specific details regarding formal permissions or licensing fees are not explicitly provided in the source abstract. Further information is available on the project website: go.wwu.de/fire.

Reference’s

• Schulte, N. & Thielsch, M. T. (2019). Evaluation of Firefighter Leadership Trainings. International Journal of Emergency Services, 8(1), 34-49. http://doi.org/10.1108/IJES-03-2018-0020
• Browne, M. W., & Cudeck, R. (1992). Alternative ways of assessing model fit. Sociological Methods & Research, 21(2), 230–258.
• Gläßer, C., Heumann, S., & Scheer, G. (2002). Erfolgreich präsentieren und moderieren: Ein Trainingshandbuch. Hogrefe.
• Grabbe, C. (2003). Der Fragebogen zur Evaluation von Lehrveranstaltungen (FEEL). Peter Lang.
• Jäger, R. S. (2004). Skalen zur Messung der Stimmung. Hogrefe.
• Revelle, W. (2021). psych: Procedures for personality and psychological research (Version 2.1.6). Northwestern University.
• Rindermann, H. (2009). The importance of students’ personality, behavior, and instructional methods for university instruction success and the validity of student evaluations of teaching. Learning and Individual Differences, 19(2), 263–271.
• Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36.
• Schulte, N., Thielsch, M. T., & Röseler, S. (2019). Development and validation of the Feedback Instrument for Rescue Force Development (FIRE). Journal of Applied Measurement, 20(4), 415–432.
• Thielsch, M. T., & Stegemöller, M. (2014). Entwicklung und Validierung eines Zusatzmoduls zur Evaluation von Moderation (MFE-ZMo). Zeitschrift für Arbeits- und Organisationspsychologie, 58(1), 16–28.
• Thielsch, M. T., & Weltzin, A. (2013). Evaluation von Lehrveranstaltungen: Ein Leitfaden für Hochschulen. Hogrefe.
• West, S. G., Finch, J. F., & Curran, P. J. (1995). Structural equation models with nonnormal variables: Problems and remedies. In R. H. Hoyle (Ed.), Structural equation modeling: Concepts, issues, and applications (pp. 56–75). Sage.
• Whittaker, T. A. (2012). Residual correlations in confirmatory factor analysis: Goodness of fit or misinterpretation? Journal of Personality Assessment, 94(3), 277–286.
• Wolf, D., Hering, T., & Schuler, H. (2001). Evaluation von Lehrveranstaltungen an Hochschulen: Der Münsteraner Fragebogen zur Evaluation (MFE). Zeitschrift für Arbeits- und Organisationspsychologie, 45(3), 136–148.

Items of the Feedback Instrument for Rescue Force Development – trainer (FIRE-DO)

IMPORTANT: The following scale items must be preserved in their original language and must not be changed in any way.