Feedback Instrument for Rescue Force Development – Seminars and One-day Events (FIRE-ST)

Abstract

The Feedback Instrument for Rescue Force Development – Seminars and One-day Events (FIRE-ST) is a specialized psychological questionnaire designed to measure the quality of short-format training evaluation for rescue personnel. Developed as an adaptation of the original Feedback Instrument for Rescue Force Development (FIRE, Schulte & Thielsch, 2019), the FIRE-ST focuses specifically on brief seminars and workshops lasting between one and three days, contrasting with the multi-week leadership courses evaluated by its predecessor.

The instrument was validated using a sample of 247 firefighters attending seminars at the Institute of the North Rhine-Westphalia Fire Brigade (IdF NRW). Confirmatory Factor Analysis (CFA) supported a five-factor structure, differentiating between three learning process factors (Trainer Behavior, Requirement Level/Overextension, and Structure) and two learning outcome factors (Competence Acquisition and Transfer). The resulting scales demonstrate acceptable to good internal consistency, alongside robust indications of construct validity and criterion validity.

Keywords

Fire service, Rescue force development, Training evaluation, Leadership training, Seminars, Instructor behavior, Competence acquisition, Transfer, Psychometrics, Confirmatory factor analysis.

Authors

Stefan Röseler, Hannah Felske, Meinald T. Thielsch.

Purpose

The primary purpose of the FIRE-ST scale is to provide systematic and standardized feedback on the quality of short-duration leadership training and specialized seminars for rescue forces, particularly those lasting between one and three days. This instrument enables training organizations, such as fire service institutes, to objectively assess key quality characteristics of their educational offerings.

The evaluation results derived from the FIRE-ST are intended to serve as actionable feedback for instructors (Dozenten) regarding their teaching effectiveness and the learning progress of participants. The instrument is specifically tailored to the characteristics of shorter courses, which often involve frontal instruction, larger groups (up to 120 participants), and less emphasis on prolonged group exercises compared to multi-week leadership courses.

Construct

The FIRE-ST measures the efficacy and quality of short-format training for rescue personnel, conceptualized across five distinct subscales based on the theoretical framework of training evaluation, particularly adapting the reaction and learning levels of Kirkpatrick’s model (1976). The construct is divided into process judgments and outcome judgments, providing a dichotomy for evaluating both the delivery quality and the attainment of training objectives.

The five core factors measured are: Dozentenverhalten (Trainer Behavior), Anforderungsniveau (Requirement Level/Overextension), Struktur (Structure), Kompetenzerwerb (Competence Acquisition), and Transfer. The first three factors assess the learning process during the seminar, while the latter two evaluate the learning outcomes achieved by the participants.

Validity

The validation process provided clear evidence for the scale’s validity across multiple dimensions, including factorial, construct, and criterion validity.

• Content Validity: Content validity is considered adequate for all scales, except the Transfer subscale, which is only considered valid when the training event includes practical components (e.g., operational or planning exercises). This is supported by research indicating that rescue personnel perceive practical training elements as more application-oriented than purely theoretical continuing education.
• Construct Validity (Convergent & Divergent): Convergent validity was established by correlating the FIRE-ST subscales with conceptually related measures from established academic evaluation instruments, such as the Heidelberger Inventar zur Lehrveranstaltungsevaluation (HILVE-I, HILVE-II) and the Training Evaluation Inventory (TEI). Results showed positive, and often high, correlations between corresponding scales (e.g., FIRE-ST Structure correlated highly with HILVE-II Structure, r = .79***). Divergent validity was supported by lower correlations with theoretically unrelated bias variables, such as job satisfaction and intention to remain in the job.
• Criterion Validity: Criterion validity was assessed using the participants’ overall grading of the seminar (Item 21, measured on a school grade scale). The FIRE-ST core subscales showed significant correlations with this criterion measure (e.g., Trainer Behavior, r = .51***; Transfer, r = .45***), indicating that the scale effectively measures factors contributing to the overall perceived success of the training.

Reliability

The internal consistency of the FIRE-ST scales was assessed using both Cronbach’s Alpha and McDonald’s Omega (ω_H), the latter being the preferred coefficient for scales exhibiting a congeneric measurement model. The scale demonstrated acceptable to good reliability across all subscales based on the validation sample (N = 247).

Summary of Reliability Coefficients:

• Dozentenverhalten (Trainer Behavior): Cronbach’s α = .89, McDonald’s ω_H = .89 (Good)
• Anforderungsniveau (Requirement Level): Cronbach’s α = .87, McDonald’s ω_H = .88 (Good)
• Struktur (Structure): Cronbach’s α = .78, McDonald’s ω_H = .79 (Acceptable)
• Kompetenzerwerb (Competence Acquisition): Cronbach’s α = .86, McDonald’s ω_H = .87 (Good)
• Transfer: Cronbach’s α = .74, McDonald’s ω_H = .76 (Acceptable)

Factor Analysis

The underlying factorial structure of the adapted questionnaire was validated using Confirmatory Factor Analysis (CFA). The analysis confirmed the hypothesized five-factor structure, which is divided into a three-factor process model and a two-factor outcome model.

The three-factor process model (Trainer Behavior, Requirement Level, and Structure) demonstrated a good fit to the data (χ²(51) = 72.40, CFI = .99, TLI = .98, RMSEA = .04). All factor loadings were statistically significant and acceptable in magnitude, aligning with expectations from previous studies involving the original FIRE scale.

The two-factor outcome model (Competence Acquisition and Transfer) was initially partially acceptable. After minor modifications suggested by indices (adding correlations between Items 13, 14, and 16, which relate to stress situations), the revised model yielded acceptable fit indices (χ²(17) = 36.05, CFI = .97, TLI = .95, RMSEA = .09). The CFA revealed that a significant number of participants skipped items related to the Transfer factor, suggesting that many evaluated seminars did not include the practical exercises required for these items to be relevant.

Instrument

Test Type: Training evaluation questionnaire/Feedback instrument.

Format: Paper & Pencil (P&P) or Online questionnaire. P&P format was used for development and validation.

Language Available: German (Item language). Documentation is also in German. An English translation of the items is available in the original FIRE instrument (Schulte & Thielsch, 2019).

Population Group: Leadership personnel and participants in specialized training within rescue organizations (primarily firefighters).

Age Group: Adults (Sample range: 23 to 60 years).

Population Details: The validation sample included 247 seminar participants (M = 41.80 years, SD = 8.76). Participants were predominantly male (96.36%) and included members of professional fire services, volunteer fire services, or both.

Test Methodology: The scale comprises 23 items. Items 1-20 utilize a 7-point response format (1 = strongly disagree to 7 = strongly agree), with an additional option for “not sensibly answerable.” Item 21 uses a 6-point school grade scale. Item 22 is dichotomous (yes/no), and Item 23 is an open feedback format. Items 21-23 are optional supplemental measures.

Keywords

Feuerwehr, Evaluation, Ausbildung, Seminare, Leadership training, Psychometrics, Rescue forces, Kirkpatrick model, Factor analysis, Internal consistency.

Authors

Author ORCID Identifier: Not provided in the source.

Affiliation Email addresses: [email protected] (Stefan Röseler); [email protected] (Hannah Felske); [email protected] (Meinald T. Thielsch).

Correspondence Address: Westfälische Wilhelms-Universität Münster, Institut für Psychologie.

Permissions & Fee and Test Year

The FIRE-ST scale was developed in cooperation between the Institute of the North Rhine-Westphalia Fire Brigade (IdF NRW) and the Organizational and Business Psychology department at Westfälische Wilhelms-Universität Münster. The validation study took place in 2018. The instrument is generally available for use in evaluating training quality in rescue services, and further details can be found on the associated project website: https://www.uni-muenster.de/OWMS/bfo/projekte/fire/.

Reference’s

Schulte, N. & Thielsch, M. T. (2019). Evaluation of Firefighter Leadership Trainings. International Journal of Emergency Services, 8, 34–49. https://doi.org/10.1108/IJES-03-2018-0020

Thielsch, M. T., Busjan, J. N., & Frerichs, K. (2018). Feedback-Instrument-zur Rettungskräfte Entwicklung – Prüfungen (FIRE-P) [Feedback instrument for rescue force development – examinations (FIRE-P)]. Zusammenstellung sozialwissenschaftlicher Items und Skalen. https://doi.org/10.6102/zis260

Thielsch, M. T., Kläpker, L., & Streppel, L. (2019). Feedback-Instrument zur Rettungskräfte-Entwicklung – Basisausbildung (FIRE-B) [Feedback instrument for rescue force development – basic training (FIRE-B)]. Zusammenstellung sozialwissenschaftlicher Items und Skalen.

Items of the Feedback Instrument for Rescue Force Development – Seminars and One-day Events (FIRE-ST)

IMPORTANT: The following scale items must be preserved in their original language and must not be changed in any way.

Tabelle 1

Items des FIRE-ST