Feedback Instrument for Rescue Force Development – Group work (FIRE-G)

Abstract

The Feedback Instrument for Rescue Force Development – Group work (FIRE-G) is a brief, three-item supplementary module designed to measure the perceived quality of group work within training courses for rescue forces, particularly leadership development programs. It functions as an extension of the core Feedback Instrument for Rescue Force Development (FIRE; Schulte & Thielsch, 2019). Validation of the FIRE-G was conducted using a sample of 375 firefighters attending training courses at the Institute of the North Rhine-Westphalia Fire Brigade (IdF NRW). A Confirmatory Factor Analysis (CFA) supported a robust unidimensional structure, indicating a single underlying factor. The scale demonstrates satisfactory internal consistency and provides clear evidence of both construct validity and criterion validity.

Keywords

Fire service, rescue forces, group work, evaluation, leadership training, learning-promotive interactions, questionnaire, psychometrics

Authors

M.Sc. Stefan Röseler, Prof. Dr. Dipl.-Psych. Meinald Thielsch

Purpose

The primary purpose of the FIRE-G instrument is the targeted evaluation of group work as a specific teaching method used in the training of leadership personnel within rescue services. It is typically applied as an optional module attached to the main FIRE core questionnaire, which evaluates broader dimensions of leadership training quality (e.g., instructor behavior, competency acquisition).

By focusing exclusively on group interactions, the FIRE-G provides specific feedback to instructors regarding the effectiveness of collaborative learning activities. High scores indicate successful learning-promotive interactions among participants. The instrument is designed to be highly economical and flexible, usable in both paper-and-pencil and online formats, with a minimal estimated completion time of approximately one minute.

Construct

The FIRE-G measures the construct of Learning-Promotive Interactions (Lernförderliche Interaktionen) within group work settings. This construct captures the perceived quality of interaction and collaboration among participants, reflecting key aspects of effective group learning:

• The extent to which participants learn from one another through mutual exchange.

• The degree to which all participants contribute according to their respective competencies.

• The effectiveness of the group work in fostering exchange and discussion among participants.

The theoretical foundation relies on the pedagogical understanding that group work, as a social form of teaching, promotes active engagement, knowledge explication, and the resolution of differing viewpoints, which are essential for deep learning, especially in high-stakes environments like rescue services where theoretical knowledge must be rapidly applied under pressure.

Validity

The scale demonstrates indications of strong content, construct, and criterion validity based on Study III (N=375).

• Content Validity: Initial item pools were rigorously reviewed by both trainers and course participants at the IdF NRW for relevance and comprehensibility (Study I). Experts confirmed that the captured aspects were essential for evaluating learning success in group settings.

• Construct Validity: Convergent and divergent validity were assessed using subscales of the FIRE core questionnaire and other measures. The FIRE-G showed a high correlation (r = .53***) with the FIRE scale measuring the general rating of the Group environment (convergent evidence). It correlated less strongly with the FIRE scale on Instructor Behavior (r = .43***) and the FIRE-E scale (Evaluation of Operational Exercises, r = .40***), demonstrating adequate divergence, as the FIRE-G intentionally focuses on peer-to-peer interaction rather than instructor-led organizational aspects. Furthermore, the non-significant correlation with entry exam scores (r = -.04 n.s.) suggests that prior knowledge does not bias the evaluation of group work quality.

• Criterion Validity: The FIRE-G exhibited significant positive correlations with self-assessed learning outcomes. Specifically, it correlated moderately with the FIRE scales measuring Competency Acquisition (r = .38***) and Transfer (r = .32***). It also showed a moderate correlation with overall course satisfaction (r = .44***) and a small, yet significant, correlation with the participants’ overall course grading (r = .29***), supporting its utility in predicting positive training outcomes.

Reliability

The reliability of the FIRE-G, measured as internal consistency, is satisfactory to good. In Study III (N=375), where an essentially tau-equivalent measurement model was established, the reliability coefficients were high:

• Cronbach’s Alpha = .83.

• McDonald’s Omega = .83.

In an earlier sample (Study II, N=240), McDonald’s Omega was calculated at .77. These values confirm that the short, three-item scale reliably measures the intended single construct of learning-promotive interactions.

Factor Analysis

The development process involved multiple steps, including Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis (CFA) to establish the final, parsimonious structure.

Initially, seven items were tested (Study II, N=240). Based on low variance (Item G_5) and the goal of distinguishing the scale from instructor evaluation, items relating to organizational aspects (instructor support, debriefing, task clarity) were excluded. The final scale retained three items focused solely on peer-to-peer interaction (G_1, G_3, G_6). A subsequent oblimin-rotated principal components analysis (EFA 2) confirmed very high factor loadings for these three remaining items, supporting a unidimensional structure.

In Study III (N=375), CFA was performed. Since the three-item structure is a just-identified model (zero degrees of freedom), an over-identified essentially tau-equivalent model was employed for verification. This model demonstrated excellent fit indices (e.g., CFI = .998, TLI = .997, RMSEA = .04, SRMR = .04), confirming that the FIRE-G successfully measures a single, distinct factor: Learning-promotive interactions during group work.

Instrument

Test Type: Paper-and-pencil questionnaire / Online survey module

Format: 3 items utilizing a seven-point Likert scale (1 = stimme gar nicht zu [strongly disagree] to 7 = stimme vollkommen zu [strongly agree]).

Language Available: German (deutsch)

Population Group: Professionals in the rescue service sector (e.g., firefighters, emergency services, civil defense, military).

Age Group: Adult professionals (Validated sample mean age approximately 31.8 to 33.4 years).

Population Details: Validation was conducted primarily on male participants (93.9% to 95.4%) attending group and platoon leader training courses at the Institute of the North Rhine-Westphalia Fire Brigade (IdF NRW). Participants had extensive professional experience (mean experience 13.7 to 14.4 years).

Test Methodology: Self-report questionnaire (PASI – Paper-and-pencil or online). Recommended administration immediately following group work activity, and prior to any final examination, to minimize response bias. Scoring involves calculating the mean score of the three items (after summing and dividing by the number of answered items).

Keywords

Evaluation, learning interaction, group dynamics, emergency services training, psychometric properties, construct validity, criterion validity

Authors

Author ORCID Identifier: Not provided in source content.

Affiliation Email addresses: [email protected]; [email protected]

Correspondence Address: Westfälische Wilhelms-Universität Münster, Institut für Psychologie (Stefan Röseler and Meinald Thielsch).

Permissions & Fee and Test Year

The scale was developed and validated through collaboration between the Institute of the North Rhine-Westphalia Fire Brigade (IdF NRW) and the Organizational and Business Psychology department at Westfälische Wilhelms-Universität Münster. The validation studies (Study I, II, and III) took place between late 2015 and March 2017. Information regarding specific usage fees or formal permissions beyond academic research and internal institutional evaluation is not provided, though the instrument is generally accessible for educational evaluation purposes, often as an add-on to the core FIRE questionnaire (go.wwu.de/fire).

Reference’s

• Czerwiński, B. & Atroszko, P. A. (2021). A note on the application of confirmatory factor analysis for short scales. Frontiers in Psychology, 12, 640134.

• Drumm, G. & Scholz, H. (2007). Lernen und Lehren. Kohlhammer.

• Gläßer, S., Rödiger, G., & Schuler, H. (2002). Evaluation von Lehre und Dozenten in der beruflichen Weiterbildung. Hogrefe.

• Grötemeier, S. & Thielsch, M. T. (2014). Lehrevaluation an der Führungsakademie der Bundeswehr. Unveröffentlichter Bericht.

• Hu, L. T. & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55.

• Jäger, R. S. (2004). Das “Smiley”-Verfahren. Universität Koblenz-Landau.

• Karau, S. J. & Williams, K. D. (1993). Social loafing: A meta-analytic review and theoretical integration. Journal of Personality and Social Psychology, 65(4), 681–706.

• Kleinstück, C., Röseler, S., & Thielsch, M. T. (2021). Feedback Instrument for Rescue Force Development – Instructor (FIRE-DO). ZPID (Leibniz Institute for Psychology Information).

• Liden, R. C., Wayne, S. J., Jaworski, R. A., & Bennett, N. (2004). Social loafing: A field investigation. Journal of Management, 30(2), 285–304.

• Niemann, J. & Thielsch, M. T. (2020). Feedback Instrument for Rescue Force Development – Basic Training (FIRE-B). ZPID (Leibniz Institute for Psychology Information).

• Pluntke, M. (2013). Methodik der feuerwehrtechnischen Ausbildung. Kohlhammer.

• Renkl, A. (2015). Lernpsychologie. Springer.

• Röseler, S., Kleinstück, C., & Thielsch, M. T. (2020). Feedback Instrument for Rescue Force Development – Operational Exercises (FIRE-E). ZPID (Leibniz Institute for Psychology Information).

• Röseler, S., Kleinstück, C., & Thielsch, M. T. (2021). Feedback Instrument for Rescue Force Development – Self-directed Learning Tasks (FIRE-S). ZPID (Leibniz Institute for Psychology Information).

• Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36.

• Schnake, M. E. (1991). Equity in effort: The ‘sucker effect’ in coworker relationships. Journal of Management, 17(1), 41–55.

• Schulte, N. & Thielsch, M. T. (2019). Evaluation of Firefighter Leadership Trainings. International Journal of Emergency Services, 8(1), 34-49. http://doi.org/10.1108/IJES-03-2018-0020

• Schulte, N., Thielsch, M. T., Vöge, S., & Löchteken, T. (2019). Feedback Instrument for Rescue Force Development (FIRE). ZPID (Leibniz Institute for Psychology Information).

• Spiel, C. (2001). Evaluation in der LehrerInnenbildung. In R. S. Jäger, P. Schräpler, M. A. Wirtz, & C. Spiel (Eds.), Ergebnisse der angewandten Psychologie und Pädagogik (pp. 37–58). Waxmann.

• Thielsch, M. T. & Hadzihalilovic, E. (2020). Feedback Instrument for Rescue Force Development – Command Unit (FIRE-CU). ZPID (Leibniz Institute for Psychology Information).

• Thielsch, M. T., & Weltzin, S. (2013). Lehrevaluation an der Führungsakademie der Bundeswehr. Unveröffentlichter Bericht.

• Thielsch, M. T., Niemann, J., & Schulte, N. (2019). Feedback Instrument for Rescue Force Development – Basic Training (FIRE-B). ZPID (Leibniz Institute for Psychology Information).

Items of the Feedback Instrument for Rescue Force Development – Group work (FIRE-G)

IMPORTANT: The following scale items must be preserved in their original language and must not be changed in any way.

Response Format:

A seven-point response format is used for Items 1 through 3, ranging from 1 = stimme gar nicht zu (strongly disagree) to 7 = stimme vollkommen zu (strongly agree). An additional option is provided for respondents to indicate that the item is nicht sinnvoll beantwortbar (not reasonably answerable).