Feedback Instrument for Rescue Force Development – Exercises for independent work (FIRE-EV)

Abstract

The Feedback Instrument for Rescue Force Development – Exercises for independent work (FIRE-EV) is a short, validated psychological instrument designed to measure the perceived quality of exercises intended for independent work (known as EVA-Aufgaben) within training programs for rescue forces. Developed as an additional module for the core Feedback Instrument for Rescue Force Development (FIRE), the FIRE-EV specifically targets the evaluation of teaching methods related to self-directed learning.

The instrument was validated based on data collected from 364 firefighters participating in leadership training courses at the Institute of the North Rhine-Westphalia Fire Brigade (IdF NRW). A confirmatory factor analysis supported a robust, one-factor structure. The scale demonstrates satisfactory internal consistency, with Cronbach’s Alpha values ranging from .73 to .77, and strong evidence supporting both construct validity and criterion validity.

Keywords

Fire service, rescue forces, exercises for independent work (EVA-Aufgaben), evaluation, self-directed learning, training quality, psychometrics, leadership development.

Authors

Stefan Röseler, Norbert Schulte, Meinald T. Thielsch

Purpose

The primary purpose of the FIRE-EV scale is the systematic evaluation of the quality of independent work assignments (EVA-Aufgaben) used in the theoretical training of rescue forces, particularly those in leadership roles. By collecting standardized participant feedback, the instrument helps instructors and training institutions assess the effectiveness of these self-directed learning methods and identify areas for improvement in curriculum design and instructional clarity.

While originally developed for evaluating leadership training within the fire service context, the instrument is flexible. It can be appended to the core FIRE questionnaire (for leadership training evaluation) or the FIRE-B questionnaire (for basic rescue force training). Due to its focus on general pedagogical criteria rather than specific operational content, the FIRE-EV is also suitable for use outside the rescue services, such as in vocational or academic education settings where self-directed learning is employed.

Construct

The FIRE-EV measures the perceived quality of self-administered assignments, specifically the construct defined as the Quality of Exercises for Independent Work (EVA-Aufgaben). This construct encompasses participants’ subjective assessment of key pedagogical features necessary for effective self-study.

The core dimensions captured by the scale items relate to whether the exercises are perceived as having an appropriate scope or volume, whether they successfully deepen the participants’ understanding of the course content, and whether their instructions and difficulty levels are clear and appropriate for the learning goals.

Validity

The validity of the FIRE-EV scale was established through detailed psychometric analyses across multiple studies, demonstrating strong content, construct, and criterion support.

Content Validity: Items were derived from an existing evaluation questionnaire (MFE-ZHa) and reviewed by experts (instructors and participants at IdF NRW) to ensure relevance and comprehensibility regarding the quality factors of EVA-Aufgaben. The items were judged as generally relevant and understandable by over 90% of the initial sample.

Construct Validity: The FIRE-EV shows theoretically expected correlations with related constructs. A high positive correlation was found with the FIRE subscale measuring “Instructor Behavior” (r = .51***), reflecting the instructors’ foundational role in designing quality assignments. Similarly, a moderate correlation with the “Group Work” scale (FIRE-G, r = .39***) was observed, as both formats rely on overall instructional design. Crucially, the FIRE-EV showed a non-significant correlation with participants’ entrance exam scores (r = .03 n.s.), indicating that the scale successfully measures the quality of the assignments rather than merely reflecting the participants’ prior knowledge.

Criterion Validity: The scale demonstrated strong predictive relationships with relevant training outcomes. The perceived quality of EVA-Aufgaben showed a moderate to high negative correlation with participants’ feeling of “Overload” (r = -.35***), confirming that better-designed tasks reduce the risk of excessive burden. High FIRE-EV scores were also strongly associated with positive self-assessed learning outcomes, correlating highly with “Competence Acquisition” (r = .40***) and “Transfer” (r = .39***). Finally, the scale showed a high correlation with “Overall Satisfaction” with the course (r = .50***) and a moderate correlation with the final course grade (r = .28***).

Reliability

The reliability of the FIRE-EV scale was assessed through measures of internal consistency across two large validation samples.

• In Study II (N=242), Cronbach’s Alpha was calculated at .73, and McDonald’s Omega was .75.
• In Study III (N=364), Cronbach’s Alpha increased to .77, and McDonald’s Omega reached .78.

These coefficients demonstrate that the FIRE-EV exhibits satisfactory reliability, confirming that the four items consistently measure the underlying single factor of perceived quality of independent work exercises.

Factor Analysis

The development process involved both exploratory and confirmatory factor analysis to establish the scale’s underlying structure.

Exploratory Factor Analysis (EFA): Initial testing (Study II, N=242) utilized an item pool of six candidate items. The EFA supported a single-factor structure among the items with high factor loadings. Two items were excluded: one concerning the post-discussion of tasks (due to low item-total correlation) and one concerning sufficient time for completion (EVA_1, excluded for content reasons, as it could reflect individual time management rather than objective task quality).

Confirmatory Factor Analysis (CFA): The final four-item, single-factor model was tested using CFA (Study III, N=364). The results indicated an excellent model fit according to recommended indices (e.g., Chi-squared (2) = 2.91, p = .234; CFI = .998; RMSEA = .04), with all standardized factor loadings being high and statistically significant (≥.65). This confirmed the unidimensional nature of the FIRE-EV scale.

Instrument

Test Type: Evaluation questionnaire / Psychometric scale

Format: 4 items utilizing a 7-point Likert scale (1 = strongly disagree to 7 = strongly agree). An optional eighth category is provided for “not meaningfully answerable.” All items are positively keyed.

Language Available: German (Deutsch)

Population Group: Rescue forces (including firefighters, police, military, and civil protection personnel), primarily those in advanced or leadership training. Can be adapted for use in basic training or general educational contexts.

Age Group: Adults (Validation samples ranged from 21 to 55 years old).

Population Details: Validation studies involved participants (N=606 combined sample) attending group and platoon leader courses at the IdF NRW. The sample was predominantly male (approx. 95%), comprised of a mix of professional and volunteer firefighters with significant operational experience.

Test Methodology: The scale was developed for Paper-and-Pencil Self-Administered Interviewing (PASI) but can be used in an online format (CASI). The estimated processing time is highly economical, lasting approximately 1 minute.

Keywords

Evaluation, EVA-Aufgaben, self-directed learning, psychometrics, leadership development, emergency services, training quality.

Authors

Author ORCID Identifier: N/A

Affiliation Email addresses: [email protected], [email protected]

Correspondence Address: Westfälische Wilhelms-Universität Münster, Institut für Psychologie

Permissions & Fee and Test Year

The FIRE-EV was developed and validated through a collaboration between the Institute of the North Rhine-Westphalia Fire Brigade (IdF NRW) and the Organizational and Business Psychology department at the Westfälische Wilhelms-Universität Münster. The validation studies took place between 2015 and 2017, confirming the instrument’s validated status. Specific fee structures are not detailed in the source content, but contact information for the authors is provided for inquiries regarding use and permissions.

Reference’s

The primary reference detailing the development and validation of the core FIRE questionnaire, to which FIRE-EV is a module, is:

• Schulte, N. & Thielsch, M. T. (2019). Evaluation of Firefighter Leadership Trainings. International Journal of Emergency Services, 8(1), 34-49.

Items of the Feedback Instrument for Rescue Force Development – Exercises for independent work (FIRE-EV)

IMPORTANT: The following scale items must be preserved in their original language and must not be changed in any way.

Tabelle 1

Items der Skala FIRE-EV

Anmerkung. Alle Items sind positiv gepolt.