Feedback Tool for Rescue Force Development Examinations (FIRE-P)

Abstract

The Feedback Tool for Rescue Force Development Examinations (FIRE-P) is a psychometric instrument designed to measure the perceived quality of written examinations within rescue force training programs. Developed specifically for use with participants in rescue training measures, the instrument is grounded in existing academic scale structures and extensive qualitative analysis of desirable examination characteristics, involving both examiners (N = 6) and course participants (N = 55) at the Institut der Feuerwehr NRW (IdF NRW).

The final instrument was rigorously tested and validated through 729 examination evaluations conducted during executive training at the fire brigade of North Rhine-Westphalia. Both exploratory and confirmatory factor analyses confirmed a robust three-factor structure, comprising the scales: Preparation, Design, and Load. The psychometric properties demonstrate acceptable to good internal consistency, alongside strong indications of content and construct validity.

Keywords

fire brigade, examination evaluation, rescue forces, psychometric scale, training assessment, test quality, preparation, test design, perceived load

Authors

Meinald Thielsch, Jessica Busjan, Katharina Frerichs

Purpose

The primary purpose of the FIRE-P is to systematically assess and enhance the quality of written examinations administered during training and certification courses for rescue forces, particularly focusing on leadership roles (e.g., Group and Platoon leaders) at institutions like the IdF NRW. By collecting feedback from course participants immediately following the examination, the instrument provides essential data for educational authorities to evaluate and improve curriculum delivery, transparency, and testing procedures. While originally developed for the fire brigade context, the scale is generally suitable for evaluating written examinations across various rescue force training domains, though application in other fields requires prior empirical verification.

Construct

The FIRE-P measures the participants’ subjective perception of written examination quality, which is operationalized through a three-dimensional latent construct. The three primary scales are:

• Preparation (Vorbereitung): Measures how well participants felt they were prepared for the examination by the course, including the availability and adequacy of preparatory materials and the transparency of requirements.
• Design (Gestaltung): Focuses on the objective structural quality of the examination itself, including the clarity and precision of tasks, the organization of the test, the appropriateness of the response format, and the relevance of content to the rescue professional’s future duties.
• Load (Belastung): Assesses the subjective psychological and temporal burden experienced by the participants, specifically relating to perceived difficulty, the extent of the curriculum covered, and the total number of examinations within the training course.

Validity

The validation process for FIRE-P established multiple indicators of validity. Content validity was ensured through a comprehensive development process involving qualitative interviews (Study I) with both examiners and participants, confirming that the items represented relevant criteria for high-quality examinations in the rescue context. This process was also informed by existing instruments like the MFE-K (Münster Questionnaire for Exam Evaluation).

Construct validity was confirmed through exploratory and confirmatory factor analyses (Studies III and IV), which verified the predicted three-factor structure. Furthermore, convergent and discriminant validity were assessed by correlating FIRE-P scales with various personality constructs (e.g., Conscientiousness, Neuroticism, Resilience, Self-Efficacy). For instance, the Load scale showed expected positive correlations with Neuroticism and performance-inhibiting anxiety, and negative correlations with Resilience and decisiveness. The Preparation and Design scales showed high correlations with the overall global judgment of the examination.

Concurrent validity was examined in Study III, revealing moderate correlations between the FIRE-P scales and actual examination grades (e.g., r = .47 for the Load scale), suggesting that while the perception of difficulty relates to actual performance, the evaluation is not entirely dependent on the participant’s individual result. The instrument also showed only a small positive correlation with participants’ mood at the time of evaluation, indicating that temporary affective states do not significantly overwhelm the assessment of the examination’s quality.

Reliability

The internal consistency of the FIRE-P scales was measured using Cronbach’s Alpha across Study III (N = 428) and Study IV (N = 301). The reliability coefficients demonstrated acceptable to good consistency for all three core scales, which is sufficient for evaluations conducted at the group or course level, which is the typical application context for the FIRE-P.

• Preparation: Cronbach’s Alpha = 0.84 (Study III) / 0.82 (Study IV)
• Design: Cronbach’s Alpha = 0.81 (Study III) / 0.86 (Study IV)
• Load: Cronbach’s Alpha = 0.79 (Study III) / 0.76 (Study IV)

Factor Analysis

The development of the FIRE-P involved four empirical studies to refine the item pool and confirm the factorial structure. Initial item generation was based on a qualitative content analysis (Study I) yielding eight categories of perceived exam quality (Preparation, Transparency, Practical Relevance, Design, Feedback, Load, Fairness, and Framework Conditions). An initial battery of 46 items was developed, drawing inspiration from the MFE-K scale used in the academic context.

Study III involved an Exploratory Factor Analysis (EFA) using principal axis factoring with Promax rotation on N=428 evaluations. Initial criteria suggested a four-factor solution, but subsequent item analysis revealed that the fourth factor was statistically weak and conceptually unstable due to cross-loadings. Items were reduced based on item difficulty, inter-item correlations, and factor loadings, resulting in a 15-item, three-factor solution.

Study IV conducted a Confirmatory Factor Analysis (CFA) on N=301 evaluations. The CFA confirmed the acceptable to good fit of the three-factor model. A further reduction of three items due to low factor loading (λ ≤ 0.50) and high error variance resulted in the final, validated version of the FIRE-P, consisting of 12 core items distributed across the three stable factors: Vorbereitung (Preparation), Gestaltung (Design), and Belastung (Load).

Instrument

Test Type: Feedback and Quality Evaluation Questionnaire

Format: Self-report questionnaire, utilizing primarily a 7-point Likert scale (for core items 1-15, 18-22) and additional open or dichotomous formats for optional items.

Language Available: German (deutsch)

Population Group: Professionals in rescue services and fire brigades, specifically those participating in leadership and executive training courses.

Age Group: Adults (Age range in validation samples: 21 to 61 years).

Population Details: Participants in Group and Platoon leader courses and other non-technical leadership training at the Institut der Feuerwehr Nordrhein-Westfalen (IdF NRW). The validation samples were predominantly male (93.9% in Study III).

Test Methodology: The core instrument consists of 12 items (plus 12 optional items related to framework conditions and bias variables, resulting in a total of 24 items). Scoring involves calculating mean values for each of the three scales (Preparation, Design, Load). High scores are desired for Preparation and Design, while low scores are desired for Load (or high scores on the recoded Load scale).

Keywords

psychological scale, psychometric properties, emergency services, fire brigade, examination, leadership training, validation, evaluation

Authors

Author ORCID Identifier: Not provided in source content.

Affiliation Email addresses: [email protected] (Meinald Thielsch), [email protected] (Jessica Busjan), [email protected] (Katharina Frerichs)

Correspondence Address: Institut für Psychologie, Westfälische Wilhelms-Universität Münster

Permissions & Fee and Test Year

Permissions: Information regarding formal permissions and licensing is not explicitly detailed in the source content, but the scale was developed in cooperation with the Institut der Feuerwehr Nordrhein-Westfalen (IdF NRW).

Fee: Not specified in source content.

Test Year: The scale development and validation studies (Studies I-IV) were conducted around 2016/2017.

Reference’s

• Bechler & Thielsch (2012).
• Beierlein, Kovaleva, Kemper und Rammstedt (2014).
• Bühner (2011).
• Busjan (2016).
• Colquitt & Simmering (1998).
• Colquitt, LePine und Noe (2000).
• Feuerwehr-Dienstvorschrift 3 (2008).
• Feuerwehr-Dienstvorschrift 100 (1999).
• Felfe, Six, Schmook und Knorz (2014).
• Frerichs (2017).
• Froncek & Thielsch (2014).
• Froncek, Hirschfeld und Thielsch (2014).
• Gilbert, Stunkard, Jensen, Detwiler, & Martinko (1996).
• Hagemann (2011).
• Jäger (2004).
• Jakoby und Jacob (2014).
• Kemper, Beierlein, Kovaleva und Rammstedt (2014).
• Lee & Olshfski (2002).
• Martocchio & Judge (1997).
• Modick (2014).
• Mummendey & Grau (2014).
• Noe (1986).
• Ong, Bergeman, Wallace, & Bisconti (2006).
• Richardson, Abraham, und Bond (2012).
• Sarubin et al. (2015).
• Scheier & Carver (1985).
• Schmidt und Riniolo (1999).
• Schuler und Prochaska (2001).
• Schulte & Thielsch (in press).
• Schupp und Gerlitz (2014).
• Smith & Dyal (2016).
• Spooren, Brockx, & Mortelmans (2013).
• Wachtel (1998).
• West, Finch und Curran (1995).
• Young & Cooper (1995).

Items of the feedback tool for rescue force development examinations (FIRE-P)

IMPORTANT: The following scale items must be preserved in their original language and must not be changed in any way.

Tabelle 1

Items der Skala FIRE-P

Tabelle 2

Je nach Evaluationskontext optionale Items