Münster questionnaire for evaluation – additional module digital teaching (MFE-ZdL)

Abstract

The Münster questionnaire for evaluation – additional module digital teaching (MFE-ZdL) is a 10-item battery designed for course evaluation by students in higher education. Its primary function is to capture student assessments regarding various aspects of digital teaching. Developed in response to the rapid shift to online learning necessitated by the COVID-19 pandemic, the scale was initially utilized at the Institute of Psychology of the Westfälische Wilhelms-Universität Münster (WWU) starting in the Summer Semester 2020.

The MFE-ZdL serves as an additional module to established basic questionnaires, such as the Münster Questionnaire for the Evaluation of Seminars (MFE-Sr) or Lectures (MFE-Vr). While optimally suited for evaluating purely digital events, it can also be used in a reduced form for hybrid courses or traditional classroom courses supplemented with digital materials. It provides essential feedback to instructors and institutional steering units regarding the quality and student experience of digital learning formats.

Keywords

Course Evaluation, Digital Teaching, Online Lecture, Higher Education, Student Feedback, COVID-19, Psychometrics, Digital Learning.

Authors

Evaluation Team of the Institute of Psychology, Westfälische Wilhelms-Universität Münster (WWU).

Purpose

The core purpose of the MFE-ZdL is to provide specific, actionable feedback to university lecturers and administrative units concerning the implementation and effectiveness of digital elements within courses. The scale was created to ensure comprehensive quality assurance during the period when digital formats became integral to almost all university teaching (Summer 2020 through Summer 2021).

The instrument is designed for economical, short administration (estimated 1 to 2 minutes processing time). It captures facets of digital instruction not covered by standard evaluation instruments. While primarily intended for internal university evaluation and feedback, the item battery is also suitable for use in empirical studies related to educational science, higher education didactics, and pedagogical psychology, offering differentiated insights into student experiences with digital learning formats. The instrument is explicitly not intended for individual diagnosis of students.

Construct

The MFE-ZdL measures students’ appraisals of various specific aspects of digital teaching. Unlike general course evaluation instruments, this module focuses on unique challenges and features introduced by digital formats.

The 10 items cover conceptually independent aspects, including the perceived fit of digital elements to the course content, the extent to which learning objectives were achieved compared to in-person classes, the adequacy of opportunities for reflection and student collaboration, the technical demands and associated extra workload, and the perceived accessibility of the instructor. Furthermore, the questionnaire assesses the specific impact of the digital format on student learning behavior, such as self-determination, time management, and procrastination tendencies.

Validity

Evidence for both content and construct validity is reported. Content validity was established through repeated review by experts, including the evaluation team members, involved lecturers, and the Dean of the Faculty, ensuring that the items covered the relevant quality characteristics of digital teaching broadly and unambiguously.

Construct validity (convergent and divergent) was examined by correlating the six rating scale items with scales and items from the core evaluation questionnaires (MFE-Sr/MFE-Vr), such as “Lecturer & Didactics,” “Overload,” and “Learning Success.” Observed correlations generally showed medium to high associations with related constructs (convergent validity). However, correlations with unrelated constructs suggested that students’ overall judgment of the course might influence their responses to specific items. To address this, partial correlations were calculated, controlling for the overall impression represented by the “Lecturer & Didactics” scale. This analysis demonstrated that most specific items (e.g., technical effort, student reflection) retained significant correlation with related constructs, supporting their unique contribution beyond general course satisfaction.

Reliability

The items comprising the MFE-ZdL are designed to measure distinct, conceptually independent aspects of digital teaching, meaning they are not homogenous manifestations of a single latent construct. Consequently, traditional measures of internal consistency and split-half reliability are not considered meaningful indicators of reliability for this battery.

For transparency, the Cronbach’s Alpha calculated across the six rating-scale items (Items 3-8) was reported as alpha = .73. Furthermore, due to the nature of anonymous student feedback, the determination of retest reliability was not feasible.

Factor Analysis

An Exploratory Factor Analysis (EFA) (Maximum Likelihood with Oblimin rotation) was performed, although the scale is not intended to measure a single, underlying factor structure. The initial version used in Summer Semester 2020 included three negatively phrased rating items. The EFA results for this version indicated that these three items loaded onto the same factor, suggesting the presence of a method factor related to negative phrasing, despite the items covering different content aspects.

Based on this finding, the instrument was revised. The three negatively formulated items were removed, and positively formulated equivalents (Items 4, 5, and 7 in the final version) were introduced. The final 10-item structure was confirmed, and since the items are intended to measure distinct aspects of digital teaching, the formation of subscales based on EFA factors is neither expected nor recommended.

Instrument

Test Type: Psychological Scale / Course Evaluation Instrument

Format: 10 items, utilizing a mix of 7-point Likert scales (6 items), multiple-choice selection (2 items), and open-ended text fields (2 items).

Language Available: German (deutsch)

Population Group: University Students (Higher Education)

Age Group: 17 to 67 years (Sample Mean = 23.08, SD = 3.61)

Population Details: The validation sample included 3,895 participating students (N=5,154 evaluations) from the Psychology Department at WWU Münster, collected during the Winter Semester 2020/21 and Summer Semester 2021. Students were enrolled in various programs (Teaching Certificate, Bachelor, Master, Minor in Psychology, and others) and ranged from the 1st to the 20th semester (M = 3.27, SD = 2.26).

Test Methodology: The instrument is administered online using the CASI (Computer-Assisted Self-Interviewing) mode. It is also suitable for PASI (Paper-and-Pencil Self-Interviewing). The estimated processing time is minimal, ranging from 1 to 2 minutes.

Keywords

Evaluation, Digitales Lernen, Hochschuldidaktik, Student Assessment, Psychometrics, E-Learning, Remote Learning, Academic Evaluation.

Authors

Author ORCID Identifier: Not provided in source content.

Affiliation Email addresses: [email protected]

Correspondence Address: Evaluation Team of the Institute of Psychology, Westfälische Wilhelms-Universität Münster, Germany. Further information available at: https://www.uni-muenster.de/PsyEval

Permissions & Fee and Test Year

The MFE-ZdL was developed starting in April 2020 in response to the COVID-19 related shift to digital teaching. It was validated using data collected during the 2020/2021 academic year.

The instrument is designed for institutional use in course evaluation and feedback to lecturers. While conceived for internal evaluation at WWU Münster, its use in empirical studies related to higher education, pedagogy, and educational psychology is explicitly encouraged by the authors. No specific fee information is provided, suggesting it is freely available for non-commercial academic use, though permissions for external use should be requested from the contact address.

Reference’s

• Bates, D., Maechler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48.

• Burnham, K. P., & Anderson, D. R. (2004). Multimodel inference: Understanding AIC and BIC in model selection. Sociological Methods & Research, 33(2), 261–304.

• Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155–159.

• Haaser, A., Thielsch, M. T., & Hirschfeld, G. (2007). Online-Evaluationen von Lehrveranstaltungen an der Universität Münster. Zeitschrift für Evaluation, 6(2), 241–261.

• R Core Team (2022). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria.

• Revelle, W. (2022). psych: Procedures for psychological, psychometric, and personality research (Version 2.2.5). Northwestern University, Evanston, Illinois.

• Thielsch, M. T., & Hirschfeld, G. (2010a). Münsteraner Fragebogen zur Evaluation von Seminaren (MFE-Sr). Evaluationsteam des Instituts für Psychologie, Westfälische Wilhelms-Universität Münster.

• Thielsch, M. T., & Hirschfeld, G. (2010b). Münsteraner Fragebogen zur Evaluation von Vorlesungen (MFE-Vr). Evaluationsteam des Instituts für Psychologie, Westfälische Wilhelms-Universität Münster.

• West, S. G., Finch, J. F., & Curran, P. J. (1995). Structural equation models with nonnormal variables: Problems and remedies. In R. H. Hoyle (Ed.), Structural equation modeling: Concepts, issues, and applications (pp. 56–75). Sage Publications, Inc.

Items of the Münster questionnaire for evaluation – additional module digital teaching (MFE-ZdL)

IMPORTANT: The following scale items must be preserved in their original language and must not be changed in any way.

Tabelle 1

Items des MFE-ZdL mit Item-spezifischem Antwortformat

Tabelle 2

Items des MFE-ZdL mit 7-stufiger Ratingskala als Antwortformat

Antwortvorgaben (Rating Items): For the items in Table 2, a 7-point scale is used: 1 = “stimme gar nicht zu” (strongly disagree), 2 = “stimme nicht zu” (disagree), 3 = “stimme eher nicht zu” (rather disagree), 4 = “neutral”, 5 = “stimme eher zu” (rather agree), 6 = “stimme zu” (agree), and 7 = “stimme vollkommen zu” (strongly agree). The option “nicht sinnvoll beantwortbar” (not meaningfully answerable) is also provided and coded as NA.