Agentic Engagement in Massive Open Online Courses Scale

Abstract

The Agentic Engagement in Massive Open Online Courses Scale (AEMOOCS), developed by Kim and Song in 2023, is a specialized inventory designed to quantify the level of agentic engagement demonstrated by learners participating in self-paced Massive Open Online Courses (MOOCs). This measure is crucial for assessing how learners proactively shape their distance learning environment.

The scale development adhered to Hinkin’s (1995) three-step procedure. Item generation began with an extensive literature review, heavily informed by the existing Agentic Engagement Scale (Reeve, 2013). A subsequent Delphi survey involving expert consensus helped refine the initial item pool, resulting in a set of 14 items.

Rigorous psychometric evaluation followed, including assessment of Content Validity Index (CVI) values, leading to the establishment of a final, seven-item scale. These items were administered to learners utilizing the K-MOOC system. Factor analysis confirmed a robust three-dimensional structure, providing strong empirical evidence supporting the reliability and validity of the measure for use in online learning contexts.

Keywords

Student Engagement, Agentic Engagement, Self-Paced Massive Open Online Courses, Agentic Support Request, Agentic Learning Strategy, Agentic Learning Construction, Psychometrics, Factor Analysis, Cronbach’s alpha.

Authors

Kim, Rang; Song, Hae-Deok

Purpose

The primary purpose of the Agentic Engagement in Massive Open Online Courses Scale is to provide a reliable and valid psychometric tool for researchers and educators to systematically quantify the level of agentic engagement exhibited by individuals enrolled in self-paced massive open online courses. This assessment is vital for understanding how learners proactively influence their instruction and learning outcomes in autonomous digital environments.

The scale is designed to capture the unique dimensions of proactive learning behavior necessary for success in MOOCs, where the lack of traditional structure places a high demand on learner initiative. By measuring this construct, the scale facilitates targeted educational interventions aimed at enhancing student autonomy and strategic learning behaviors in distance education settings.

Construct

The scale measures Agentic Engagement, a proactive form of student involvement where learners intentionally contribute to the flow of instruction and personalize their learning experiences. Unlike passive forms of engagement, agentic behavior emphasizes the learner’s volitional actions to shape the learning environment to better suit their individual needs and goals.

The seven-item scale is empirically structured around three distinct dimensions of this construct, as confirmed through factor analysis:

1. Agentic Support Request (ASR): Measures the proactive seeking of help, references, or resources from instructors or peers.
2. Agentic Learning Strategy (ALS): Measures the intentional application of self-regulated strategies, such as organizing ideas and seeking relevant supplemental content.
3. Agentic Learning Construction (ALC): Captures the learner’s efforts to personalize and reorganize the course material and learning sequence (e.g., choosing among video lectures, quizzes, and forums) for more effective understanding.

Validity

The Agentic Engagement in Massive Open Online Courses Scale demonstrates strong evidence of psychometric validity across multiple rigorous statistical analyses, confirming its appropriateness for educational research.

Content and Construct Validity: Content validity was established through expert review via the Delphi survey and the calculation of the Content Validity Index (CVI). Construct validity was confirmed by Factor Analysis results, which provided robust support for the structural integrity of the measure, showing that the items appropriately represent the theoretical three-factor construct.

Convergent and Discriminant Validity: Convergent validity was established as all standardized factor loadings exceeded the threshold of 0.5, with a high mean magnitude of 0.77, surpassing the recommended threshold of 0.7 (Hair et al., 2018). Discriminant validity was also confirmed through two measures: the square root of the Average Variance Extracted (AVE) for each construct (ranging from 0.71 to 0.91) was consistently higher than the correlations observed among the constructs. Additionally, the Heterotrait-Monotrait (HTMT) ratio, which ranged from 0.56 to 0.78, indicated adequate distinction between the factors (Henseler et al., 2015).

Reliability

The scale exhibits high reliability, demonstrated through consistent internal consistency measures across both exploratory and confirmatory analyses.

During the Exploratory Factor Analysis (EFA), the overall scale achieved a high reliability coefficient of 0.87, which surpasses the standard threshold of 0.7 (Nunnally & Bernstein, 1994). Reliability for the individual dimensions during EFA was also strong: Agentic Support Request (0.86), Learning Strategy (0.87), and Learning Construction (0.91).

For the Confirmatory Factor Analysis (CFA), Cronbach’s alpha values were 0.91 for Agentic Support Request, 0.68 for Learning Strategy, and 0.76 for Learning Construction. Furthermore, Composite Reliability (CR) values were strong across all factors (0.91 for ASR, 0.75 for ALS, and 0.76 for ALC), all successfully exceeding the minimum reliability threshold of 0.60 (Fornell & Larcker, 1981).

Factor Analysis

Both Exploratory and Confirmatory Factor Analyses were critical in establishing the underlying structure of the Agentic Engagement in Massive Open Online Courses Scale.

Exploratory Factor Analysis (EFA): EFA was conducted using Exploratory Structural Equation Modeling (ESEM). Initial tests demonstrated that 1-factor and 2-factor models provided an inadequate fit to the data. Parallel analysis provided robust support for the 3-factor model as the optimal structure, as the eigenvalue for random data (0.99) began to surpass that of the actual data (0.50) when considering a 4-factor model.

Confirmatory Factor Analysis (CFA): CFA was performed to confirm the fit of the hypothesized 3-factor structure. The results indicated an acceptable model fit, evidenced by key fit indices: χ²v2 = 30.972, df = 10, TLI = 0.938, CFI = 0.970, and SRM = 0.048. These statistics affirm that the three dimensions (Agentic Support Request, Agentic Learning Strategy, and Agentic Learning Construction) are distinct and well-represented by the collected data.

Instrument

Test Type: Original

Format: The items are rated using a 5-point Likert-type scale.

Language Available: English

Population Group: Human (Male and Female).

Age Group: Adulthood (18 years and older), specifically including Young Adulthood (18-29 years), Thirties (30-39 years), and Middle Age (40-64 years).

Population Details: The respondents were university students located in the Republic of Korea, who were utilizing the K-MOOC system.

Test Methodology: The methodology included Test Validity (Construct Validity, Content Validity, Convergent Validity, Discriminant Validity), Test Reliability (Internal Consistency), Factor Analysis (Confirmatory Factor Analysis, Exploratory Factor Analysis), and Structural Equation Modeling.

Keywords

Psychological Inventory, Student Autonomy, Online Learning Engagement, Likert Scale, Educational Measurement, Structural Equation Modeling, K-MOOC.

Authors

Kim, Rang

• Author ORCID Identifier: 0000-0002-4657-1170
• Affiliation: Chung-Ang University Center for Teaching and Learning

Song, Hae-Deok

• Author ORCID Identifier: 0000-0003-0879-9999
• Affiliation Email addresses: [email protected]
• Correspondence Address: [email protected]

Permissions & Fee and Test Year

Permissions: Contact Publisher

Fee: No

Test Year: 2023

Commercial: No

Reference’s

Kim, R., & Song, H.-D. (2023). Developing an agentic engagement scale in a self-paced MOOC. Distance Education, 44(1), 120–136. doi.org/10.1080/01587919.2022.2155619

Bong et al. (2012). (Referenced for comparison of Korean item versions during scale development).

Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 18(1), 39-50.

Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2018). Multivariate data analysis (8th ed.). Cengage Learning.

Henseler, J., Ringle, C. M., & Sarstedt, M. (2015). A new criterion for assessing discriminant validity in variance-based structural equation modeling. Journal of the Academy of Marketing Science, 43(1), 115-135.

Hinkin, T. R. (1995). A review of scale development practices in the study of organizations. Journal of Management, 21(5), 967-988.

Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). McGraw-Hill.

Reeve, J. (2013). How students create their own motivation and engagement: Agentic engagement. Journal of Educational Psychology, 105(3), 579–595.

Reeve, J., & Tseng, C. M. (2011). Agency as a fourth element of student engagement. Contemporary Educational Psychology, 36(4), 257-266.

Items of the Agentic Engagement in Massive Open Online Courses Scale

Number of Items: This is a 7-item measure.

Factors and Subscales: The scale comprises 3 dimensions:

• Agentic support request

• Learning strategy

• Learning construction

Test Items Available: Yes

Note: ASR: agentic support request; ALS: agentic learning strategy; ALC: agentic learning construction. Items are rated using a 5-point Likert-type scale that ranges from “strongly disagree” to “strongly agree”.