Computational Thinking Test for Lower Primary (CTtLP)

Abstract

The Computational Thinking Test for Lower Primary (CTtLP), developed by Zhang and Wong (2023), is a specialized instrument comprising 27 constructed-response items. Its primary function is to accurately assess the Computational Thinking (CT) abilities of students in lower primary grades (Grades 1–3), typically spanning ages 6 to 10.

The development process strictly adhered to the principles of Evidence-Centered Design (ECD). Initial item generation focused on three distinct scenarios requiring students to analyze instruction sequences, predict their output, and identify errors within the sequences. Following rigorous procedures—including expert reviews, cognitive interviews, field tests, and a pilot study—the final version maintained 27 of the original 30 items. Psychometric validation was performed using both Classical Test Theory (CTT) and Item Response Theory (IRT) on a sample of Grades 1–3 students from public schools in northern China. Participants were given 60 minutes to complete the assessment, and comprehensive results regarding the scale’s reliability, validity, and factor structure were reported.

Keywords

Computational Thinking, Classical Test Theory, Computer Sciences, Early Childhood Education, Error Identification, Evidence Centered Design, Instruction Sequence, Lower Primary Students, Psychoeducational Assessment.

Authors

Zhang, Shuhan, Wong, Gary K. W.

Purpose

The primary purpose of the CTtLP instrument is to provide a standardized, psychometrically sound measure for evaluating the acquisition and development of computational thinking skills in young children, specifically those enrolled in the lower primary grades (ages 6–10).

This assessment is crucial for researchers and educators seeking to track the effectiveness of early computer science curricula and intervention strategies aimed at building fundamental logical and algorithmic reasoning skills.

Construct

The CTtLP is designed to measure Computational Thinking (CT), which is generally defined as the mental process involved in formulating a problem and expressing its solution in a way that a computer—or human—can effectively execute. For lower primary students, this construct focuses specifically on foundational elements of CT, such as algorithmic thinking, decomposition, and abstraction, assessed through practical tasks.

The items evaluate two core CT components: the ability to correctly interpret the output resulting from specific instruction sequences (algorithmic understanding) and the capacity for error identification (debugging) within those sequences. These tasks represent the practical application of logical reasoning necessary for early programming concepts.

Validity

The validity of the CTtLP was established through multiple statistical approaches, confirming the measure’s suitability for assessing CT in this population.

Regarding Criterion Validity, the CTtLP scores demonstrated a moderate positive correlation with students’ overall course performance (r = 0.443, p < 0.000). This statistically significant correlation suggests that the test has fair criterion validity regarding academic success related to the skills measured.

In terms of Test Validity, analysis using the 3-Parameter Logistic (3PL) model within Item Response Theory (IRT) indicated that all items were retained, yielding acceptable psychometric properties. The mean difficulty index was 0.353 (ranging from -1.096 to 1.892), confirming that the test covers a broad spectrum of ability levels. Furthermore, the mean item discrimination was high at 2.188 (range: 1.437–3.331), demonstrating the items’ strong capacity to differentiate between students of high and low ability. Crucially, the mean guessing rate was very low (0.136), well below the acceptable threshold of 0.35, suggesting the inclusion of the “I don’t know” option effectively mitigated random guessing.

Reliability

The reliability of the CTtLP was assessed using both CTT and IRT methods, indicating high internal consistency and stability over time.

The Internal Consistency, measured by Cronbach’s alpha, was reported at 0.873, which is considered excellent for educational assessment. The Test-Retest Reliability, evaluated over an 8-week interval using the Intraclass Correlation Coefficient (ICC), was 0.757, confirming the instrument’s temporal stability.

The Test Reliability, examined via the Test Information Function (TIF) in the IRT analysis (Aesaert et al., 2014), provided significant insight. The test provided maximum information (14.65) when student ability was approximately 0.9. This finding suggests the CTtLP is most precise and reliable when measuring participants who possess marginally higher-than-average CT ability, though it maintains good coverage across a broad range of ability levels overall.

Factor Analysis

A Confirmatory Factor Analysis (CFA) was performed to test the underlying structure of the CTtLP data set. The researchers input a single-factor model to examine the fit of a unidimensional model, aligning with the intent to measure a singular underlying construct of computational thinking.

The fit indices demonstrated a proper model fit: RMSEA = 0.041, CFI = 0.911, and TLI = 0.900 (Hu & Bentler, 1999). These values confirm that the data set achieved unidimensionality, supporting the use of a single total score for the CTtLP.

Instrument

Test Type: Original

Format: Responses are provided using a multiple-choice format. Each item offers five choices: one correct key, three distractors, and a crucial additional option labeled ‘I don’t know’. This specific option is employed strategically to reduce the likelihood of random guessing and improve measurement accuracy.

Language Available: English; Chinese

Population Group: Human; Male; Female

Age Group: Childhood (birth–12 yrs); School Age (6–12 yrs)

Population Details:

• Age Range: 6–10 Yrs.

• Location: China

• Respondents: Lower Primary School Students (Grades 1–3)

Test Methodology: Test Validity; Content Validity; Criterion Validity; Test Reliability; Internal Consistency; Test-Retest Reliability; Factor Analysis; Confirmatory Factor Analysis; Item Response Theory

Keywords

Classical Test Theory, Critical Thinking, Logical Thinking, Primary School Students, Computational Modeling, Computer Science, Executive Functioning Measures, Psychoeducational Assessment, Early Childhood Education.

Authors

Author ORCID Identifier:

• Zhang, Shuhan: http://orcid.org/0000-0002-6493-9979

• Wong, Gary K. W.: http://orcid.org/0000-0003-1269-0734

Affiliation:

• Zhang, Shuhan: Faculty of Education, University of Hong Kong

• Wong, Gary K. W.: Faculty of Education, University of Hong Kong

Affiliation Email addresses:

• Zhang, Shuhan: [email protected]

Correspondence Address:

• Zhang, Shuhan: University of Hong Kong, Faculty of Education, Pokfulam Road, Hong Kong, China, [email protected]

Permissions & Fee and Test Year

Permissions: Contact Corresponding Author

Fee: No

Test Year: 2023

References

Zhang, S., & Wong, G. K. W. (2023). Development and validation of a computational thinking test for lower primary school students. Educational Technology Research and Development, 71(4), 1595–1630. https://doi.org/10.1007/s11423-023-10231-2

Aesaert, K., Van Braak, J., & Van Deursen, A. (2014). The reliability of ICT competence tests: A test information function approach. Computers & Education, 74, 15-25.

Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55.

Items of the Computational Thinking Test for Lower Primary (CTtLP)

IMPORTANT: The following scale items must be preserved in their original language and must not be changed in any way.

This measure consists of 27 items. No data is Available.