AI Knowledge Scale

Abstract

The AI Knowledge Scale (AIKS), developed by Kerstan, Bienefeld, and Grote in 2024, is a concise, six-item instrument designed to assess participants’ objective knowledge regarding fundamental principles of Artificial Intelligence (AI). This instrument was initially utilized within a larger study investigating the complex relationship between objective AI knowledge, trust associations, and preferences concerning the integration of AI systems into healthcare settings, particularly in comparison to human physicians. The scale’s development followed rigorous procedures, including an extensive literature review (e.g., Topol, 2019) and a pretest using a Prolific sample, culminating in evaluation via factor analysis, Item Response Theory (IRT), and standard reliability and validity assessments.

Keywords

Artificial Intelligence, Objective Knowledge, General Principles, Human-Computer Interaction, Knowledge Level, Trust in AI, Healthcare Technology, Psychometrics

Authors

Kerstan, Sophie, Bienefeld, Nadine, Grote, Gudela

Purpose

The primary purpose of the AI Knowledge Scale is to provide a reliable and objective measure of an individual’s verifiable understanding of general artificial intelligence concepts. Unlike subjective self-assessments of familiarity, the AIKS aims to capture precise knowledge of technical and conceptual aspects of AI, such as machine learning paradigms and computational limitations.

The scale is instrumental for research focusing on how actual knowledge levels influence attitudes, trust associations, and decision-making regarding emerging technologies. Specifically, it was designed to determine how knowledge relates to risk-benefit perceptions and preferences for AI integration in high-stakes fields like healthcare.

Construct

The AI Knowledge Scale measures the construct of Artificial Intelligence Knowledge. This construct is defined as the objective understanding of foundational AI principles, including knowledge of AI architecture (such as neural networks), performance metrics (like FLOPS), common misconceptions (such as bias and inherent autonomy), and key historical concepts (like the Turing test).

The six items specifically target core areas of AI literacy necessary for informed interaction with AI systems, ensuring that the measurement is grounded in established technical and theoretical knowledge rather than mere familiarity with the term “AI.”

Validity

Validity testing for the AI Knowledge Scale employed both advanced psychometric modeling and standard correlational analysis. Using Item Response Theory (IRT), a Mokken scale analysis was performed, confirming appropriate item scalability with a coefficient of H = 0.34, suggesting the items effectively measure a unidimensional construct across different levels of knowledge.

Furthermore, evidence for Construct Validity was established through the finding of a positive correlation between participants’ scores on the AI Knowledge Scale and measures of general scientific knowledge. This correlation supports the interpretation that the scale successfully measures a form of technical and factual understanding relevant to the scientific domain of computing, distinguishing it from general intelligence or non-specific technological affinity.

Reliability

The internal consistency of the AI Knowledge Scale was assessed using the Molenaar Sijtsma statistic, yielding a value of 0.62. This statistic, designed for objective knowledge tests where items may not be strictly parallel, is reported to be consistent with reliability statistics observed in similar, short-form objective knowledge measures documented in the literature (Bearth et al., 2019).

It is important to note that objective knowledge scales often exhibit lower internal consistency (relative to attitude scales) because the items are designed to test distinct factual domains, meaning not all items are expected to correlate perfectly with one another.

Factor Analysis

The structural integrity of the AI Knowledge Scale was examined using Confirmatory Factor Analysis (CFA). A specific five-factor model was tested and demonstrated acceptable psychometric fit to the collected data. Key fit indices confirmed the model’s appropriateness:

• Chi-square (χ²) = 1008.46, with df = 547 (p < 0.001)
• Ratio of Chi-square to degrees of freedom (χ²/df) = 1.84
• Comparative Fit Index (CFI) = 0.91
• Tucker-Lewis Index (TLI) = 0.90
• Root Mean Square Error of Approximation (RMSEA) = 0.04
• Standardized Root Mean Square Residual (SRMR) = 0.06

These indices, specifically the CFI and RMSEA, indicate a good fit between the hypothesized factor structure and the observed data. The analysis further confirmed that all six items significantly contributed to their designated factors, supporting the intended conceptual structure of the scale.

Instrument

Test Type: Original Rating Scale (Objective Knowledge)

Format: The scale consists of six True/False statements where participants also indicate their confidence level. The scoring method requires both correctness and high confidence to register a point. Participants indicate confidence on a four-point scale (0 = unsure, 1 = slightly unsure, 2 = slightly sure, 3 = sure). Correct responses coupled with the highest confidence level (score of 3) are coded as 1; all other responses (incorrect, or correct but uncertain) are coded as 0.

Language Available: English (Original)

Population Group: Human (Male and Female)

Age Group: Adulthood (18 years and older)

Population Details: Respondents were adult participants located in the United States. The administration method was electronic.

Test Methodology: Test Validity; Construct Validity; Test Reliability; Internal Consistency; Factor Analysis; Confirmatory Factor Analysis; Item Response Theory

Keywords

Artificial Intelligence, Objective Knowledge, Psychometric Scale, Trust in AI, Healthcare Technology, Molenaar Sijtsma statistic, Human-Computer Interaction Measures

Authors

Author ORCID Identifier:

• Sophie Kerstan: ORCID
• Nadine Bienefeld: ORCID
• Gudela Grote: ORCID

Affiliation Email addresses: Sophie Kerstan: [email protected]

Correspondence Address: Sophie Kerstan, ETH Zurich, Department of Management, Technology, and Economics, Work and Organizational Psychology, Weinbergstrasse 56/58, Zurich, Switzerland, 8092

Permissions & Fee and Test Year

The AI Knowledge Scale was developed and published in 2024. As no public files are available, researchers interested in utilizing the scale must contact the corresponding author, Sophie Kerstan, for permission and usage details. The scale was published in the journal Risk Analysis.

Reference’s

Kerstan, S., Bienefeld, N., & Grote, G. (2024). Choosing human over AI doctors? How comparative trust associations and knowledge relate to risk and benefit perceptions of AI in healthcare. Risk Analysis, 44(4), 939–957. https://doi.org/10.1111/risa.14216

Items of the AI Knowledge Scale

IMPORTANT: The following scale items must be preserved in their original language and must not be changed in any way.

The scale consists of six True/False statements covering core AI concepts:

• Item 1: Deep learning employs artificial neural networks with multiple layers. (Correct Response: True)
• Item 2: AI-based outputs are free of biases. (Correct Response: False)
• Item 3: FLOPS is a measure of computer performance. (Correct Response: True)
• Item 4: Unsupervised machine learning methods make use of training cases with labelled data. (Correct Response: False)
• Item 5: The Turing test determines if a human is more intelligent than a machine. (Correct Response: False)
• Item 6: At its core, AI always relies on decision rules that are predefined by humans. (Correct Response: False)

Participants respond using a two-faceted scale, first selecting True/False, and then rating their certainty:

1. Unsure (1)
2. Fairly unsure (2)
3. Fairly sure (3)
4. Sure (4)