AI Risks and Benefits Scale

Abstract

The AI Risks and Benefits Scale (ARBS), developed by Kerstan, Bienefeld, and Grote in 2024, is a specialized psychometric instrument designed to quantify individuals’ perceptions of potential risks and anticipated benefits associated with the implementation of Artificial Intelligence (AI)-based technologies within the healthcare sector. This 19-item questionnaire was intentionally constructed to assess generalized attitudes toward AI in clinical settings rather than focusing on specific applications. The scale originated from a study aimed at comparing preferences for human doctors versus AI-generated treatment recommendations among an online sample of adults recruited in the United States. Its development involved deriving themes from prior research, conducting a pretest, and rigorously assessing the psychometric properties through Factor Analysis and reliability assessments.

The ARBS measures two primary constructs: Risk Perceptions and Benefit Perceptions, utilizing a 7-point Likert-type scale ranging from “Very Unlikely” to “Very Likely.” Initial validation efforts confirmed acceptable internal consistency for both subscales and supported a multi-factor structure representing distinct risk and benefit dimensions.

Keywords

Artificial Intelligence, AI in Healthcare, Risk Perceptions, Benefit Perceptions, Client Attitudes, Health Care Delivery, Human-Computer Interaction, Telemedicine, Therapeutic Processes, Treatment.

Authors

Sophie Kerstan, Nadine Bienefeld, Gudela Grote

Purpose

The primary purpose of the AI Risks and Benefits Scale is to systematically measure the perceived likelihood of specific risks or benefits occurring due to the integration of artificial intelligence into day-to-day healthcare practice. It serves as a diagnostic tool for understanding public and patient apprehension or optimism regarding technological disruption in medicine.

By quantifying these perceptions, the scale enables researchers and policymakers to gauge acceptance levels and identify specific areas of concern (e.g., data security, ethical issues, dehumanization of care) or positive anticipation (e.g., personalized care, error reduction) related to AI implementation, independent of any specific AI application.

Construct

The scale measures an overarching construct defined as Attitudes toward Artificial Intelligence in Healthcare. This construct is operationalized through two distinct subscales: Risk Perceptions and Benefit Perceptions.

The Risk Perceptions subscale captures concerns related to potential negative outcomes, such as ethical dilemmas, job loss among professionals, data breaches, and medical inaccuracies resulting from AI reliance. Conversely, the Benefit Perceptions subscale measures optimism regarding positive outcomes, including improved accessibility, accurate diagnostics, personalized treatment planning, and reduction of medical errors.

Validity

While the initial summary does not explicitly report external or criterion validity measures, the structural validity of the scale was thoroughly examined through both Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis (CFA). These analyses were critical in establishing that the items appropriately load onto the intended theoretical dimensions of risk and benefit perceptions, thereby confirming the internal structure of the instrument.

The distinction between risk and benefit factors was supported by the finding that a model combining both perceptions into a single factor showed significantly poorer fit compared to the preferred multi-factor model.

Reliability

The AI Risks and Benefits Scale demonstrated acceptable levels of internal consistency across its primary subscales, indicating that items within each factor measure the underlying construct cohesively. Reliability was assessed using Cronbach’s alpha.

• Risk Perceptions: Internal consistency was reported as Cronbach’s alpha = 0.87.

• Benefit Perceptions: Internal consistency was reported as Cronbach’s alpha = 0.84.

Factor Analysis

Extensive factor analysis procedures were undertaken to establish the optimal factor structure of the 19 items.

Exploratory Factor Analysis (EFA): The EFA identified a two-factor solution for the risk perception items, while the benefit perception items generally formed a single factor. One benefit perception item was excluded during this phase due to its failure to load appropriately onto the intended factor, resulting in the final 19-item scale.

Confirmatory Factor Analysis (CFA): Subsequent Confirmatory Factor Analysis supported a five-factor model (likely reflecting the two main subscales further broken down into specific thematic risk/benefit clusters). This model showed acceptable fit statistics: (χ² = 1008.46, df = 547, p < 0.001, χ²/df = 1.84, CFI = 0.91, TLI = 0.90, RMSEA = 0.04, SRMR = 0.06). Crucially, a competing four-factor model that attempted to combine risk and benefit perceptions into a single factor demonstrated significantly poorer fit, reinforcing the conceptual separation between perceived risks and perceived benefits regarding AI in healthcare (Link 2/5).

Instrument

Test Type: Original

Format: Participants rate items on a 7-point scale (1 = Very Unlikely to 7 = Very Likely).

Language Available: English (as utilized in the original U.S. study sample).

Population Group: Human (Male; Female)

Age Group: Adulthood (18 years & older)

Population Details: Respondents were adult participants recruited via Prolific, located in the United States.

Test Methodology: Test Reliability, Internal Consistency, Factor Analysis (Link 2/5), Confirmatory Factor Analysis, Exploratory Factor Analysis.

Authors

Author ORCID Identifier:

• Sophie Kerstan: ORCID

• Nadine Bienefeld: ORCID

• Gudela Grote: ORCID

Affiliation Email addresses: Sophie Kerstan: [email protected]

Correspondence Address: Sophie Kerstan, ETH Zurich, Department of Management, Technology, and Economics, Work and Organizational Psychology, Weinbergstrasse 56/58, Zurich, Switzerland, 8092, [email protected]

Permissions & Fee and Test Year

Test Year: 2024

Permissions and Licensing: The research underlying the scale development is associated with a Creative Commons License (CC BY 4.0), suggesting open access and use, provided appropriate attribution is given to the original authors.

Administration Method: Electronic questionnaire administration.

Reference’s

Kerstan, S., Bienefeld, N., & Grote, G. (2024). AI Risks and Benefits Scale: Measuring risk-benefit perceptions of AI-based technologies in healthcare. ETH Zurich, Department of Management, Technology, and Economics.

Items of the AI Risks and Benefits Scale

IMPORTANT: The following scale items must be preserved in their original language and must not be changed in any way.

Stimulus Prompt: In your opinion, how likely are the following risks or benefits to occur? AI in healthcare might …

Response Options: The scale uses a 7-point rating system, where 1 = Very Unlikely, 4 = Neutral, and 7 = Very Likely.

Risk Perceptions Items (R1-R10)

• R1: … fail to recognize the uniqueness of each patient’s condition

• R2: … result in a loss of jobs for healthcare professionals

• R3: … increase patient data security breaches

• R4: … lead to ethical problems

• R5: … introduce new bugs and equipment failures

• R6: … result in overreliance on technology

• R7: … dehumanize care

• R8: … increase medical errors

• R9: … provide unreliable information

• R10: … measure patient parameters inaccurately

Benefit Perceptions Items (B1-B9)

• B1: … reduce medical errors

• B2: … enable a more personalized care

• B3: … facilitate the prediction of negative health events

• B4: … improve information sharing between patients and healthcare providers

• B5: … improve the accessibility of care

• B6: … provide more accurate health data measurements

• B7: … help in monitoring treatment efficiency

• B8: … help in diagnosing health problems

• B9: … help in choosing adequate treatments