Daily Experience of Simulation Training (DEST)

Abstract

The Daily Experience of Simulation Training (DEST), developed by Chiu et al. (2023), is a specialized self-report instrument consisting of 20 items. Its primary function is to measure daily, transient fluctuations in self-reported confidence levels among child welfare trainees during intensive training periods. Beyond measuring confidence, the DEST serves to gather crucial feedback regarding the training team and facilitates self-reflection among participants.

This scale is integral to the evaluation of the week-long simulation training program delivered by the Child Protection Training Academy (CPTA), a program initially established at the University of Illinois at Springfield (UIS) and supported by the Illinois Department of Children and Family Services (DCFS). Initial validation efforts, utilizing a sample of new child protection investigators, included reported results for factor analysis and reliability. The instrument’s overarching purpose is to provide continuous evaluation for each cohort participating in this statewide training initiative.

Keywords

Child Protection Trainees, Child Welfare, Confidence, Simulation Training Effectiveness, Program Evaluation, Professional Competence, Protective Services, Professional Development

Authors

Chiu, Yu-Ling, Cross, Theodore P., Wheeler, Amy B., Evans, Susan M., Goulet, Betsy P.

Purpose

The fundamental purpose of the DEST instrument is to provide a structured, daily evaluation of cohorts participating in the intensive, week-long statewide simulation training program specifically designed for new child protection investigators.

This daily measurement approach allows for the tracking of immediate learning outcomes, monitoring changes in self-efficacy, and assessing emotional responses to the high-stakes training environment, thereby allowing the CPTA to refine its curriculum and delivery methods.

Construct

The primary psychological construct measured by the DEST is self-reported confidence, specifically focusing on a trainee’s perceived ability to execute critical skills required for child protection work. The scale assesses confidence across 13 distinct skill areas, ranging from gathering information from collateral contacts and engaging families to assessing safety and handling court testimony.

A secondary set of constructs assessed includes the perceived quality of feedback received from professional roles (e.g., simulation and classroom trainers) and the trainee’s capacity for daily self-reflection, captured through open-ended log questions.

Validity

The initial documentation for the Daily Experience of Simulation Training (DEST) does not explicitly indicate traditional measures of scale validity (such as criterion validity or construct validity) in the provided literature. However, the subsequent publication detailing its development and application suggests that the instrument is grounded in the operational requirements of the Child Protection Training Academy (CPTA), implying strong face validity and content validity related to the training curriculum.

Reliability

The scale demonstrates exceptional internal consistency across repeated administrations throughout the training week. The Cronbach’s alpha reliability coefficients for the primary confidence scale were consistently reported to be larger than 0.95 across all six measurement time points. This high level of internal consistency confirms the strong homogeneity and reliability of the multi-item scale in measuring the underlying construct of confidence daily.

Factor Analysis

A Principal Component Analysis (PCA) utilizing varimax rotation was conducted on the confidence items at each of the six time points during the training week. The results consistently yielded a clear, single-factor structure at every administration.

The eigenvalues for this first factor demonstrated high explanatory power, ranging from 9.28 (accounting for 71.4% of the variance) to 9.94 (accounting for 76.5% of the variance). This finding strongly supports the measure’s unidimensionality regarding the core construct of self-reported confidence in investigative skills.

Instrument

Test Type: Original Inventory/Questionnaire

Format: The instrument, comprising 20 items, is administered electronically. Trainees rate their confidence across 13 specific skill items using a 7-point scale (1 being low, 7 being high). Additionally, trainees provide feedback on the perceived quality of support from professional roles (simulation trainer, classroom trainer) using a 4-point Likert scale, ranging from “very helpful” to “very unhelpful.” The DEST also incorporates open-ended questions for daily self-reflection.

Language Available: English

Population Group: Human; Male; Female

Age Group: Adulthood (18 yrs & older)

Population Details: Location: United States; Respondents: New Child Protection Investigators

Test Methodology: Test Reliability; Internal Consistency; Factor Analysis; Principal Component Analysis

Keywords

Child Welfare, Professional Competence, Professional Development, Program Evaluation, Protective Services, Training, Professional Personnel Measures

Authors

Author ORCID Identifier: No data is Available

Affiliation Email addresses: Chiu, Yu-Ling: [email protected] (University of Illinois at Urbana-Champaign, School of Social Work Children and Family Research Center)

Correspondence Address: Chiu, Yu-Ling: [email protected]

Permissions & Fee and Test Year

Permissions: Contact Publisher

Commercial: No

Fee: No

Test Year: 2023

Reference’s

Chiu, Y.-L., Cross, T. P., Wheeler, A. B., Evans, S. M., & Goulet, B. P. (2023). Development and application of a self-report measure for measuring change during simulation training in child protection. Journal of Public Child Welfare, 17(2), 239–257. https://doi.org/10.1080/15548732.2021.2016546

Items of the Daily Experience of Simulation Training (DEST)

This measure consists of 20 items.
Subscale: Confidence.

Part I: Survey Content

• At which site are you taking the training?

• With (1) being lowest and (7) being highest, please check the appropriate number to indicate your level of confidence in the following skill areas TODAY.

• Please answer the following questions regarding the feedback that you received in today’s training:

• Today’s reflective log: What were the most meaningful concepts or skills you learned today?

---

Part II: Implementation Guidelines for Simulation Trainers

1. BEFORE the simulation training:

• Please send the following survey link to trainees before the simulation training week: (link withhold).

2. Implementation timing on each day DURING the simulation training week:

• Day 1 morning: do the baseline survey as soon as the orientation starts. Give them time to do it BEFORE you proceed to do any simulation training preparation or activities.

• Day 2 through Day 5 afternoons: Do the survey in the afternoon after the debriefings but before making arrangements for the next day.

• Day 5: Do the survey BEFORE you tell them the true stories of the Rhodes and Jones cases.

3. Trainee Privacy:

• Please respect trainees’ privacy when they are taking the survey.

4. What to tell trainees:

• Please explain the purpose of doing the survey every day and why the survey is important to DCFS and to trainees. The log section can help them to reflect on what they’ve learned every day.

• The survey can be done on laptops, computers, smart phones, or tablets.

• The email address has to be all in lowercase. They have to use their DCFS email throughout the week. (Please contact the evaluation team ASAP if you have a trainee without a DCFS email. We will make a special arrangement for the week).

• Only the CFRC team has the access to their email information and won’t use it for any analysis.

• The survey is anonymous. They can quit anytime and they skip any questions.

5. Trouble shooting:

• Please ask trainees to click the link from your email every time when taking the survey. If they would like to save the link on their browser, they have to remember to reload the page in his/her web browser. Otherwise, they won’t be able to access the right survey. (How to reload a webpage: Ctrl + F5 for Windows, Command + Shift + R for Mac, or Command + R on Safari for Mac).

• There have been instances where upon clicking the ‘Submit’ button at the end of the survey, if their answers were not submitted or even saved, please tell trainees to reload the page in his/her web browser.

• Should this happen, please have trainees re-enter their responses and submit again.

• After successful confirmation and submission of their answers, trainees should arrive at a final “Thank You” page, which explicitly states that their responses have been saved in our database.

• On this final “Thank You” page, trainees should click the “Exit” button to clear any responses saved on their devices.

• Please report any technical issues with the survey immediately to the evaluation team.