Direct Observation Form (DOF)

Abstract

The Direct Observation Form (DOF; Butterfield et al., 2023) was developed as a structured assessment index designed to accurately evaluate medical student performance while conducting psychiatric interviews during the core psychiatry clerkship rotation. The scale’s content and domains were established through a comprehensive review of clinical education standards, including guidelines set forth by the Association of Directors of Medical Student Education in Psychiatry (ADMSEP). Following pilot testing, the final 4-item DOF was administered to a sample of third-year medical students. Ratings were provided by supervising clinicians, including residents, fellows, and attending physicians. Subsequent evaluation demonstrated the instrument’s strong reliability and validity, confirming the DOF as a valuable tool for direct observation and clinical assessment in psychiatric training.

Keywords

Test Responsiveness, Direct Observation, Medical Student Performance, Psychiatric Interviews, Psychiatry Clerkship, Educational Measures, Medical Residency, Psychiatric Training, Student Characteristics, Entrustable Professional Activity.

Authors

Austin Butterfield, Ashley Curry, Joel Yager, Joseph Sakai

[quads id=5]

Purpose

The primary objective of the Direct Observation Form (DOF) is to provide a focused, accurate, and comprehensive method for assessing specific clinical skills of medical students within the psychiatry clerkship. The instrument targets the essential competencies required for successfully conducting psychiatric interviews.

By standardizing the observation process, the DOF ensures that evaluations of performance related to communication, rapport, and technical interviewing skills are objective and consistent across different clinical supervisors and educational settings.

Construct

The DOF measures the construct of observable clinical interviewing competence, specifically tailored for the psychiatric setting. This construct is operationalized across four key dimensions of performance, ranging from procedural adherence to interpersonal skill.

These dimensions include the ability to manage the structure of the interview (opening and closing), technical proficiency in eliciting information (interview techniques), interpersonal skills (presence and attunement), and the ability to manage the therapeutic relationship (alliance building and professional boundaries).

Validity

The validation study demonstrated multiple lines of evidence supporting the DOF’s utility in medical education assessment.

Test Responsiveness: The instrument showed significant test responsiveness, evidenced by a substantial increase in mean DOF scores from Time 1 to Time 2 (p < 0.001). This confirms the DOF’s sensitivity to capturing genuine improvement in student performance over the duration of the clerkship.

Predictive Validity: Scores from the DOF collected at Time 2 showed a strong positive and significant association with the students’ final end-of-course clinical grades (p < 0.001). This relationship remained statistically significant even after controlling for potential confounding factors, such as rater rank, case complexity, and academic timing. Furthermore, the DOF scores correlated strongly with an Entrustable Professional Activity (EPA) assessment of the interview (Time 2 r = 0.79, p < 0.001), suggesting high convergent validity with measures of professional readiness.

Discriminant Validity: Notably, the DOF scores demonstrated a lack of correlation with standardized written shelf exam scores (Time 2 r = 0.11, p = 0.21). This suggests that the DOF successfully assesses a distinct set of performance-based clinical skills that are separate from the knowledge base measured by traditional examinations.

Reliability

Internal Consistency: The Direct Observation Form demonstrated strong internal consistency, confirming that its four items reliably measure the common underlying construct of clinical interviewing competence. The calculated Cronbach’s alpha values for the scale ranged from 0.88 to 0.89, which is indicative of excellent reliability.

Factor Analysis

No data is Available.

Instrument

Test Type: Original

Format: The Direct Observation Form is a 4-item scale, with each item scored on a 5-point scale (where 5 represents the highest possible performance). The form also mandates an overall score framed as an Entrustable Professional Activity (EPA) assessment question and includes optional checkboxes for indicating high case complexity or difficulty relative to the learner’s training level.

Language Available: English

Population Group: Human (Male and Female)

Age Group: Adulthood (18 years and older)

Population Details: The instrument is specifically designed for the evaluation of third-year medical students during their psychiatry clerkship. Raters providing the assessment include supervising residents, fellows, attending physicians, or other team members located primarily in the United States.

Test Methodology: Test Validity, Predictive Validity, Test Reliability, Internal Consistency.

Keywords

Test Responsiveness, Direct Observation, Medical Student Performance, Psychiatric Interviews, Psychiatry Clerkship, Educational Measures, Medical Residency, Psychiatric Training, Student Characteristics.

[quads id=5]

Authors

Author ORCID Identifier:

• Butterfield, Austin: http://orcid.org/0000-0003-0231-7266

• Sakai, Joseph: http://orcid.org/0000-0003-1600-6100

Affiliation Email addresses:

• Butterfield, Austin: University of Colorado School of Medicine

• Curry, Ashley: University of Colorado School of Medicine

• Yager, Joel: University of Colorado School of Medicine

• Sakai, Joseph: University of Colorado School of Medicine

• Email (Corresponding Author): [email protected]

Correspondence Address:

• Butterfield, Austin: [email protected]

Permissions & Fee and Test Year

Permissions: Contact Corresponding Author.

Commercial: No

Fee: No

Test Year: 2023

References

Butterfield, A., Curry, A., Yager, J., & Sakai, J. (2023). A direct observation form for evaluation of the psychiatric interview: Pilot testing during the psychiatry clerkship. Academic Psychiatry, 47(4), 368–373. https://doi.org/10.1007/s40596-023-01762-0

[quads id=5]

Items of the Direct Observation Form (DOF)

The Direct Observation Form (DOF) consists of four core items, each rated on a 1-5 scale (with 5 being the highest score). These items are designed to assess key aspects of a medical student’s performance during a psychiatric interview:

1. Opening Portion and Closing of the Interview: This item evaluates the student’s ability to effectively initiate and conclude the interview, encompassing aspects like establishing rapport, setting the agenda, and summarizing findings.

2. Interview Techniques: This item focuses on the student’s proficiency in employing various interviewing techniques, such as active listening, empathetic responses, appropriate questioning, and managing the flow of the conversation.

3. Presence/Attunement to Patient: This item assesses the student’s ability to be fully present and engaged with the patient, demonstrating empathy, understanding non-verbal cues, and maintaining a respectful and therapeutic demeanor.

4. Alliance Building and Boundaries: This item evaluates the student’s skill in establishing a positive therapeutic alliance with the patient while maintaining appropriate professional boundaries throughout the interview.

In addition to these four scored items, the form includes optional boxes for the rater to indicate if the interview was particularly challenging for the learner’s level of training or if the case had a high level of complexity. Finally, the rater is instructed to provide an overall score for the interview, framed as an Entrustable Professional Activity (EPA) assessment question, which reflects the student’s readiness to perform this activity with decreasing supervision.