| JOURNAL HOME | CME HOME | THIS MONTH | PAST ISSUES | ETOC | COLLECTIONS |
| AUTHORS | REVIEWERS | EDITORIAL BOARD | FEEDBACK | RSS | HELP |
| ||||||||||||||
| ||||||||||||||
|
|
|
|
||||||||||||
|
||||||||||||||
From the Nucleus for Teaching and Research in Medical Education, Department of Anesthesiology, Hospital Governador Celso Ramos, Florianópolis, SC, Brazil.
Address correspondence and reprint requests to Getúlio R de Oliveira Filho, PhD, Hospital Governador Celso Ramos, Rua Luiz Delfino 111/902-88015360 Florianópolis, SC, Brazil. Address e-mail to grof{at}th.com.br.
Abstract
BACKGROUND AND OBJECTIVES: We aimed 1) to develop a valid and reliable instrument for faculty supervision evaluation by anesthesia residents and 2) to disclose the sources of error in residents’ ratings.
METHODS: A qualitative study involving residents and faculty identified constructs of supervisory ability, which were entered as items in a measurement instrument used by 19 residents to evaluate 39 instructors during a 6-mo period. The instrument was psychometrically tested under classical item and generalizability theories. A decision study, using the parameters of the generalizability (G) study, estimated the number of resident ratings needed to produce dependable measures of a single faculty.
RESULTS: Nine dimensions emerged from the qualitative study: planning perianesthesia care, providing feedback ("the instructor provides me timely, informal, non-threatening comments on my performance and shows me ways to improve"); being available ("the instructor is promptly available to help me solve problems with patients and procedures"); giving opportunities/fostering resident autonomy; stimulating patient-based learning; demonstrating professionalism; being present during the critical events; demonstrating interpersonal skills; being concerned about safety. Residents provided 970 evaluations. The instrument exhibited internal consistency (Cronbach’s 
= 0.93), content and face validities, and a single-factor structure. Generalizability and dependability coefficients were 0.93. Between-instructors differences accounted for 56% of score variance. Resident-instructor interactions accounted for 44% of score variance, indicating that scores were influenced by each resident’s unique perceptions of instructors (halo effect). According to the results of the decision study, dependability of measures within the 75% to 95% range could be expected with 3 to 33 residents rating each faculty member, respectively.
CONCLUSIONS: The nine-item instrument produced valid and reliable measures of faculty supervision. However, a significant amount of halo effect biased such measures. G-studies may help identify the type and magnitude of rater biases affecting resident-generated faculty supervision evaluations, and can be useful for interpreting their results, especially if personnel decisions (e.g., tenure, promotion) rely on such measures.
|
