medigraphic.com
ISSN 2007-5057 (Print)
Investigación en Educación Médica

2019, Number 29

<< Back Next >>

Inv Ed Med 2019; 8 (29)

Development of validity evidences of a performance instrument on teaching competences: EDOE

Lobato VM, Alpuche HA, Trejo MJA, Martínez GA

Language: Spanish
References: 53
Page: 45-57
PDF size: 500.11 Kb.


ABSTRACT

Introduction: The evaluation of teaching competences is related to the improvement of the educational activity which is oriented towards personal and formative development. There are no high-fidelity evaluations in our specific setting. The aim was to obtain sources of evidence of validity for the Objective & Structured Teaching Examination (OSTE) to evaluate the teaching competencies.
Method: A quantitative study was carried out in three phases: 1) Planning phase: analysis of the competence model, validation by judges, validation of content by student instructors, and workshop with experts for the creation of the stations. 2) Pilot phase: to obtain the degree of concordance among evaluators, validation and internal structure of the stations (discriminant analysis, Cronbach’s alpha, means and standard deviation), 3) Final phase: Application to a non-random sample formed by 20 professors of the Department of Biomedical Informatics at the Faculty of Medicine of UNAM.
Results: An item discrimination analysis of eight stations was carried out. We observed that two stations have five items that discriminate .05, seven had acceptable factorial loads and explained 88.35% of the variance with a Cronbach’s alpha of .68. The average of teaching performance in the OSTE was 73.23 with a standard deviation of 8.4. The best-evaluated station was learning strategies (85 ± 16.98), and the lowest was discussion guidance (67.48 ± 15.62). Confirmatory evidence was obtained with similar examinations used in the Faculty of Medicine, the objectives set for the creation of this instrument were met, an OSTE pre-evaluation mean was detected, and the teachers got feedback on their performance for training purposes.
Discussion and Conclusions: The OSTE assessment is proposed as an instrument with sufficient evidence of validity that can be used in the evaluation of the degree of teaching competencies. We aim to create teaching evaluation instruments that follow good practices in the development of instruments based on the chain of evidence of validity.


REFERENCES

  1. Centra JA. Types of faculty development programs, J of Higher Ed. 1978;(49):151-62.

  2. Benor DE. Faculty development, teacher training and teacher accreditation in medical education: twenty years from now, Med Teach. 2000;22:503-12.

  3. Guskey TR. What makes professional development effective? Phi Delta Kappan. 2003;84:748-50.

  4. Steinert Y. Staff development. En Harden R, editor A Practical Guide for Medical Teacher. 2005;390-9.

  5. Harris DL, Krause KC, Parish DC, Smith MU. Academic competencies for medical faculty. Fam Med. 2007;39:343-50.

  6. Spencer JA. Learner centred approaches in medical education. BMJ. 1999;318:1280-3.

  7. Harden RM, Crosby JR. AMEE Education Guide No 20: The good teacher is more than a lecturer – the twelve roles of the teacher. Med Teach. 2000;22:334-47.

  8. Bland CJ, Schmitz C, Stritter F, Henry R, Aluise J. Successful faculty in academic medicine: Essential skills and how to acquire them. New York, NY: Springer Publishing; 1990.

  9. Martínez-González A, López-Bárcena J, Herrera Saint-leu P, et al. Modelo de competencias del profesor de medicina. Educ Med. 2008;11:157-67.

  10. Linn RL, Gronlund NE. Measurement and Assessment in Teaching. Prentice-Hall, 8th Ed. 2000.

  11. Haladyna TM, Downing SM, Rodriguez MC .A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment. Applied Measurement in Education. 2002; 15:309-34.

  12. Downing SM. Validity: on meaningful interpretation of assessment data. Med Educ. 2003;37:830-7.

  13. Downing SM, Yudkowsky R. Introduction to Assessment in the Health Professions. En: Downing SM, Yudkowsky (Eds) Assess in Health Prof Educ. New York, NY: Routledge; 2009. p. 217-40.

  14. Simpson D, Lawrence S, Krogull S. Using standardized ambulatory teaching situations for faculty development. Teach Learn Med.1992;4:58-61.

  15. Dunnington GL, DaRosa DA. A prospective randomized trial of a residents as teachers training program. Academic Medicine.1998;73,(6):696-700.

  16. Prislin MD, Fitzpatrick C, Giglio M, Lie D, Radecki S. Initial experience with a multi-station objective structured teaching skills evaluation. Acad Med.1998;73:1116-8.

  17. Zabar S, Hanley K, Stevens D, Kalet A, Schwartz M, Pearlman E, Brenner, Kachur E, Lipkin M. Measuring the Competence of Residents as Teachers. JGIM. 2004;19:530.

  18. Morrison EH, Boker JR, Hollingshead J, Prislin MD, Hitchcock MA, Litzelman DK. Reliability and validity of an objective structured teaching examination for generalist resident teachers. Acad Med. 2002;77:S29-S32.

  19. Trowbridge RL, Snydman LK, Skolfield J, Hafler J, Bing- You RG. A systematic review of the use and effectiveness of the Objective Structured Teaching Encounter. Med Teach. 2011;33:893-903.

  20. Sturpe DA et al. A Primer for Objective Structured Teaching Exercises.American J of Pharma Educ. 2014;78:1-6.

  21. Creswell J. Choosing a mixed methods desing. En: Designing and conducting. Mixed methods research. United States of America: SAGE. 2011:53-106.

  22. Kerlinger FN, y Lee HB. Investigación del comportamiento. México: McGraw-Hill; 2002.

  23. Thangaratinam S, Redman C. The Delphi technique. The Obstetrician & Gynaecologist. 2005;7:120-5.

  24. Von der Gracht H. The future of logistics. Glaber Edition. The Obstetrician & Gynaecologist. 2008;120-5.

  25. Varela-Ruiz et al. Descripción y usos del método Delphi en investigaciones del área de la salud. Inv Ed Med. 2012; 1(2):90-5.

  26. Hodges B, Regehr G, McNaughton N, Tiberius R, Hanson M. OSCE checklists do not capture increasing levels of expertise. Acad Med. 1999;74:1129-34.

  27. Trejo-Mejía JA, Blee-Sánchez GB, Peña-Balderas J. Elaboración de estaciones para el examen clínico objetivo estructurado (ECOE). RIEM. 2013;3:56-9.

  28. McAndrew M, et al. Creating an Objective Structured Teaching Examination to Evaluate a Dental Faculty Development Program. J of Dental Educ. 2012;76:461-8.

  29. Boillat M. Twelve tips for using the Objective Structured Teaching Exercise for faculty development. Med Teach. 2012;34:269-73.

  30. Schol S. A multiple-station test of the teaching skills of general practice preceptors in Flanders, Belgium. Acad Med. 2001;76:176-80.

  31. Schol S, Goedhuys J, Notten T, Betz W. Individualised training to improve teaching competence of general practitioner trainers: A randomised controlled trial. Med Educ. 2005;39:991-8.

  32. Maureen M, et al. Creating an Objective Structured Teaching Examination to Evaluate a Dental Faculty Development Program. JDE. 2012;76:461-8.

  33. Deborah A, et al. An objective structured teaching examination (OSTE) for faculty development. Curr in Phar Teach and Learn. 2013;5:616-9.

  34. Kachur EK. Multiple station exams and teaching exercises (MSEs) for teaching about teaching. In: Scherpbier AJJA, van der Vleuten CPM, Rethans JJ, van der Steeg AFW (Eds). Adv Med Educ. Dordrecht: Kluwer Academic Publishers. 1997. p. 226-7

  35. Ellen J, Giardino AP, Edinburgh K, Ende J. Simulated students: A new method for studying clinical precepting. Teach Learn Med. 1994;6:132-5.

  36. Gelula MH. Using standardized medical students to improve junior faculty teaching. Acad Med. 1998;73:611-2.

  37. Gelula MH, Yudkwsky R. Microteaching and standardized students support faculty development for clinical teaching. Acad Med. 2002;77:941.

  38. Lang F, Bennard B, Belanger A. Using standardized students to educate preceptors. Acad Med.1995;70:855-6.

  39. Pangaro LN, Worth-Dickstein H, Macmillan MK, Klass DJ, Shatzer JH. Performance of “standardized examinees” in a standardized-patient examination of clinical skills. Acad Med. 1997;72:1008-11.

  40. Morrison EH, Lewis EM, Gabbert CC, Boker JR, Kumar B, Harthill M. Evaluating a ‘service elective’ in clinical teaching for medical students. Med Teach. 2003;25:662-3.

  41. Morrison EH, Rucker L, Boker JR, Gabbert CC, Hubbell FA, Hitchcock MA ,Prislin MD. The effect of a 13-hour curriculum to improve residents’ teaching skills: A randomized trial. Ann Intern Med. 2004;141:257-63.

  42. Morrison EH, Rucker L, Boker JR, Hollingshead J, Hitchcock MA, Prislin MD, Hubbell FA. A pilot randomized, controlled trial of a longitudinal residents-as-teachers curriculum. Acad Med. 2003;78:722-9.

  43. Stone S, Mazor K, Devaney-O’Neil S, Starr S, Ferguson W, Wellman S, Jacobson E, Hatem DS, Quirk M. Development and implementation of an objective structured teaching exercise (OSTE) to evaluate improvement in feedback skills following a faculty development workshop. Teach Learn Med. 2003;15:7-13.

  44. Srinivasan M, Litzelman D, Seshadri R, Lane K, Zhou W, Bogdewic S, Gaffney M, Galvin M, Mitchell G, Treadwell P, et al. Developing an OSTE to address lapses in learners’ profesional behavior and an instrument to code educators’ responses. Acad Med. 2004;79:888-96.

  45. Quirk M, Mazor K, Haley HL, Wellman S, Keller D, Hatem D, Keller LA. Reliability and validity of checklists and global ratings by standardized students, trained raters, and faculty raters in an objective structured teaching environment. Teach Learn Med. 2005;17:202-9.

  46. Wamsley MA, Julian KA, Vener MH, Morrison EH. Using an objective structured teaching evaluation for faculty development. Med Educ. 2005;39:1160-1.

  47. Julian K, et al. The Impact of an Objective Structured Teaching Evaluation on Faculty Teaching Skills. 2012;24:3-7.

  48. Wilkerson L, Irby DM. Strategies for improving teaching practices: A comprehensive approach to faculty development. Acad Med. 1998;73:387-96.

  49. Wilkes MS, Hoffman JR, Usatine R, Baillie S. An innovative program to augment community preceptors’ practice and teaching skills. Acad Med. 2006;81:32-341.

  50. Gelula MH, Yudkowsky R. Using standardised students in faculty development workshops to improve clinical teaching skills. Med Educ. 2003;37:621-9.

  51. Padilla JL, Benítez I. Validity evidence based on response processes. Psico. 2014;26:136-44.

  52. Cerronea S, et al. Using Objective Structured Teaching Encounters (OSTEs) to prepare chief residents to be emotionally intelligent leaders. Med Educ online. 2017;22:1-7.

  53. Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: Theory and application. Am J Med. 2006;119:165-66.




2020     |     www.medigraphic.com