medigraphic.com
ISSN 2007-5057 (Print)
Investigación en Educación Médica

2017, Number 22

<< Back Next >>

Inv Ed Med 2017; 6 (22)

Analysis of an objective structured clinical examination in dentistry using generalisability theory

Espinosa-Vázquez O, Martínez-González A, Sánchez-Mendiola M, Leenen I

Language: Spanish
References: 45
Page: 109-118
PDF size: 136.59 Kb.


ABSTRACT

Introduction: Various studies have examined Objective Structured Clinical Examinations (OSCEs) in Dentistry in order to accumulate validity evidence for their use as an assessment tool of clinical competence in students. In this article, a newly designed OSCE in Dentistry (OSCE-D) is introduced and discussion is presented on the results of an analysis from the perspective of generalisability theory using data obtained from an application of the examination.
Method: An observational and cross-sectional study was conducted in the Faculty of Dentistry at UNAM. One hundred and twenty pre-graduate students participated in an OSCE that consisted of 18 stations, with a duration of 6 min each, in the context of a fourth-grade Paediatric Dentistry course. An analysis based on generalisability theory, with raters and stations being considered as facets, identified the main sources of variability in the data.
Results: The overall mean (and standard deviation) of the OSCE score, across participants and stations, was 44% (7%), with the station means varying between 23% and 63%. The generalisability study showed that the facet of the raters explained a significant portion (13%) of the variance in the station results, which was more than the clinical competence of the participants (6%). The decision study produced a generalisability index of 0.63 and a dependability index of 0.55.
Conclusions: In view of the rather low reliability indices from the decision study, it is important to make a further analysis of the OSCE-D so as to minimise the effect of sources that introduce construct-irrelevant variance into the results. In particular, an adjustment of the stations may be required, as well as a better standardising in the use of evaluation criteria by the raters.


REFERENCES

  1. Davenport ES, Davis JE, Cushing AM, Holsgrove GJ. An innovationin the assessment of future dentists. Br Dent J.1998;184:192---5.

  2. Waas V, van der Vleuten C, Shatzer J, Jones R. Assessment ofclinical competence. Lancet. 2001;357:945---9.

  3. Epstein RM. Assessment in medical education. N Engl J Med.2007;356:387---96.

  4. Martínez GA, Trejo MJA, Fortoul GT, Flores HF, Morales LS,Sánchez MM. Evaluación diagnóstica de conocimientos y competenciasen estudiantes de medicina al término del segundoa˜no de la carrera: el reto de construir el avión mientras vuela.Gac Med Mex. 2014;150:35---48.

  5. Arnold, Walmsley RC. The use of the OSCE in postgraduate education.Eur J Dent Edu. 2008;12:16---30.

  6. Schoonheim-Klein M, Walmsley AD, Habets L, van der VeldenU, Manogue M. An implementation strategy for introducing anOSCE into a dental school. Eur J Dent Educ. 2005;9:143---9.

  7. Durante M, Lozano S, Martínez G, Morales L, Sánchez M.Evaluación de competencias en Ciencias de la Salud. MédicaPanamericana. 2010:20---6.

  8. Kramer GA, Albino JEN, Andrieu SC, Hendricson WD, Henson L,Horn BD, et al. Dental student assessment toolbox. J Dent Educ.2009;73:12---35.

  9. Harden R, Stevenson M, Downie WW, Wilson GM. Clinical competencein using objective structured examination. Br Med J.1975;1:447---51.

  10. Harden RM, Gleeson FA. ASME medical educational booklet no8: Assessment of medical competence using an objective structuredclinical examination (OSCE). J Med Educ. 1979;13:41---54.

  11. Näpänkangas R, Harila V, Lahti S. Experiences in addingmultiple-choice questions to an objective structural clinicalexamination (OSCE) in undergraduate dental education. Eur JDent Educ. 2012;16:146---50.

  12. Trejo A, Blee G, Pe˜na J. Elaboración de estaciones para elexamen clínico objetivo estructurado (ECOE). Inv Ed Med.2014;3:56---9.

  13. Bhowate R, Panchbhai A, Vagha S, Tankhiwale S. Introductionof objective structured clinical examination (OSCE) in dentaleducation in India in the subject of oral medicine and radiology.J Educ Ethics Dent. 2014;4:23---7.

  14. Trejo MJA, Martínez GA, Méndez RI, Morales LS, Ruiz PL, SánchezMM. Evaluación de la competencia clínica con el examen clínicoobjetivo estructurado en el internado médico de la UniversidadNacional Autónoma de México. Gac Med Mex. 2014;150:8---17.

  15. Manogue M, Brown G. Developing and implementing an OSCE indentistry. Eur J Dent Educ. 1998;2:51---7.

  16. Mossey PA, Newton JP, Stirrups DR. Scope of the OSCE inthe assessment of clinical skills in dentistry. Br Dent J.2001;190:323---6.

  17. Mendel N, Fuks J, Levy T, Fernández M, de Preliasco VF, AmanteaA. Examen clínico objetivo y estructurado (ECOE): unapropuesta innovadora en la evaluación de la Odontopediatría.Revista de la Facultad de Odontología (UBA). 2005;30:31---6.

  18. Hodder RV, Rivington RN, Calcutt LE, Hart IR. The effectivenessof immediate feedback during the Objective Structured ClinicalExamination. Med Educ. 1989;23:184---8.

  19. Harden R, Lilley P, Patrício M. The Definitive Guide to the OSCE.The Objective Structured Clinical Examination as a PerformanceAssessment. Elsevier; 2016. p. 149---58, 176-179.

  20. Downing SM. Validity: On the meaningful interpretation ofassessment data. Med Educ. 2003;37:830---7.

  21. Sánchez MM. La calidad del proceso de evaluación para la certificacióndel médico especialista. México: Comité NormativoNacional de Consejos de Especialidades Médicas; 2007 [consultado28 Abr 2015]. Disponible en: http://www.conacem.org.mx/assets/boletin calidad.pdf

  22. Wetzel AP. Factor analysis methods and validity evidence: Areview of instrument development across the medical educationcontinuum. Acad Med. 2011;87:1060---9.

  23. Delgado AR, Prieto G. Fiabilidad y validez. Papeles del Psicólogo20103167-74 [consultado 25 Mar 2016]. Disponible en:http://www.redalyc.org/articulo.oa?id=77812441007

  24. Webb NM, Shavelson RJ, Haertel EH. Reliability Coefficients andGeneralizability Theory. En: Rao CR, Sinharay S, editores. Handbookof Statistics 26. USA: Elsevier; 2007. p. 81---120.

  25. Brown G, Manogue M, Martin M. The validity and reliability ofan OSCE in dentistry. Eur J Dent Educ. 1999;3:117---25.

  26. Gerrow JD, Murphy HJ, Boyd MA, Scott DA. Concurrent validityof written and OSCE components of the Canadian dentalcertification examinations. J Dent Educ. 2003;67:896---901.

  27. Näpänkangas R, Karaharju-Suvanto T, Pyörälä E, Harila V, OllilaP, Lähdesmäki R, et al. Can the results of the OSCE predict theresults of clinical assessment in dental education? Eur J DentEduc. 2016;20:3---8.

  28. Schuwirth L, van der Vleuten C. General overview of the theoriesused in assessment: AMEE Guide No. 57. Med Teacher.2011;33:783---97.

  29. Crossley J, Davies H, Humphris G, Jolly B. Generalisability: A keyto unlock professional assessment. Med Edu. 2002;36:9972---8.

  30. Schoonheim-Klein M, Mujitens A, Habets L, Manogue M, vander Vleuten C, Hoogstraten J, et al. On the reliability of adental OSCE, using SEM: Effect of different days. Eur J DentEduc. 2008;12:131---7.

  31. Eberhard L, Hassel A, Bäumer A, Becker F, Beck-Mubotter J,BömickeW, et al. Analysis of quality and feasiability of an objectivestructured clinical examination (OSCE) in preclinical dentaleducation. Eur J Dent Educ. 2011;15:172---8.

  32. Bergus GR, Kreiter CD. The reliability of summative judgementsbased on objective structured clinical examination cases distributedacross the clinical year. Med Edu. 2007;41:661---6.

  33. Trejo MA, Sánchez MM, Méndez RI, Martínez GA. Reliabilityanalysis of the objective structured clinical examinationusing generalizability theory. Med Educ Online. 2016;21:31650,http://dx.doi.org/10.3402/meo.v21.31650.

  34. Taguchi N, Ogawa T. OSCEs in Japanese postgraduate clinicaltraining Hiroshima experience 2000-2009. Eur J Dent Educ.2010;14:203---9.

  35. Larsen T, Jeppe-Jensen D. The introduction and perception ofan OSCE with an element of self-and peer-assessment. Eur JDent Educ. 2008;12:2---7.

  36. Schoonheim-Klein M, Habets L, Artman A, van der Vleuten C,Hoogstraten J, van der Velden U. Implementing an ObjectiveStructured Clinical Examination (OSCE) in dental education:Effects on students’ learning strategies. Eur J Dent Educ.2006;10:226---35.

  37. Schoonheim-Klein M, Mujitens A, Habets L, Manogue M, van derVleuten C. Who will pass the dental OSCE? Comparison of theAngoff and the borderline regression standard setting methods.Eur J Dent Educ. 2009;13:162---71.

  38. Brennan RL. Generalizability Theory. En: Instructional Topics inEducational Measurement. Iowa: National Council of MedicalEducation; 1992. p. 27---34.

  39. Downing SM. Reliability: On the reproducibility of assessmentdata. Med Educ. 2004;38:1006---12.

  40. Downing SM, Yudkowsky R. Reliability. En: Downing SM, YudkowskyR, editores. Assessment in Health Professions Education.New York, London: Routdlege. Taylor and Francis Group; 2009.p. 57---74.

  41. Park SE, Kim A, Kristiansen J, Karimbux NY. The influence ofexaminer type on dental students’ OSCE scores. J Dent Edu.2015;79:89---94.

  42. Patrício M, Juliao M, Fareleira F, Vaz A. Is the OSCE a feasibletool to asses competencies in undergraduate medical education.Med Teacher. 2013;35:503---14.

  43. Brannick MT, Tugba EH, Prewett M. A systematic review of thereliability of objective structured clinical examination scores.Med Educ. 2011;45:1181---9.

  44. Youngman MB. Analysing social and research education data.En: Brown G, Manogue M, Martin M, editors. The validity andreliability of an OSCE in dentistry, 3. Eur J Dent Educ; 1999. p.117---25.

  45. Espinosa V.O., Martínez G.A., Sánchez M.M. Evidencias de validezpara el Examen Clínico Objetivo Estructurado. Cartel ypresentación oral. XX Conferencia Panamericana de EducaciónMédica. V Congreso Internacional de Educación Médica. IV CongresoInternacional de Simulación. Cancún, México, 2016.




2020     |     www.medigraphic.com