medigraphic.com
ISSN 2007-5057 (Print)
Investigación en Educación Médica

2024, Number 50

<< Back Next >>

Inv Ed Med 2024; 13 (50)

Validity evidence in the admission process to a medical school in Mexico

Carrillo-Avalos BA, Leenen I, Trejo-Mejía JA, Sánchez-Mendiola M

Language: Spanish
References: 54
Page: 37-55
PDF size: 609.30 Kb.


ABSTRACT

Introduction: Medical school admission processes need to demonstrate the validity of their scores’ interpretations, since they are high-stakes assessments.
Objective: To apply an integrative model using Messick’s and Kane’s frameworks to obtain validity evidence for a medical school admission process.
Method: An analytical observational study was performed at the Autonomous University of San Luis Potosí Faculty of Medicine in Mexico, with the 2013 and 2014 cohorts. The admission process included a psychometric test (PT), a general knowledge test (KT), and EXANI-II. The model was applied in two steps: 1) uses and interpretations argument, and 2) validity argument, which has three stages: instrumental validity, interpretation and decision verification, and action usefulness. Hypotheses were established at each stage using Kane’s inferences, which were supported using Messick’s sources of evidence.
Results: The results of 1,373 applicants from 2013 and 1,554 from 2014 were analyzed, as well as the academic performance of the 145 students admitted in each cohort. A general intelligence factor (g) was identified as influencing the results of admission process. KT explains 15% of the variance in the first-year grades (2013), PT and EXANI-II have no impact on first-year’s academic performance (2013 and 2014). There was a correlation among some subcomponents that measure a similar construct. There was a difference in Biology results (KT) between men and women. Students’ degree of satisfaction with the admission process is adequate.
Discussion and conclusions: The sources of validity evidence provide relevant information to support and improve the admission process, more data and an interdisciplinary team to retrieve more sources of quality evidence are needed. It is recommended to use welldefined frameworks in these processes and plan data collection and retrieval using these models.


REFERENCES

  1. Instituto Nacional para la Evaluación de la Educación. CriteriosTécnicos Para El Desarrollo, Uso y Mantenimiento de Instrumentosde Evaluación. 2017. http://tinyurl.com/2cnrfs7w

  2. American Educational Research Association, American PsychologicalAssociation, National Council on Measurementin Education. Standards for Educational and PsychologicalTesting. 6th ed. American Educational Research Association,American Psychological Association & National Councilon Measurement in Education; 2014.

  3. Ferrara S. Our Field Needs a Framework to Guide Developmentof Validity Research Agendas and Identification of Validity ResearchQuestions and Threats to Validity. Measurement (MahwahN J). 2007;5(2-3):156-164. doi:10.1080/15366360701487500

  4. Violato C, Gauer JL, Violato EM, Patel D. A study of thevalidity of the new MCAT exam. Academic Medicine. 2020;95(3):396-400. doi:10.1097/ACM.0000000000003064

  5. Paton LW, McManus IC, Cheung KYF, Smith DT, Tiffin PA.Can achievement at medical admission tests predict futureperformance in postgraduate clinical assessments? A UKbasednational cohort study. BMJ Open. 2022;12(2):1-12.doi:10.1136/bmjopen-2021-056129

  6. Simpson PL, Scicluna HA, Jones PD, et al. Predictive validityof a new integrated selection process for medical school admission.BMC Med Educ. 2014;14(1):1-10. doi:10.1186/1472-6920-14-86

  7. Bala L, Pedder S, Sam AH, Brown C. Assessing the predictivevalidity of the UCAT—A systematic review and narrativesynthesis. Med Teach. 2022;44(4):401-409. doi:10.1080/0142159X.2021.1998401

  8. Barajas-Ochoa A, Ramos-Remus C, Castillo-Ortiz JD, et al.Flaws in the design of the Examen Nacional para Aspirantesa Residencias Médicas produce inequity. Salud Pública Mex.2019;61(2):125-135. doi:10.21149/9790

  9. Crawford C, Black P, Melby V, Fitzpatrick B. An explorationof the predictive validity of selection criteria on progressoutcomes for pre-registration nursing programmes—Asystematic review. J Clin Nurs. 2021;30(17-18):2489-2513.doi:10.1111/jocn.15730

  10. Cunningham C, Patterson F, Cleland J. A literature reviewof the predictive validity of European dental school selectionmethods. European Journal of Dental Education. 2019;23(2):73-87. doi:10.1111/eje.12405

  11. Hafferty FW, O’Brien BC, Tilburt JC. Beyond High-StakesTesting: Learner Trust, Educational Commodification, and theLoss of Medical School Professionalism. Academic Medicine.2020;95(6):833-837. doi:10.1097/ACM.0000000000003193

  12. Panda N, Bahdila D, Abdullah A, Ghosh AJ, Lee SY, FeldmanWB. Association between USMLE Step 1 Scores andIn-Training Examination Performance: A Meta-Analysis.Academic Medicine. 2021;96(12):1742-1754. doi:10.1097/ACM.0000000000004227

  13. Salehi PP, Azizzadeh B, Lee YH. Pass/Fail Scoring of USMLEStep 1 and the Need for Residency Selection Reform. Otolaryngology- Head and Neck Surgery (United States). 2021;164(1):9-10. doi:10.1177/0194599820951166

  14. Villegas Vizcaíno R. ¿Es el EXANI II un predictor o un factorde exclusión social? Educación y Ciencia. 2017;6(48):43-52.Accessed July 11, 2023. https://bit.ly/42s5bgS

  15. Bárquez Antillán IL, Vivian Mascareño MF. La validez predictivade la prueba de aptitud académica (PAA) respecto aldesempeño académico de los estudiantes de la Universidad LaSalle Noroeste , A.C. In: Consejo Mexicano de InvestigaciónEducativa AC, ed. X Congreso Nacional de Investigación Educativa.Consejo Mexicano de Investigación Educativa, A.C.;2009. Accessed July 12, 2023. https://bit.ly/490R58T

  16. Morales Ibarra R, Barrera Baca A, Mandujano Garnett E. Validezpredictiva y concurrente del EXANI-II, en la UniversidadAutónoma del Estado de México. In: Consejo Mexicano deInvestigación Educativa AC, ed. X Congreso Nacional de InvestigaciónEducativa. Consejo Mexicano de Investigación Educativa,A.C.; 2009. Accessed July 12, 2023. https://bit.ly/3SLdXDG

  17. Martínez Villarreal RT. Valor Predictivo Del Examen Nacionalde Ingreso En La Licenciatura de Medicina En La UniversidadAutónoma de Nuevo León, México. Facultad de Medicinade la Universidad Complutense de Madrid; 2014. Accessed July12, 2023. https://hdl.handle.net/20.500.14352/38125

  18. Cronbach LJ, Meehl PE. Construct validity in psychologicaltests. Psychol Bull. 1955;52(4):281-302. doi:10.1037/h0040957

  19. Messick S. Validity of test interpretation and use. In: Encyclopediaof Educational Research. 6th ed. Macmillan; 1990:1487-1495. https://doi.org/10.1002/j.2333-8504.1990.tb01343.x

  20. Messick S. Validity. In: Linn RL, ed. Educational Measurement.3rd ed. Macmillan; 1989:13-104. doi:10.1002/j.2330-8516.1987.tb00244.x

  21. Cook DA, Brydges R, Ginsburg S, Hatala R. A contemporary approachto validity arguments: A practical guide to Kane’s framework.Med Educ. 2015;49(6):560-575. doi:10.1111/medu.12678

  22. Kane M. Validating score interpretations and uses: MessickLecture, Language Testing Research Colloquium,Cambridge, April 2010. Language Testing. 2012;29(1):3-17.doi:10.1177/0265532211417210

  23. Kane MT. An argument-based approach to validity. PsycholBull. 1992;112(3):527-535. doi:10.1037/0033-2909.112.3.527

  24. Carrillo Avalos BA, Leenen I, Sánchez Mendiola M. Un acercamientoa la unificación de los marcos de referencia de validezde Messick y de Kane. In: XVI Congreso Nacional deInvestigación Educativa. Consejo Mexicano de InvestigaciónEducativa A.C.; 2021.

  25. Russell M. Clarifying the Terminology of Validity and the InvestigativeStages of Validation. Educational Measurement:Issues and Practice. 2022;41(2):25-35. doi:10.1111/emip.12453

  26. UASLP. Guía Temática del Examen de Conocimientos Facultadde Medicina. Published online 2014.

  27. Carrillo-Avalos BA, Sánchez Mendiola M, Leenen I. Amenazasa la validez en evaluación: implicaciones en educaciónmédica. Investigación en Educación Médica. 2020;(34):100-107. doi:10.22201/facmed.20075057e.2020.34.221

  28. UASLP. Plan Institucional de Desarrollo 2013-2023.; 2014.

  29. UASLP. Instructivo Para Aspirantes de Nuevo Ingreso 2015-2016.; 2015.

  30. CENEVAL. Guía Del Examen Nacional de Ingreso a La EducaciónSuperior (EXANI-II) 2014. 19th ed. Centro Nacionalde Evaluación para la Educación Superior, A. C.; 2013.

  31. UASLP. Guía Temática del Examen de Conocimientos Facultadde Medicina. 2017.

  32. CENEVAL. Guía Del Examen Nacional de Ingreso a La EducaciónSuperior (EXANI-II); 2013.

  33. Miranda López F, Pérez Güemes EE, Villamil Serrano E,Márquez Gutiérrez Y, González Chávez GA, Hernández GómezJE. Encuesta de Satisfacción de Los Procesos de Evaluaciónde Ingreso y Promoción En Educación Básica y MediaSuperior 2017; 2018. https://bit.ly/485bwAe

  34. Cook DA, Hatala R. Validation of educational assessments: aprimer for simulation and beyond. Advances in Simulation.2016;1(1):1-12. doi:10.1186/s41077-016-0033-y

  35. Dancey C, Reidy J. Statistics without Maths for Psychology.7th ed. Pearson; 2017. www.pearson-books.com

  36. Rios J, Wells C. Evidencia de validez basada en la estructurainterna. Psicothema. 2014;26(1):108-116. doi:10.7334/psicothema2013.260

  37. Luschin-Ebengreuth M, Dimai HP, Ithaler D, Neges HM,Reibnegger G. Medical University admission test: a confirmatoryfactor analysis of the results. Wien Klin Wochenschr.2016;128(9-10):376-383. doi:10.1007/s00508-015-0911-7

  38. Habersack M, Dimai HP, Ithaler D, Reibnegger G. Time: anunderestimated variable in minimizing the gender gap inmedical college admission scores. Wien Klin Wochenschr.2015;127(7-8):241-249. doi:10.1007/s00508-014-0649-7

  39. Ganjoo R, Schwartz L, Boss M, McHarg M, Dobrydneva Y.Predictors of success on the MCAT among post-baccalaureatepre-medicine students. Heliyon. 2020;6(4). doi:10.1016/j.heliyon.2020.e03778

  40. Haladyna TM, Downing SM. Construct-Irrelevant Variancein High-Stakes Testing. Educational Measurement: Issuesand Practice. 2004;23(1):17-27. https://bit.ly/49nXOtd

  41. Leiner JEM, Scherndl T, Ortner TM. How do men andwomen perceive a high-stakes test situation? Front Psychol.2018;9(DEC). doi:10.3389/fpsyg.2018.02216

  42. Arenas A, Calsamiglia C. Gender Differences in High-StakesPerformance and College Admission Policies. 2022. http://hdl.handle.net/10419/265771

  43. Violato C, Donnon T. Does the medical college admissiontest predict clinical reasoning skills? A longitudinal studyemploying the Medical Council of Canada clinical reasoningexamination. Acad Med. 2005;80(10):S14-S16. doi:80/10_suppl/S14 [pii]

  44. Davies DJ, Sam AH, Murphy KG, Khan SA, Choe R, Cleland J.BMAT’s predictive validity for medical school performance:A retrospective cohort study. Med Educ. 2022;56(9):936-948.doi:10.1111/medu.14819

  45. Lambe P, Waters C, Bristow D. The UK clinical aptitude Test:Is it a fair test for selecting medical students? Med Teach.2012;34(8). doi:10.3109/0142159X.2012.687482

  46. Patterson F, Knight A, Dowell J, Nicholson S, Cousans F,Cleland J. How effective are selection methods in medicaleducation? A systematic review. Med Educ. 2016;50(1):36-60.doi:10.1111/medu.12817

  47. Association of American Medical Colleges, Howard HughesMedical Institute. Scientific Foundations for Future PhysiciansReport of the AAMC-HHMI Committee.; 2009. AccessedJuly 9, 2023. https://bit.ly/3UBXItI

  48. Patterson F, Roberts C, Hanson MD, et al. 2018 Ottawa consensusstatement: Selection and recruitment to the healthcareprofessions. Med Teach. Published online 2018:1-11.doi: 10.1080/0142159X.2018.1498589

  49. UKCAT Consortium. UCAT Official Guide 2023.; 2023. AccessedJuly 9, 2023. https://bit.ly/3w41bal

  50. Hernández-Mata JM, Hernández-Castro R, Nieto-CaraveoA, Hernández Sierra JF. Factores de riesgo para la deserciónde estudiantes en la Facultad de Medicina de la UniversidadAutónoma de San Luis Potosí (UASLP), México. Gac MedMex. 2005;141(5):445-447. Accessed July 11, 2023. https://bit.ly/3w6hHqB

  51. Royal KD, Guskey TR. Editorial: On the appropriatenessof norm-and criterion-referenced assessments in medicaleducation. Ear Nose Throat J. 2015;94(7):150-152. doi:10.1177/014556131509400701

  52. Tavakol M, Dennick R. The foundations of measurement andassessment in medical education. Med Teach. 2017;39(10):1010-1015. doi: 10.1080/0142159X.2017.1359521

  53. Lok B, McNaught C, Young K. Criterion-referenced andnorm-referenced assessments: compatibility and complementarity.Assess Eval High Educ. 2016;41(3):450-465. doi:10.1080/02602938.2015.1022136

  54. Sireci SG, Greiff S. Editorial: On the importance of educationaltests. European Journal of Psychological Assessment.2019;35(3):297-300. doi: 10.1027/1015-5759/a000549




2020     |     www.medigraphic.com