medigraphic.com
Colegio de Medicina Interna de México.

2015, Number 3

<< Back Next >>

Med Int Mex 2015; 31 (3)

Modification of psychometric indexes of departmental examinations designed to assess residents of Internal Medicine, Graduate School of Medicine, UNAM, according to five or four answer choices

Borrego-Mora PP, Santana-Borrego MA

Language: Spanish
References: 45
Page: 259-273
PDF size: 493.14 Kb.


ABSTRACT

Background: The number of options has been widely discussed among the supporters of the multiple-choice items. It is possible to choose the reliability, validity, or both as a criterion for establishing a number of possible answers.
Objective: To analyze if reducing for five to four options of answer modifies psychometric properties of departmental exams designed to assess medical specialties residents and, thus, the quality of assessment instrument.
Material and method: A comparative quasi-experimental study was done applying a test with 300 multiple choice questions in two versions: 5 and 4 possible answers to two groups of 117 internal medicine residents, each one.
Results: We obtained a correlation coefficient of 0.925 (p ‹ 0.05), the coefficient of reliability with 95% CI for five answers was 0.913 and for four options was 0.905. The index of difficulty for five options was 63 and for four was 65, 54% of the items had a better index of difficulty in the version of five options. The point biserial correlation for five and four options was 0.16 and 0.18, respectively.
Conclusions: In the version with four options, 54% of the items answered a greater number of residents, including low achievers; they could hit the random (20% probability), allowing bordering residents to obtain a passing grade. By comparing the two versions, with four options 44% were reactive with a lower discrimination index, which indicates that they were less able to discriminate between residents of high and low score. By observing these changes we believe that five options are better than four to assess medical specialties.


REFERENCES

  1. Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika 1951;16:297-234.

  2. Bruno JE, Dirkzwager A. Determining the optimal number of alternatives to a multiple-choice test item: An information Theoretic Perspective. Educational and Psychological Measurement 1995;55:959-966.

  3. Haladyna TM, Downing, SM. How many options is enough for a multiple-choice test item? Educational and Psychological Measurement 1993;5:999-1010.

  4. Haladyna TM, Downing, SM. A taxonomy of multiple-choice item-writing rules. Applied Measurement in Education 1989;2:37-50.

  5. Ebel R. Expected reliability as a function of choices per item. Educational and Psychological Measurement 1969;29:565-570.

  6. Ruch GM, Charles JW. A comparison of five types of objective tests in elementary psychology. J Applied Psychol 1928;12:398-404.

  7. Muñiz J. Las teorías de los test: Teoría clásica y teoría de respuesta a los ítems. Papeles del Psicólogo 2010;31:57-66.

  8. Costin F. The optimal number of alternatives in multiple-choice achievement tests: Some empirical evidence for a mathematical proof. Educational and Psychological Measurement 1970;30:353-358.

  9. Straton RG, Catts RM. A comparison of two, three and four choice item test given a fixed total number of choices. Educational and Psychological Measurement 1980;40:357-365.

  10. Tversky A. On the optimal number of alternatives at a choice point. J Mathematical Psychology 1964;1:386-391.

  11. Costin F. Three-choice versus four-choice items: implications for reliability and validity of objective achievement test. Educational and Psychological Measurament 1972;32:1035-1038.

  12. Garner WR. Rating scales, discriminability and information transmission. Psychological Review 1960;67:343-352.

  13. Grier JB. The number of alternatives for optimum test reliability. J Educational Measurement 1975;12:109-113.

  14. Grier JB. The optimal number of alternatives at a choice point with travel time considered. J Mathematical Psychology 1976;12:31-97.

  15. Rogers WT, Harley D. An empirical comparison of three- and four-choice items and test: Susceptibility to testwiseness and internal consistency reliability. Educational and Psychological Measurement 1999;59:234-247.

  16. Olea J, Ponsoda V, Revuelta J, Hontangas P, Abad F. Analysis of the optimum number alternatives from the item response theory. Psicothema 2001;13:152-158.

  17. Lord FM. Optimal number of choices per item: A comparison of four approaches. J Educational Measurement 1977;14:33-48.

  18. Lissitz RW, Green SB. Effect of the number of scale points on reliability: A Monte Carlo approach. J Applied Psychology 1975;60:10-13.

  19. Komorita SS, Graham WK. Number of scale points and the reliability of scales. Educational and Psychological Measurement 1965;25:987-995.

  20. Masters JR. The relationship between number of response categories and reliability of Likert type questionnaires. J Educational Measurement 1974;11:49-53.

  21. Matell MS, Jacoby J. Is there an optimal number of alternatives for Likert scale items? Study I: Reliability and validity. Educational and Psychological Measurement 1971;31:657-674.

  22. Peabody D. Two components in bipolar scales: Direction and extremeness. Psychological Review 1962;69:65-73.

  23. Sancerni MD, Meliá JL y González V. Formato de respuesta, fiabilidad y validez, en la medición del conflicto de rol. Psicológica 1990;11:167-175.

  24. Churchill GA Jr., Peter JP. Research design effects on the reliability of rating scales: A meta-analysis. J Marketing Research 1984;21:360-375.

  25. Ferrando PJ. Saturaciones factoriales e índices de discriminación en la teoría clásica del test y en la teoría de respuesta al ítems. Anuario de Psicología 1994;2:55-65.

  26. Preston CC, Colman AM. Optimal number of response categories in rating scales: reliability, validity, discriminating power, and respondent preferences. Acta Psychologica 2000;104:1-15.

  27. Ramsay JO. The effect of number of categories in rating scales on precision of estimation of scale values. Psycho-metrika 1973;38:513-533.

  28. Symonds PM. On the loss of reliability in ratings due to coarseness of the scale. J Experimental Psychology 1924;December:456-461.

  29. Champney H, Marshall H. Optimal refinement of the rating scale. J Applied Psychology 1980;23:323-331.

  30. Cox EP. The optimal number of response alternatives for a scale: A review. J Marketing Research 1980;17:407-422.

  31. Aiken LR. Number of response categories and statistics on a teacher rating scale. Educational and Psychological Measurement 1983;43:397-401.

  32. Bandalos DL, Enders CK. The effects of non-normality and number of response categories on reliability. Applied Measurement in Education 1996;9:151-160.

  33. Cicchetti DV, Showalter D, Tyrer PJ. The effect of number of rating scale categories on levels of inter-rater reliability: A Monte-Carlo investigation. Applied Psychological Measurement 1985;9:31-36.

  34. Jenkins GD, Taber TD. A Monte Carlo study of factors affecting three indices of composite scale reliability. J Applied Psychology 1977;62:392-398.

  35. Wen LJ. Impact of the number of response categories and anchor labels on coefficient alpha and test retest reliability. Educational and Psychological Measurement 2004;64:956-972.

  36. Comrey AL, Montag I. Comparison of factor analytic results with two choice and seven choice personality item formats. Applied Psychological Measurement 1982;6:285-289.

  37. Olsson U. Maximum likelihood estimation of the polychoric correlation coefficient. Psychometrika 1979;44:443-460.

  38. Chang L. A psychometric evaluation of four-point and six-point Liker type scales in relation to reliability and validity. Applied Psychological Measurement 1994;18:205-215.

  39. McCallum DM, Keith BR, Wiebe DJ. Comparison of response formats for multidimensional healt locus of control scales: Six levels versus two levels. J Personality Assessment 1988;52:732-736.

  40. Spearman C. General intelligence, objectively determined and measured. Am J Psychol 1904;15:202-293.

  41. Spearman C. The proof and measurement of association between two things. Am J Psychol 1904;15:72-101.

  42. Spearman C. Demonstration of formulae for true measurement of correlation. Am J Psychol 1907;18:161-169.

  43. Spearman C. Correlation calculated with faulty data. Br J Psychol 1910;3:271-295.

  44. Spearman C. Correlations of sums and differences. Br J Psychol 1913;5:417-426.

  45. Allen MJ, Yen WM. Introduction to measurement theory. Monterey, CA: Brooks/Cole, 1979, 210-212.




2020     |     www.medigraphic.com