AUTOMATION OF TEST ITEMS QUALITY ANALYSIS AND IT STUDENTS TEST RESULTS ASSESSMENT

Keywords: statistic methods, IRT models, Rasch model, test quality, test items, knowledge testing, validity, reliability, discriminant ability

Abstract

The subject of this article is statistical models and methods for test items quality assessment, as well as processes and information support technologies for test results and test quality evaluating. The goal is to improve the quality of test items and the accuracy of the evaluation of test results in the context of constant changes in the content of study programs and the content of test tasks, and to reduce the time for test results processing through automation, and developed algorithms and an information system usage. The following tasks was solved: analysis of the test preparation and testing process; an analytical review of statistical methods for the test items quality assessment and the test items quality indicators; generalized process of the test preparation and conduct of testing development; and automation of IT students test results assessment. To solve these problems, the following methods was used: statistical analysis methods, the Rasch IRT (Item Response Theory) model, and software application development technologies were used. The following results are obtained. The process of the test preparation and testing was analyzed; its modification is proposed taking into account the peculiarities of IT students testing. The test task quality indicators, models and methods for it calculation were selected, in particular, the Rasch IRT model. The interpretation of the test item quality indicators values, and the calculation of the coefficients of the anomalies in the test item quality, what are the basis for the generation of an advisory solution to improve the test quality, are given. A prototype of an information system for support of processes of test items quality indicators evaluating, test results assessment and decision making about test modification has been developed. The system was tested during the module control of third-year students results evaluating. Conclusions: automation and information support of the test items quality indicators calculating and analyzing, and test results assessment will improve the accuracy of the assessment and reduce the time for test results evaluating.

Downloads

Download data is not yet available.

Author Biographies

Leonid Smidovych, National Aerospace University "Kharkiv Aviation Institute"
PhD (Engineering Sciences), Associate Professor, Associate Professor of the Department of Computer Science and Information Technologies
Olesia Artiukh, Kharkiv School of Radio Engineering
Foreign Language Teacher
Nataliia Kosenko, O. M. Beketov Kharkiv National University of Urban Economy
PhD (Engineering Sciences), Associate Professor at the Department of Project Management in Urban Economy and Construction

References

Avanesov, V. S. (1994), Scientific problems of test knowledge measurement [Nauchnye problemy testovogo kontrolya znanii], Moscow, ITsPKPS, 86 p.

Avanesov, V. S. (2006), Form of the tests [Forma testovykh zadanii], Moscow, Tsentr testirovaniya Publ., 156 p.

Omarov, М., Muradova, V. (2018), "Infological support of the information technology for managemant of distance education", Innovative Technologies and Scientific Solutions for Industries, No. 4 (6), P. 146–153. DOI: https://doi.org/10.30837/2522-9818.2018.6.146

Kovalʹčuk, Ju. O. (2012), Educational Measurement Theory [Teorija osvitnix vymirjuvanʹ], Nizhyn, Publisher PP Lysenko M. M., 200 p.

Lisova, T. V. (2012), Models and methods of the modern test theory [Modeli ta metody suchasnoyi teoriyi testiv], Nizhyn, Publisher PP Lysenko M. M. 112 p.

An, X. and Yung, Y. (2014), "Item Response Theory: What it is and how you can use the IRT procedure to apply it", Paper SAS364-2014 – Cary, NC, USA : SAS Institute Inc., 14 p., available at : http://support.sas.com/resources/papers/proceeding (last accessed 11.02.2020).

Razak, N. A., Razak, A. Z. K. and Lei, M.T. (2012), "Examining Quality of Mathemtics Test Items Using Rasch Model: Preminarily Analysis", Procedia - Social and Behavioral Sciences, Vol. 69, P. 2205–2214. DOI: https://doi.org/10.1016/j.sbspro.2012.12.187

Maiorov, A. N. (2002), Theory and practice of tests creating for the education system [Teoriya i praktika sozdaniya testov dlya sistemy obrazovaniya], Moscow, Intellekt-Tsentr Publ., 267 p.

Malieieva, Ju., Kosenko, V., Malyeyeva, O., Svetlichnyj, D. (2019), "Creation of collaborative development in the system of distance learning", Innovative Technologies and Scientific Solutions for Industries, No. 2 (8), P. 62–71. DOI: https://doi.org/10.30837/2522-9818.2019.8.062

Croker, L., Algina, J. (2010), Introduction to Classical and Modern Test Theory. [Vvedenie v klassicheskuyu i sovremennuyu teoriyu testov], Moscow, Logos Publ., 668 p.

Rasch, G. (1980), Probabilistic Models for Some Intelligence and Attainment Tests. Copenhagen, Danish Institute of Educational Research. (Expanded edition), 324 p.

Kim, V. S. (2005), "Test results analysis during Rasch measurement" ["Analiz rezul'tatov testirova-niya v protsesse Rasch measurement"], Pedagogicheskie izmereniya – Pedagogical Measurements, Vol. 4, P. 39–45.

Rusch, T., Lowry, P. B., Mair, P., Treiblmaier, H. (2017), "Breaking free from the limitations of classical test theory: Developing and measuring information systems scales using item response theory", Information & Management, Vol. 54, Issue 2, P. 189–203. DOI: https://doi.org/10.1016/j.im.2016.06.005

Wilson, M. (2005), Constructing Measures: An Item Response Modeling Approach, NJ : Lawrence Erlbaum Associates, 228 p. DOI: https://doi.org/10.4324/9781410611697

Mikheev, O. V. (2004), "The mathematical model of pedagogical measurements" ["Matematicheskie modeli pedagogicheskikh izmerenii"], Pedagogicheskie izmereniya – Pedagogical Measurements, Vol. 2, P. 75–88.

Dykhovichnyi, A. A., Dudko, A. F. (2013), "Automated system for computer testing in higher mathematics results analysis" ["Avtomatizirovannaya sistema analiza rezul'tatov komp'yuternogo testirovaniya po vysshei matematike"], Naukovі pratsі DonNTU. Serіya: "Pedagogіka, psikhologіya і sotsіologіya", Vol. 2 (14), P. 103–110.

Suparman, U. (2016), "Analyzing the quality of english test items of daily, mid semester and final school examinations in bandar lampung: (assessment and evaluation in language teaching)", International Conference On Educational Research & Evaluation, Yogyakarta, 14 p. URL: http://repository.lppm.unila.ac.id/id/eprint/3121 (last accessed 11.02.2020).

Quaigrain, K., Arhin, A. K. (2017), "Using reliability and item analysis to evaluate a teacher-developed test in educational measurement and evaluation", Cogent Education, Vol. 4 (1). P. 1–11. DOI: https://doi.org/10.1080/2331186X.2017.1301013.


Abstract views: 21
PDF Downloads: 23
Published
2020-03-23
How to Cite
Smidovych, L., Artiukh, O. and Kosenko, N. (2020) “AUTOMATION OF TEST ITEMS QUALITY ANALYSIS AND IT STUDENTS TEST RESULTS ASSESSMENT”, INNOVATIVE TECHNOLOGIES AND SCIENTIFIC SOLUTIONS FOR INDUSTRIES, (1 (11), pp. 19-27. doi: 10.30837/2522-9818.2020.11.019.