Psychometrics of an Elo-based large-scale online learning system
The Elo rating system (ERS), an intuitive and computationally efficient algorithm, offers a means to effectively update estimates of item difficulties and learner abilities as they evolve. This method proves to be highly advantageous in online learning environments. Computerized adaptive practice (C...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2025-06-01
|
| Series: | Computers and Education: Artificial Intelligence |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2666920X25000165 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1825206899074662400 |
|---|---|
| author | Hanke Vermeiren Joost Kruis Maria Bolsinova Han L.J. van der Maas Abe D. Hofman |
| author_facet | Hanke Vermeiren Joost Kruis Maria Bolsinova Han L.J. van der Maas Abe D. Hofman |
| author_sort | Hanke Vermeiren |
| collection | DOAJ |
| description | The Elo rating system (ERS), an intuitive and computationally efficient algorithm, offers a means to effectively update estimates of item difficulties and learner abilities as they evolve. This method proves to be highly advantageous in online learning environments. Computerized adaptive practice (CAP) endeavors to present learners with items that are well-suited to their individual ability levels, with the ultimate goal of enhancing motivation and optimizing learning outcomes. The objective of this paper is to outline common challenges that arise in an Elo-based CAP system and to present the psychometric enhancements implemented in the Prowise Learn environments to address these concerns. More specifically, we focus on three main aspects; 1) the development of a new scoring rule balancing response time and accuracy, 2) a way to fix the item scale to deal with item drift, and 3) an improved adaptive K-factor algorithm to speed up convergence in estimation. Using data from the Prowise Learn environment, analyses were done to illustrate the effect of the enhancements. Results show that these enhancements result in more dynamic tracking of the ratings, solve the issue of item drift, and capture the speed-accuracy trade-off more accurately. |
| format | Article |
| id | doaj-art-b71909f3432b439fa271904cb7b2e5bb |
| institution | Kabale University |
| issn | 2666-920X |
| language | English |
| publishDate | 2025-06-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Computers and Education: Artificial Intelligence |
| spelling | doaj-art-b71909f3432b439fa271904cb7b2e5bb2025-02-07T04:48:28ZengElsevierComputers and Education: Artificial Intelligence2666-920X2025-06-018100376Psychometrics of an Elo-based large-scale online learning systemHanke Vermeiren0Joost Kruis1Maria Bolsinova2Han L.J. van der Maas3Abe D. Hofman4Faculty of Psychology and Educational Sciences, and Imec research group Itec, KU Leuven, Kortrijk, Belgium; Corresponding author.Cito Institute for Educational Measurement, Arnhem, the NetherlandsMethodology and Statistics, Tilburg University, Tilburg, the NetherlandsPsychological Methods, University of Amsterdam, Amsterdam, the NetherlandsPsychological Methods, University of Amsterdam, Amsterdam, the Netherlands; Prowise, Amsterdam, the NetherlandsThe Elo rating system (ERS), an intuitive and computationally efficient algorithm, offers a means to effectively update estimates of item difficulties and learner abilities as they evolve. This method proves to be highly advantageous in online learning environments. Computerized adaptive practice (CAP) endeavors to present learners with items that are well-suited to their individual ability levels, with the ultimate goal of enhancing motivation and optimizing learning outcomes. The objective of this paper is to outline common challenges that arise in an Elo-based CAP system and to present the psychometric enhancements implemented in the Prowise Learn environments to address these concerns. More specifically, we focus on three main aspects; 1) the development of a new scoring rule balancing response time and accuracy, 2) a way to fix the item scale to deal with item drift, and 3) an improved adaptive K-factor algorithm to speed up convergence in estimation. Using data from the Prowise Learn environment, analyses were done to illustrate the effect of the enhancements. Results show that these enhancements result in more dynamic tracking of the ratings, solve the issue of item drift, and capture the speed-accuracy trade-off more accurately.http://www.sciencedirect.com/science/article/pii/S2666920X25000165Architectures for educational technology systemComputer adaptive practiceElo rating system |
| spellingShingle | Hanke Vermeiren Joost Kruis Maria Bolsinova Han L.J. van der Maas Abe D. Hofman Psychometrics of an Elo-based large-scale online learning system Computers and Education: Artificial Intelligence Architectures for educational technology system Computer adaptive practice Elo rating system |
| title | Psychometrics of an Elo-based large-scale online learning system |
| title_full | Psychometrics of an Elo-based large-scale online learning system |
| title_fullStr | Psychometrics of an Elo-based large-scale online learning system |
| title_full_unstemmed | Psychometrics of an Elo-based large-scale online learning system |
| title_short | Psychometrics of an Elo-based large-scale online learning system |
| title_sort | psychometrics of an elo based large scale online learning system |
| topic | Architectures for educational technology system Computer adaptive practice Elo rating system |
| url | http://www.sciencedirect.com/science/article/pii/S2666920X25000165 |
| work_keys_str_mv | AT hankevermeiren psychometricsofanelobasedlargescaleonlinelearningsystem AT joostkruis psychometricsofanelobasedlargescaleonlinelearningsystem AT mariabolsinova psychometricsofanelobasedlargescaleonlinelearningsystem AT hanljvandermaas psychometricsofanelobasedlargescaleonlinelearningsystem AT abedhofman psychometricsofanelobasedlargescaleonlinelearningsystem |