The effect of rubric on rater’s severity and bias in TVET laboratory practice assessment: analysis using many-facet rasch measurement

Ab Rahman, Azmanirah and Muhamad Hanafi, Nurfirdawati and Yusof, Yusmarwati and Mukthar, Marina Ibrahim and Awang, Halizah and Mohamed Yusof, Anizam (2020) The effect of rubric on rater’s severity and bias in TVET laboratory practice assessment: analysis using many-facet rasch measurement. Journal of Technical Education and Training, 12 (1). pp. 57-67. ISSN 2229-8932

Text
AJ 2020 (305).pdf
Restricted to Registered users only
Download (483kB) | Request a copy

Official URL: https://doi.org/10.30880/jtet.2020.12.01.006

Abstract

Performance assessments such as laboratory practice in Technical and Vocational Education and Training (TVET) are difficult to measure because they are subjective in their characteristics, which can cause bias. The use of a rubric scoring scale is suitable for measuring students’ practical competency as it could translate the qualitative criteria into quantitative forms on a grading scale. The purpose of this study is to analyse the effect of rater severity for the performance of component installation of an electronic circuit project using Many-Facet Rasch Measurement (MFRM). Nine raters have examined the competency performance of 68 students using a four-point rating scale rubric consisting of 16 items. This study was conducted at three vocational colleges. The results showed that all raters measured with different severities but very high consistency. The effect of rater severity did not exist at the group level but exhibited at the individual level. It can also be examined that all raters were biased on the item and also towards students, either by being severe or lenient. Thus, the rubric cannot eliminate the effect of bias during the assessment. However, it can help raters to be consistent so that they have the same understanding when translating the rubric. These findings have implications for the psychometric quality of the electronic circuit project performance assessment.

Item Type:	Article
Uncontrolled Keywords:	Performance assessment; rater severity; Many Facet Rasch Measurement
Subjects:	T Technology > T Technology (General) L Education > LC Special aspects of education > LC980-1099.5 Types of education > LC1041-1048 Vocational education (General)
Divisions:	Faculty of Technical and Vocational Education > Department of Vocational Education
Depositing User:	Miss Afiqah Faiqah Mohd Hafiz
Date Deposited:	30 Jan 2022 08:04
Last Modified:	30 Jan 2022 08:04
URI:	http://eprints.uthm.edu.my/id/eprint/6376

Actions (login required)

View Item