A Study for developing a success test: examination of validity and classification accuracy by ROC analysis

Abstract

In this research the aim is to develop a success test including grammar, reading-comprehension, and vocabulary and sentence knowledge in order to determine the Turkish lesson learning skills of students at secondary school fourth grade. in determination of cognitive property stages the comprehension processes of PIRLS were used. While the research group was determined a suitable sampling method among the non-random sampling methods was used. the research group consists of 111 students having education in secondary school fourth grade in Etimesgut province of Ankara. Open ended questions were developed as well as multiple choice questions and rubrics were developed for open ended questions. in success test validation study, ROC analysis was performed. Material and test statistics were calculated on the data obtained at the end of the study. the KR20 reliability coefficient was calculated as 0.76 for Turkish success test. With material analysis the average material difficulty and distinctiveness indexes were calculated respectively as 0.49 and 0.34. These results show that valid and reliable results are obtained by the developed test. At the end of ROC analysis, the area under the operation characteristic curve (AUC) was calculated as 0.88 and was found to be significant. (p<0.05) This value shows that Turkish success test can make a separation in determining the student success which means successful and unsuccessful students can be classified with an accuracy ratio of 88% (good level). This result also shows that ROC analysis can be used as a validity detection tool in test development studies. (C) 2015 the Authors. Published by Elsevier Ltd.