Characteristics of fundamental physics higher-order thinking skills test using Item Response Theory analysis
DOI:
https://doi.org/10.47750/pegegog.12.04.28Keywords:
FundPhysHOTS, Prospective Physics Teachers, Item Response TheoryAbstract
This study aims to determine the characteristics of the Fundamental Physics Higher-order Thinking Skill (FundPhysHOTS) test for prospective physics teachers using Item Response Theory (IRT) analysis. This study uses a quantitative approach. Two hundred fifty-four (254) prospective physics teachers at West Java and Banten, Indonesia. Data were collected through tests to respond to the FundPhysHOTS test. The FundPhysHOTS test instrument consists of 26 items and Two-Tier Multiple-Choice (TTMC) in form with a polytomous score of 4 categories (1,2,3 and 4). The results show that the data are unidimensional and local independence based on empirical data so that the IRT assumption is fulfilled. The IRT analysis used is the generalized partial credit model (GPCM). The results of the item parameter analysis show that all items have good discriminatory power parameters (0.394 < ai < 1.397) and are classified as good. The difficulty level analysis showed that almost all items had good step parameters ( b1 < b2 < b3), with mean of difficulty index (b mean) in the range of -0.332 to 0.144 and categorized as moderate. The ability (Ө) of prospective physics teachers is in the range of -3.976 < Ө < 1.646. Information function analysis shows that the FundPhysHOTS instrument is reliable in measuring ability in this range.
Downloads
References
Apino, E., & Retnawati, H. (2016). Creative Problem Solving to Improve Students' Higher Order Thinking Skills in Mathematics Instructions. Proceeding of 3Rd International Conference on Research, Implementation and Education of Mathematics and Science, May, 339–346.
Arlinwibowo, J., Retnawati, H., & Kartowagiran, B. (2021). Item Response Theory Utilization for Developing the Student Collaboration Ability Assessment Scale in STEM Classes. Journal homepage: http://iieta. org/journals/isi, 26(4), 409-415.
Bond, M., Zawacki-Richter, O., & Nichols, M. (2019). Revisiting five decades of educational technology research: A content and authorship analysis of the British Journal of Educational Technology. British Journal of Educational Technology, 50(1), 12–63. https://doi.org/10.1111/bjet.12730
Brookhart, S. M. (2010). How to Assess Higher-Order Thinking Skills in Your Classroom advances. In Journal of Education (Vol. 1, Issue 18). ASCD. www.ascd.org/memberbooks
Chan, S. W., Looi, C. K., & Sumintono, B. (2021). Assessing computational thinking abilities among Singapore secondary students: A rasch model measurement analysis. Journal of Computers in Education, 8(2), 213-236.
Conklin, W. (2012). Higher-Order Thinking Skills to Develop 21st Century Learners. In Shell Education. Shell Educational Publishing, Inc.
Cullinane, A., & Liston, M. (2011). Two-tier Multiple Choice Question: An Alternative Method of Formatif Assessment for First Year Undergraduate Biology Students. National Center for Excellence In Mathematics and Education Science Teaching and Learning (NCE-MSTL.
du Toit, M. (2003). IRT from SSI: Bilog-MG, Multilog, Parscale, Testfact. Scientific Software International.
Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Maheah.
Hair, J. F., Black, W. C., Babin, B. J., Anderson, R. E., & Tatham, R. L. (2009). Analise multivariada de dados. In Bookman. Bookman Editora.
Hambleton, R. K., Shavelson, R. J., Webb, N. M., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of Item Response Theory. 2.
Hariharasudan, A., & Kot, S. (2018). A scoping review on Digital English and Education 4.0 for Industry 4.0. Social Sciences, 7(11), 227. https://doi.org/10.3390/socsci7110227
Hasan, A., & Pardjono, P. (2019). The Correlation of Higher Order Thinking Skills and Work Readiness of Vocational High School Students.
Jurnal Pendidikan Teknologi Dan Kejuruan, 25(1), 52–61. https://doi.org/10.21831/jptk.v25i1.19118
Hestenes, D., Wells, M., & Swackhamer, G. (1992). Force concept inventory. The Physics Teacher, 30(3), 141–158. https://doi.org/10.1119/1.2343497
Iriyanti, I., & Darwis, M. (2021). Increasing Speaking Skills Through the Drama Method in Class IV Students of SD Unggulan Putra Kaili Permata Bangsa. The 2nd International Conference of Linguistics and Culture (ICLC-2, 128–132.
Istiyono, E, Mardapi, D., & Suparno, S. (2014). Pengembangan tes kemampuan berpikir tingkat tinggi fisika (pysthots) peserta didik SMA. Jurnal Penelitian Dan Evaluasi Pendidikan, 18(1), 1–12.
Istiyono, E., Dwandaru, W. S. B., Asysyifa, D. S., & Viana, R. V. (2020). Development of computer-based test in critical thinking skill assessment of physics. Journal of Physics: Conference Series, 1440(1), 12062. https://doi.org/10.1088/1742-6596/1440/1/012062
Istiyono, Edi, Dwandaru, W. S. B., Setiawan, R., & Megawati, I. (2020). Developing of computerized adaptive testing to measure physics higher order thinking skills of senior high school students and its feasibility of use. European Journal of Educational Research, 9(1), 91–101. https://doi.org/10.12973/eu-jer.9.1.91
Jailani, Sugiman, Retnawati, H., Bukhori, Apino, E., Djidu, H., & Arifin, Z. (2018). Desain Pembelajaran Matematika Untuk Melatihkan Higher Order Thinking Skills (p. 26).
Jansen, R. S., van Leeuwen, A., Janssen, J., Jak, S., & Kester, L. (2019). Self-regulated learning partially mediates the effect of self-regulated learning interventions on achievement in higher education: A meta-analysis. Educational Research Review, 28, 100292. https://doi.org/10.1016/j.edurev.2019.100292
José Sá, M., & Serpa, S. (2018). Transversal competences: Their importance and learning processes by higher education students. Education Sciences, 8(3), 126. https://doi.org/10.3390/educsci8030126
Kim, Y. C., & Jung, J. H. (2019). Conceptualizing shadow curriculum: definition, features and the changing landscapes of learning cultures. Journal of Curriculum Studies, 51(2), 141–161. https://doi.org/10.1080/00220272.2019.1568583
King, F. J., Goodson, L., & Rohani, F. (2010). Higher order thinking skills: Definition, Teaching Strategies, Assessment. http://goo.gl/su233T.
Krulik, S., & Rudnick, J. A. (1999). Innovative tasks to improve critical and creative thinking skills. In D. L. V Stiff & F. R. Curcio (Eds.), from Developing Mathematical reasoning in Grades K-12 (pp. 138–145). NCTM.
Kumar, H., Kumar, S., Dalabh, M., & Ahmad, J. (2016). Measurement and Evaluation in Education. Retrieved October, 10, 1–248. http://www.ipesp.ac.th/learning/websatiti/chapter9/unit9_1_4.html
Liu, O. L., Lee, H. S., & Linn, M. C. (2011). An investigation of explanation multiple-choice items in science assessment. Educational Assessment, 16(3), 164–184. https://doi.org/10.1080/10627197.2011.611702
Maphalala, M. C., & Adigun, O. T. (2020). Academics' experience of implementing e-learning in a south african higher education institution. International Journal of Higher Education, 10(1), 1–13. https://doi.org/10.5430/ijhe.v10n1p1
Maryani, I., Prasetyo, Z. K., Wilujeng, I., & Purwanti, S. (2022). Higher-order thinking test of science for college students using multidimensional item response theory analysis. Pegem Journal of Education and Instruction, 12(1), 292-300.
Mitani, H. (2021). Test Score Gaps in Higher Order Thinking Skills: Exploring Instructional Practices to Improve the Skills and Narrow the Gaps. AERA Open, 7, 23328584211016470. https://doi.org/10.1177/23328584211016470
Montoya, A. K., & Edwards, M. C. (2021). The Poor Fit of Model Fit for Selecting Number of Factors in Exploratory Factor Analysis for Scale Evaluation. Educational and Psychological Measurement, 81(3), 413–440. https://doi.org/10.1177/0013164420942899
Mukminin, A., Habibi, A., Prasojo, L. D., Idi, A., & Hamidah, A. (2019). Curriculum reform in indonesia: Moving from an exclusive to inclusive curriculum. Center for Educational Policy Studies Journal, 9(2), 53–72. https://doi.org/10.26529/cepsj.543
Nichols, P., & Sugrue, B. (1999). The Lack of Fidelity Between Cognitively Complex Constructs and Conventional Test Development Practice. Educational Measurement: Issues and Practice, 18(2), 18–29. https://doi.org/10.1111/j.1745-3992.1999.tb00011.x
Ockey, G. J. (2013). Item response theory. In The Routledge Handbook of Language Testing. Oxford University Press. https://doi.org/10.4324/9780203181287-36
Ostini, R., & Nering, M. L. (2006). Polytomous item response theory models (No. 144). Sage.
Pratama, G. S., & Retnawati, H. (2018). Urgency of Higher Order Thinking Skills (HOTS) Content Analysis in Mathematics Textbook. Journal of Physics: Conference Series, 1097(1), 12147. https://doi.org/10.1088/1742-6596/1097/1/012147
Presseisen, B. Z. (1988). Thinking skills: Meanings and models. In D. A. L. Costa (Ed.), Developing minds: A resource book for teaching thinking (pp. 43–48). ASCD.
Retnawati, H. (2011). Mengestimasi kemampuan peserta tes uraian matematika Dengan pendekatan teori respons butir Dengan penskoran politomus Dengan Generalized partial credit model [Estimating the ability of the participants in the mathematical description test Using the item response theory approach With polytomic scoring With the Generalized partial credit model]. In Prosiding Semnas Penelitian Pendidikan dan Penerapan MIPA. UNY (pp. 53–62).
Retnawati, H. (2014). Teori respons butir dan penerapannya [Item response theory and its application]. Nuha Medika.
Retnawati, H., Munadi, S., Arlinwibowo, J., Wulandari, N. F., & Sulistyaningsih, E. (2017). Teachers' difficulties in implementing thematic teaching and learning in elementary schools. New Educational Review, 48(2), 201–212. https://doi.org/10.15804/tner.2017.48.2.16
Saepuzaman, D., Retnawati, H., & Istiyono, E. (2021). Can innovative learning affect student HOTS achievements?: A meta-analysis study. Pegem Journal of Education and Instruction, 11(4), 290-305.
Si, C.-F., & Schumacker, R. E. (2004). Ability Estimation Under Different Item Parameterization and Scoring Models. International Journal of Testing, 4(2), 137–181. https://doi.org/10.1207/s15327574ijt0402_3
Si, C.-F., & Schumacker, R. E. (2004). Ability Estimation Under Different Item Parameterization and Scoring Models. International Journal of Testing, 4(2), 137–181. https://doi.org/10.1207/s15327574ijt0402_3
Songer, N. B., Kelcey, B., & Gotwals, A. W. (2009). How and when does complex reasoning occur? Empirically driven development of a learning progression focused on complex reasoning about biodiversity. Journal of Research in Science Teaching, 46(6), 610–631. https://doi.org/10.1002/tea.20313
Stone, C. A., & Zhang, B. (2003). Assessing Goodness of Fit of Item Response Theory Models: A Comparison of Traditional and Alternative Procedures. Journal of Educational Measurement, 40(4), 331–352. https://doi.org/10.1111/j.1745-3984.2003.tb01150.x
Suhendro, Sugandi, D., & Ruhimat, M. (2021). The Urgency of HOTS-Oriented Learning and Assessment Towards Quality of Education in Facing Indonesia Sustainable Development Goals (SDGs) 2030. Proceedings of the 5th Asian Education Symposium 2020 (AES 2020), 566, 237–250. https://doi.org/10.2991/assehr.k.210715.052
Sukatiman, S., Akhyar, M., Siswandari, & Roemintoyo. (2020). Enhancing higher-order thinking skills in vocational education through scaffolding-problem based learning. Open Engineering, 10(1), 612–619. https://doi.org/10.1515/eng-2020-0070
Teräs, M., Suoranta, J., Teräs, H., & Curcher, M. (2020). Post-Covid-19 Education and Education Technology 'Solutionism': a Seller's Market. Postdigital Science and Education, 2(3), 863–878. https://doi.org/10.1007/s42438-020-00164-x
Tondeur, J., Scherer, R., Baran, E., Siddiq, F., Valtonen, T., & Sointu, E. (2019). Teacher educators as gatekeepers: Preparing the next generation of teachers for technology integration in education. British Journal of Educational Technology, 50(3), 1189–1209. https://doi.org/10.1111/bjet.12748
Tyas, E. H., & Naibaho, L. (2021). Hots Learning Model Improves the Quality of Education. International Journal of Research -GRANTHAALAYAH, 9(1), 176–182. https://doi.org/10.29121/granthaalayah.v9.i1.2021.3100
Van der Linden, W. J. (Ed.). (2017). Handbook of Item Response Theory: Volume 2: Statistical Tools. CRC Press.
Wang, M., & Zheng, X. (2021). Using Game-Based Learning to Support Learning Science: A Study with Middle School Students. Asia-Pacific Education Researcher, 30(2), 167–176. https://doi.org/10.1007/s40299-020-00523-z
Warman, W., Lorensius, L., & Rohana, R. (2021). Curriculum of Management in Improving the Quality of Catholic School Education in Samarinda City, East Kalimantan, Indonesia. In Budapest International Research and Critics Institute (BIRCI-Journal): Humanities and Social Sciences (Vol. 4, Issue 3). East Kalimantan.
Wells, C. S., & Purwono, U. (2009). Assesing the fit of IRT models to item response data. Makalah Pelatihan Psikometri Kerjasama Pascasarjana UNY dengan USAID.
Widarjono, A. (2020). Analisis multivariat terapan dengan program SPSS, AMOS, dan SMARTPLS.[ Applied multivariate analysis with SPSS, AMOS, and SMART PLS programs]
Xiao, Y., Han, J., Koenig, K., Xiong, J., & Bao, L. (2018). Multilevel Rasch modeling of two-Tier multiple-choice test: A case study using Lawson's classroom test of scientific reasoning. Physical Review Physics Education Research, 14(2), 20104. https://doi.org/10.1103/PhysRevPhysEducRes.14.020104
Yeung, S. yin S. (2015). The conception of teaching higher-order thinking: perspectives of Chinese teachers in Hong Kong. Curriculum Journal, 26(4), 553–578. https://doi.org/10.1080/09585176.2015.1053818
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2022 Pegem Journal of Education and Instruction
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
NonCommercial — You may not use the material for commercial purposes.
No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.
