Characteristics of fundamental physics higher-order thinking skills test using Item Response Theory analysis




FundPhysHOTS, Prospective Physics Teachers, Item Response Theory


This study aims to determine the characteristics of the Fundamental Physics Higher-order Thinking Skill (FundPhysHOTS) test for prospective physics teachers using Item Response Theory (IRT) analysis. This study uses a quantitative approach. Two hundred fifty-four  (254) prospective physics teachers at West Java and Banten, Indonesia. Data were collected through tests to respond to the FundPhysHOTS test. The FundPhysHOTS test instrument consists of  26 items and Two-Tier Multiple-Choice (TTMC) in form with a polytomous score of 4 categories (1,2,3 and 4). The results show that the data are unidimensional and local independence based on empirical data so that the IRT assumption is fulfilled. The IRT analysis used is the generalized partial credit model (GPCM). The results of the item parameter analysis show that all items have good discriminatory power parameters (0.394 < ai < 1.397) and are classified as good. The difficulty level analysis showed that almost all items had good step parameters ( b1 < b2 < b3), with mean of difficulty index  (b mean)   in the range of -0.332 to 0.144 and categorized as moderate. The ability (Ө) of prospective physics teachers is in the range of -3.976 < Ө < 1.646. Information function analysis shows that the FundPhysHOTS instrument is reliable in measuring ability in this range. 


Download data is not yet available.

Author Biographies

Duden Saepuzaman, Universitas Pendidikan Indonesia (Lecturer) ; Universitas Negeri Yogyakarta(Graduate School )

Physics Education Department

Universitas Pendidikan Indonesia 


Edi Istiyono, Universitas Negeri Yogyakarta

Educational Research and Evaluation

Haryanto, Universitas Negeri Yogyakarta

Educational Research and Evaluation


Apino, E., & Retnawati, H. (2016). Creative Problem Solving to Improve Students' Higher Order Thinking Skills in Mathematics Instructions. Proceeding of 3Rd International Conference on Research, Implementation and Education of Mathematics and Science, May, 339–346.

Arlinwibowo, J., Retnawati, H., & Kartowagiran, B. (2021). Item Response Theory Utilization for Developing the Student Collaboration Ability Assessment Scale in STEM Classes. Journal homepage: http://iieta. org/journals/isi, 26(4), 409-415.

Bond, M., Zawacki-Richter, O., & Nichols, M. (2019). Revisiting five decades of educational technology research: A content and authorship analysis of the British Journal of Educational Technology. British Journal of Educational Technology, 50(1), 12–63.

Brookhart, S. M. (2010). How to Assess Higher-Order Thinking Skills in Your Classroom advances. In Journal of Education (Vol. 1, Issue 18). ASCD.

Chan, S. W., Looi, C. K., & Sumintono, B. (2021). Assessing computational thinking abilities among Singapore secondary students: A rasch model measurement analysis. Journal of Computers in Education, 8(2), 213-236.

Conklin, W. (2012). Higher-Order Thinking Skills to Develop 21st Century Learners. In Shell Education. Shell Educational Publishing, Inc.

Cullinane, A., & Liston, M. (2011). Two-tier Multiple Choice Question: An Alternative Method of Formatif Assessment for First Year Undergraduate Biology Students. National Center for Excellence In Mathematics and Education Science Teaching and Learning (NCE-MSTL.

du Toit, M. (2003). IRT from SSI: Bilog-MG, Multilog, Parscale, Testfact. Scientific Software International.

Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Maheah.

Hair, J. F., Black, W. C., Babin, B. J., Anderson, R. E., & Tatham, R. L. (2009). Analise multivariada de dados. In Bookman. Bookman Editora.

Hambleton, R. K., Shavelson, R. J., Webb, N. M., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of Item Response Theory. 2.

Hariharasudan, A., & Kot, S. (2018). A scoping review on Digital English and Education 4.0 for Industry 4.0. Social Sciences, 7(11), 227.

Hasan, A., & Pardjono, P. (2019). The Correlation of Higher Order Thinking Skills and Work Readiness of Vocational High School Students.

Jurnal Pendidikan Teknologi Dan Kejuruan, 25(1), 52–61.

Hestenes, D., Wells, M., & Swackhamer, G. (1992). Force concept inventory. The Physics Teacher, 30(3), 141–158.

Iriyanti, I., & Darwis, M. (2021). Increasing Speaking Skills Through the Drama Method in Class IV Students of SD Unggulan Putra Kaili Permata Bangsa. The 2nd International Conference of Linguistics and Culture (ICLC-2, 128–132.

Istiyono, E, Mardapi, D., & Suparno, S. (2014). Pengembangan tes kemampuan berpikir tingkat tinggi fisika (pysthots) peserta didik SMA. Jurnal Penelitian Dan Evaluasi Pendidikan, 18(1), 1–12.

Istiyono, E., Dwandaru, W. S. B., Asysyifa, D. S., & Viana, R. V. (2020). Development of computer-based test in critical thinking skill assessment of physics. Journal of Physics: Conference Series, 1440(1), 12062.

Istiyono, Edi, Dwandaru, W. S. B., Setiawan, R., & Megawati, I. (2020). Developing of computerized adaptive testing to measure physics higher order thinking skills of senior high school students and its feasibility of use. European Journal of Educational Research, 9(1), 91–101.

Jailani, Sugiman, Retnawati, H., Bukhori, Apino, E., Djidu, H., & Arifin, Z. (2018). Desain Pembelajaran Matematika Untuk Melatihkan Higher Order Thinking Skills (p. 26).

Jansen, R. S., van Leeuwen, A., Janssen, J., Jak, S., & Kester, L. (2019). Self-regulated learning partially mediates the effect of self-regulated learning interventions on achievement in higher education: A meta-analysis. Educational Research Review, 28, 100292.

José Sá, M., & Serpa, S. (2018). Transversal competences: Their importance and learning processes by higher education students. Education Sciences, 8(3), 126.

Kim, Y. C., & Jung, J. H. (2019). Conceptualizing shadow curriculum: definition, features and the changing landscapes of learning cultures. Journal of Curriculum Studies, 51(2), 141–161.

King, F. J., Goodson, L., & Rohani, F. (2010). Higher order thinking skills: Definition, Teaching Strategies, Assessment.

Krulik, S., & Rudnick, J. A. (1999). Innovative tasks to improve critical and creative thinking skills. In D. L. V Stiff & F. R. Curcio (Eds.), from Developing Mathematical reasoning in Grades K-12 (pp. 138–145). NCTM.

Kumar, H., Kumar, S., Dalabh, M., & Ahmad, J. (2016). Measurement and Evaluation in Education. Retrieved October, 10, 1–248.

Liu, O. L., Lee, H. S., & Linn, M. C. (2011). An investigation of explanation multiple-choice items in science assessment. Educational Assessment, 16(3), 164–184.

Maphalala, M. C., & Adigun, O. T. (2020). Academics' experience of implementing e-learning in a south african higher education institution. International Journal of Higher Education, 10(1), 1–13.

Maryani, I., Prasetyo, Z. K., Wilujeng, I., & Purwanti, S. (2022). Higher-order thinking test of science for college students using multidimensional item response theory analysis. Pegem Journal of Education and Instruction, 12(1), 292-300.

Mitani, H. (2021). Test Score Gaps in Higher Order Thinking Skills: Exploring Instructional Practices to Improve the Skills and Narrow the Gaps. AERA Open, 7, 23328584211016470.

Montoya, A. K., & Edwards, M. C. (2021). The Poor Fit of Model Fit for Selecting Number of Factors in Exploratory Factor Analysis for Scale Evaluation. Educational and Psychological Measurement, 81(3), 413–440.

Mukminin, A., Habibi, A., Prasojo, L. D., Idi, A., & Hamidah, A. (2019). Curriculum reform in indonesia: Moving from an exclusive to inclusive curriculum. Center for Educational Policy Studies Journal, 9(2), 53–72.

Nichols, P., & Sugrue, B. (1999). The Lack of Fidelity Between Cognitively Complex Constructs and Conventional Test Development Practice. Educational Measurement: Issues and Practice, 18(2), 18–29.

Ockey, G. J. (2013). Item response theory. In The Routledge Handbook of Language Testing. Oxford University Press.

Ostini, R., & Nering, M. L. (2006). Polytomous item response theory models (No. 144). Sage.

Pratama, G. S., & Retnawati, H. (2018). Urgency of Higher Order Thinking Skills (HOTS) Content Analysis in Mathematics Textbook. Journal of Physics: Conference Series, 1097(1), 12147.

Presseisen, B. Z. (1988). Thinking skills: Meanings and models. In D. A. L. Costa (Ed.), Developing minds: A resource book for teaching thinking (pp. 43–48). ASCD.

Retnawati, H. (2011). Mengestimasi kemampuan peserta tes uraian matematika Dengan pendekatan teori respons butir Dengan penskoran politomus Dengan Generalized partial credit model [Estimating the ability of the participants in the mathematical description test Using the item response theory approach With polytomic scoring With the Generalized partial credit model]. In Prosiding Semnas Penelitian Pendidikan dan Penerapan MIPA. UNY (pp. 53–62).

Retnawati, H. (2014). Teori respons butir dan penerapannya [Item response theory and its application]. Nuha Medika.

Retnawati, H., Munadi, S., Arlinwibowo, J., Wulandari, N. F., & Sulistyaningsih, E. (2017). Teachers' difficulties in implementing thematic teaching and learning in elementary schools. New Educational Review, 48(2), 201–212.

Saepuzaman, D., Retnawati, H., & Istiyono, E. (2021). Can innovative learning affect student HOTS achievements?: A meta-analysis study. Pegem Journal of Education and Instruction, 11(4), 290-305.

Si, C.-F., & Schumacker, R. E. (2004). Ability Estimation Under Different Item Parameterization and Scoring Models. International Journal of Testing, 4(2), 137–181.

Si, C.-F., & Schumacker, R. E. (2004). Ability Estimation Under Different Item Parameterization and Scoring Models. International Journal of Testing, 4(2), 137–181.

Songer, N. B., Kelcey, B., & Gotwals, A. W. (2009). How and when does complex reasoning occur? Empirically driven development of a learning progression focused on complex reasoning about biodiversity. Journal of Research in Science Teaching, 46(6), 610–631.

Stone, C. A., & Zhang, B. (2003). Assessing Goodness of Fit of Item Response Theory Models: A Comparison of Traditional and Alternative Procedures. Journal of Educational Measurement, 40(4), 331–352.

Suhendro, Sugandi, D., & Ruhimat, M. (2021). The Urgency of HOTS-Oriented Learning and Assessment Towards Quality of Education in Facing Indonesia Sustainable Development Goals (SDGs) 2030. Proceedings of the 5th Asian Education Symposium 2020 (AES 2020), 566, 237–250.

Sukatiman, S., Akhyar, M., Siswandari, & Roemintoyo. (2020). Enhancing higher-order thinking skills in vocational education through scaffolding-problem based learning. Open Engineering, 10(1), 612–619.

Teräs, M., Suoranta, J., Teräs, H., & Curcher, M. (2020). Post-Covid-19 Education and Education Technology 'Solutionism': a Seller's Market. Postdigital Science and Education, 2(3), 863–878.

Tondeur, J., Scherer, R., Baran, E., Siddiq, F., Valtonen, T., & Sointu, E. (2019). Teacher educators as gatekeepers: Preparing the next generation of teachers for technology integration in education. British Journal of Educational Technology, 50(3), 1189–1209.

Tyas, E. H., & Naibaho, L. (2021). Hots Learning Model Improves the Quality of Education. International Journal of Research -GRANTHAALAYAH, 9(1), 176–182.

Van der Linden, W. J. (Ed.). (2017). Handbook of Item Response Theory: Volume 2: Statistical Tools. CRC Press.

Wang, M., & Zheng, X. (2021). Using Game-Based Learning to Support Learning Science: A Study with Middle School Students. Asia-Pacific Education Researcher, 30(2), 167–176.

Warman, W., Lorensius, L., & Rohana, R. (2021). Curriculum of Management in Improving the Quality of Catholic School Education in Samarinda City, East Kalimantan, Indonesia. In Budapest International Research and Critics Institute (BIRCI-Journal): Humanities and Social Sciences (Vol. 4, Issue 3). East Kalimantan.

Wells, C. S., & Purwono, U. (2009). Assesing the fit of IRT models to item response data. Makalah Pelatihan Psikometri Kerjasama Pascasarjana UNY dengan USAID.

Widarjono, A. (2020). Analisis multivariat terapan dengan program SPSS, AMOS, dan SMARTPLS.[ Applied multivariate analysis with SPSS, AMOS, and SMART PLS programs]

Xiao, Y., Han, J., Koenig, K., Xiong, J., & Bao, L. (2018). Multilevel Rasch modeling of two-Tier multiple-choice test: A case study using Lawson's classroom test of scientific reasoning. Physical Review Physics Education Research, 14(2), 20104.

Yeung, S. yin S. (2015). The conception of teaching higher-order thinking: perspectives of Chinese teachers in Hong Kong. Curriculum Journal, 26(4), 553–578.




How to Cite

Saepuzaman, D., Edi Istiyono, & Haryanto. (2022). Characteristics of fundamental physics higher-order thinking skills test using Item Response Theory analysis. Pegem Journal of Education and Instruction, 12(4), 269–279.