Authors : Dian Pertiwi Josua; Anan Sutisna; Muchlas Suseno; Riyan Arthur

Volume/Issue : Volume 10 - 2025, Issue 11 - November

Google Scholar : https://tinyurl.com/2p8yr6m8

Scribd : https://tinyurl.com/ykb89t9r

DOI : https://doi.org/10.38124/ijisrt/25nov1212

Note : A published paper may take 4-5 working days from the publication date to appear in PlumX Metrics, Semantic Scholar, and ResearchGate.

Note : Google Scholar may take 30 to 40 days to display the article.

Abstract : The digital transformation in education has driven the need for policy evaluation models that can integrate technology-based data and processes more accurately and sustainably. This study aims to conduct a meta-analysis of the effectiveness of various education policy evaluation models (CIPP, Stake, and responsive) implemented in the context of Indonesia's digital ecosystem. This study analyzed 117 quantitative data sets from published articles from 2020 to 2025, selected using the PRISMA procedure, with inclusion criteria emphasizing the use of technology-based evaluation instruments and the reporting of effect sizes. Data were analyzed using a random-effects model to estimate the pooled effect size, conduct heterogeneity tests, perform moderator analyses (level of education, type of policy, and form of technology), and assess publication bias. The results indicate that the implementation of digital-based evaluation models has a significant positive effect on the quality of education policy evaluations, with a pooled effect size in the medium range (ES = 0.54) and heterogeneity partially explained by variations in the level and type of policy intervention. The findings also indicate relatively low publication bias and consistency of results across studies. The practical implications of this research emphasize the importance of strengthening stakeholders' digital literacy, developing standardized, integrated evaluation instruments for information systems, and formulating policies that support the equitable use of technology to improve the accountability and effectiveness of digital education policy evaluation.

Keywords : Educational Policy Evaluation; Effectiveness; Meta-Analysis; Responsive Model; Technology in Education.

References :

1. Xiong Y, Zhang C, Qi H. How effective is the fire safety education policy in China? A quantitative evaluation based on the PMC-index model. Saf Sci. 2023;161.
2. Chen Z, Song Z, Yuan S, Chen W. Influence Analysis of Education Policy on Migrant Children’s Education Integration Using Artificial Intelligence and Deep Learning. Front Psychol. 2022;13.
3. Reinoso-Avecillas RL, Chicaiza-Aucapiña DI. Quality referents in Ecuadorian higher technological education. Sophia(Ecuador). 2022;2022(33).
4. Crysdian C. The evaluation of higher education policy to drive university entrepreneurial activities in information technology learning. Cogent Education. 2022;9(1).
5. Marzal MÁ, Vivarelli M. The convergence of Artificial Intelligence and Digital Skills: a necessary space for Digital Education and Education 4.0. JLIS.it. 2024;15(1).
6. Spinnewijn L, Scheele F, Braat D, Aarts J. Assessing the educational quality of shared decision-making interventions for residents: A systematic review. Vol. 123, Patient Education and Counseling. 2024.
7. Véliz Salazar MI, Gutiérrez Marfileño VE. Teaching models on good teaching practices in virtual classrooms. Apertura. 2021;13(1).
8. Orji FA, Vassileva J. Automatic modeling of student characteristics with interaction and physiological data using machine learning: A review. Vol. 5, Frontiers in Artificial Intelligence. 2022.
9. Mousavinasab E, Zarifsanaiey N, R. Niakan Kalhori S, Rakhshan M, Keikha L, Ghazi Saeedi M. Intelligent tutoring systems: a systematic review of characteristics, applications, and evaluation methods. Vol. 29, Interactive Learning Environments. 2021.
10. Lee JH, Lee H, Kim S, Choi M, Ko IS, Bae JY, et al. Debriefing methods and learning outcomes in simulation nursing education: A systematic review and meta-analysis. Vol. 87, Nurse Education Today. 2020.
11. Özer E, Çetinkaya Şen Y, Canlı S, Güvenç G. Effects of Virtual Reality Interventions on the Parameters of Normal Labor: A Systematic Review and Meta-Analysis of Randomized Controlled Trials. A Meta-Analysis of Virtual Reality Interventions on the Parameters of Normal Labor. Pain Management Nursing. 2024;25(1).
12. Jayanegara A, Mukhtarom A, Marzuki I. Innovative learning methods of Islamic education subject in Indonesia: a meta-analysis. International Journal of Evaluation and Research in Education. 2024;13(2).
13. Zeng HL, Chen DX, Li Q, Wang XY. Effects of seminar teaching method versus lecture-based learning in medical education: A meta-analysis of randomized controlled trials. Med Teach. 2020;42(12).
14. Hurtado-Parrado C, Pfaller-Sadovsky N, Medina L, Gayman CM, Rost KA, Schofill D. A Systematic Review and Quantitative Analysis of Interteaching. J Behav Educ. 2022;31(1).
15. Kaya İT, Aktaş MT. The Effectiveness of Nudge Methods in Education: A Meta-Analysis Study. Milli Egitim. 2024;53(241).
16. Kambach S, Bruelheide H, Gerstner K, Gurevitch J, Beckmann M, Seppelt R. Consequences of multiple imputation of missing standard deviations and sample sizes in meta-analysis. Ecol Evol. 2020;10(20).
17. Cai S, Zhou J, Pan J. Estimating the sample mean and standard deviation from order statistics and sample size in meta-analysis. Stat Methods Med Res. 2021;30(12).
18. Ridwan MR, Hadi S, Jailani J. A meta-analysis study on the effectiveness of a cooperative learning model on vocational high school students’ mathematics learning outcomes. Participatory Educational Research. 2022;9(4).
19. Gallardo‐Gómez D, Richardson R, Dwan K. Standardized mean differences in meta‐analysis: A tutorial. Cochrane Evidence Synthesis and Methods. 2024;2(3).
20. Lin L, Aloe AM. Evaluation of various estimators for standardized mean difference in meta-analysis. Stat Med. 2021;40(2).
21. Sun RW, Cheung SF. The influence of nonnormality from primary studies on the standardized mean difference in meta-analysis. Behav Res Methods. 2020;52(4).
22. Fox JW. How much does the typical ecological meta-analysis overestimate the true mean effect size? Ecol Evol. 2022;12(11).
23. Bakbergenuly I, Hoaglin DC, Kulinskaya E. Estimation in meta-analyses of mean difference and standardized mean difference. Stat Med. 2020;39(2).
24. Andrade C. Mean Difference, Standardized Mean Difference (SMD), and Their Use in Meta-Analysis: As Simple as It Gets. Journal of Clinical Psychiatry. 2020;81(5).
25. Chi KY, Li MY, Chen C, Kang E. Ten circumstances and solutions for finding the sample mean and standard deviation for meta-analysis. Syst Rev. 2023;12(1).
26. Kwon D, Reddy RRS, Reis IM. ABCMETAapp: R shiny application for simulation-based estimation of mean and standard deviation for meta-analysis via approximate Bayesian computation. Res Synth Methods. 2021;12(6).
27. McGrath S, Zhao XF, Steele R, Thombs BD, Benedetti A, Levis B, et al. Estimating the sample mean and standard deviation from commonly reported quantiles in meta-analysis. Stat Methods Med Res. 2020;29(9).
28. Sandercock G. The Standard Error/Standard Deviation Mix-Up: Potential Impacts on Meta-Analyses in Sports Medicine. Sports Medicine. 2024;54(6).
29. Maassen E, Van Assen MALM, Nuijten MB, Olsson-Collentine A, Wicherts JM. Reproducibility of individual effect sizes in meta-analyses in psychology. PLoS One. 2020;15(5).
30. Ratan R, Beyea D, Li BJ, Graciano L. Avatar characteristics induce users’ behavioral conformity with small-to-medium effect sizes: a meta-analysis of the proteus effect. Media Psychol. 2020;23(5).
31. Gucciardi DF, Lines RLJ, Ntoumanis N. Handling effect size dependency in meta-analysis. Int Rev Sport Exerc Psychol. 2022;15(1).
32. Nuijten MB, Van Assen MALM, Augusteijn HEM, Crompvoets EAV, Wicherts JM. Effect sizes, power, and biases in intelligence research: A meta-meta-analysis. J Intell. 2020;8(4).
33. Braga AA, Weisburd DL. Does Hot Spots Policing Have Meaningful Impacts on Crime? Findings from An Alternative Approach to Estimating Effect Sizes from Place-Based Program Evaluations. J Quant Criminol. 2022;38(1).
34. Almulhim AN, Hartley H, Norman P, Caton SJ, Doğru OC, Goyder E. Behavioural Change Techniques in Health Coaching-Based Interventions for Type 2 Diabetes: A Systematic Review and Meta-Analysis. BMC Public Health. 2023;23(1).
35. Kim J, Castelli DM. Effects of gamification on behavioral change in education: A meta-analysis. Vol. 18, International Journal of Environmental Research and Public Health. 2021.
36. Costa JM, Miranda GL, Melo M. Four-component instructional design (4C/ID) model: a meta-analysis on use and effect. Learn Environ Res. 2022;25(2).
37. TURGUT S. A Meta-Analysis of the Effects of Realistic Mathematics Education-based Teaching on Mathematical Achievement of Students in Turkey. Journal of Computer and Education Research. 2021;9(17).
38. Dalgaard NT, Bondebjerg A, Klokker R, Viinholt BCA, Dietrichson J. Adult/child ratio and group size in early childhood education or care to promote the development of children aged 0–5 years: A systematic review. Vol. 18, Campbell Systematic Reviews. 2022.
39. Zhao J, Xu X, Jiang H, Ding Y. The effectiveness of virtual reality-based technology on anatomy teaching: A meta-analysis of randomized controlled studies. BMC Med Educ. 2020;20(1).
40. Efendi D, Apriliyasari RW, Prihartami Massie JGE, Wong CL, Natalia R, Utomo B, et al. The effect of virtual reality on cognitive, affective, and psychomotor outcomes in nursing staffs: systematic review and meta-analysis. BMC Nurs. 2023;22(1).
41. Rafati S, Baniasadi T, Dastyar N, Zoghi G, Ahmadidarrehsima S, Salari N, et al. Prevalence of self-medication among the elderly: A systematic review and meta-analysis. Vol. 12, Journal of Education and Health Promotion. 2023.
42. Kossmeier M, Tran US, Voracek M. Charting the landscape of graphical displays for meta-analysis and systematic reviews: A comprehensive review, taxonomy, and feature analysis. Vol. 20, BMC Medical Research Methodology. 2020.
43. Fernández-Castilla B, Declercq L, Jamshidi L, Beretvas SN, Onghena P, van den Noortgate W. Visual representations of meta-analyses of multiple outcomes: Extensions to forest plots, funnel plots, and caterpillar plots. Methodology. 2020;16(4).
44. Wang N. Conducting Meta-analyses of Proportions in R. Journal of Behavioral Data Science. 2023;3(2).
45. Quintana DS. A Guide for Calculating Study-Level Statistical Power for Meta-Analyses. Adv Methods Pract Psychol Sci. 2023;6(1).
46. Liang YE, Ho SYC, Chien TW, Chou W. Analyzing the number of articles with network meta-analyses using chord diagrams and temporal heatmaps over the past 10 years: Bibliometric analysis. Vol. 102, Medicine (United States). 2023.
47. Eaton K, Disher T, Peterson S, Cameron C. PNS147 Visualization And Communication Of Results From Bayesian Network Meta-Analysis: Past, Present, And Future. Value in Health. 2020;23.
48. Kossmeier M, Tran US, Voracek M. Power-Enhanced Funnel Plots for Meta-Analysis: The Sunset Funnel Plot. Zeitschrift fur Psychologie / Journal of Psychology. 2020;228(1).
49. Cheema HA, Shahid A, Ehsan M, Ayyan M. The misuse of funnel plots in meta-analyses of proportions: Are they really useful? Vol. 15, Clinical Kidney Journal. 2022.
50. Brush PL, Sherman M, Lambrechts MJ. Interpreting Meta-Analyses A Guide to Funnel and Forest Plots. Clin Spine Surg. 2024;37(1).
51. Nakagawa S, Lagisz M, Jennions MD, Koricheva J, Noble DWA, Parker TH, et al. Methods for testing publication bias in ecological and evolutionary meta-analyses. Vol. 13, Methods in Ecology and Evolution. 2022.
52. Ismaniati C, Muhtadi A, Cobena DY, Soeparno PL. Effectiveness of Flipped Classroom on Students’ Learning Outcome in Vocational High School: A Meta-Analysis. International Journal of Instruction. 2023;16(1).
53. Br Bangun HK, Simanjuntak DC. The Effects Of Vocabulary Mastery On English-Speaking Ability: A Meta-Analysis Study. Journal of Languages and Language Teaching. 2022;10(2).
54. Vashisht S, Kaushal P, Vashisht R. Emotional intelligence, Personality Variables and Career Adaptability: A Systematic Review and Meta-analysis. Vision. 2023;27(3).
55. Orosoo M, Jamiyansuren B. Language in education planning: Evaluation policy in Mongolia. Journal of Language and Linguistic Studies. 2021;17(3).
56. Dwomoh D, Godi A, Tetteh J, Amoatey C, Otoo R, Tornyevah L, et al. The Impact of the Free Senior High School Education Policy and Double-Track System on Quality Education Outcomes: A Quasi-Experimental Policy Evaluation Study in Ghana. Africa Education Review. 2022;19(2).
57. Fadhil I, Sabic-El-Rayess A. Providing Equity of Access to Higher Education in Indonesia: A Policy Evaluation. Indonesian Journal on Learning and Advanced Education (IJOLAE). 2021;
58. De Mattos LK, Flach L, de Melo PA. Educational scholarship policies for higher education, internationalization, and evaluation of brazilian graduate programs: A study with panel regression. Educ Policy Anal Arch. 2020;28.
59. Street C, Guenther J, Smith J, Robertson K, Ludwig W, Motlap S, et al. Do numbers speak for themselves? Exploring the use of quantitative data to measure policy ‘success’ in historical Indigenous higher education in the Northern Territory, Australia. Race Ethn Educ. 2022;25(3).
60. Mazibuko X, Chimbari M. Development and evaluation of the Ingwavuma receptive vocabulary test: A tool for assessing receptive vocabulary in Isizulu-speaking preschool children. South African Journal of Communication Disorders. 2020;67(1).
61. Langenberg B, Janczyk M, Koob V, Kliegl R, Mayer A. A tutorial on using the paired t test for power calculations in repeated measures ANOVA with interactions. Behav Res Methods. 2023;55(5).
62. O’Regan M, Carthy A, McGuinness C, Owende P. Employer collaboration in developing graduate employability: a pilot study in Ireland. Education and Training. 2022;65(10).
63. Schuetze BA, Yan VX. Psychology Faculty Overestimate the Magnitude of Cohen’s d Effect Sizes by Half a Standard Deviation. Collabra Psychol. 2023;9(1).
64. Springham M, Williams S, Waldron M, Strudwick AJ, Mclellan C, Newton RU. Prior workload has moderate effects on high-intensity match performance in elite-level professional football players when controlling for situational and contextual variables. J Sports Sci. 2020;38(20).
65. Filges T, Dalgaard NT, Viinholt BCA. Outreach programs to improve life circumstances and prevent further adverse developmental trajectories of at-risk youth in OECD countries: A systematic review. Vol. 18, Campbell Systematic Reviews. 2022.
66. De Blume APG. Calibrating Calibration: A Meta-Analysis of Learning Strategy Instruction Interventions to Improve Metacognitive Monitoring Accuracy. J Educ Psychol. 2021;114(4).
67. Havard B, Podsiad M. A meta-analysis of wearables research in educational settings published 2016–2019. Educational Technology Research and Development. 2020;68(4).
68. Migliavaca CB, Stein C, Colpani V, Barker TH, Ziegelmann PK, Munn Z, et al. Meta-analysis of prevalence: I2 statistic and how to deal with heterogeneity. Res Synth Methods. 2022;13(3).
69. Kulinskaya E, Hoaglin DC, Bakbergenuly I, Newman J. A Q statistic with constant weights for assessing heterogeneity in meta-analysis. Res Synth Methods. 2021;12(6).
70. Lin L. Comparison of four heterogeneity measures for meta-analysis. J Eval Clin Pract. 2020;26(1).
71. Kuk S, Lee W. On the finite sample distribution of the likelihood ratio statistic for testing heterogeneity in meta-analysis. Biometrical Journal. 2020;62(8).
72. Hemming K, Hughes JP, McKenzie JE, Forbes AB. Extending the I-squared statistic to describe treatment effect heterogeneity in cluster, multi-centre randomized trials and individual patient data meta-analysis. Stat Methods Med Res. 2021;30(2).
73. Dettori JR, Norvell DC, Chapman JR. Fixed-Effect vs Random-Effects Models for Meta-Analysis: 3 Points to Consider. Vol. 12, Global Spine Journal. 2022.
74. Zhai C, Guyatt G. Fixed-effect and random-effects models in meta-analysis. Vol. 137, Chinese Medical Journal. 2024.
75. Baragilly M, Willis BH. On estimating a constrained bivariate random effects model for meta-analysis of test accuracy studies. Stat Methods Med Res. 2022;31(2).
76. Jackson D, Viechtbauer W, van Aert RCM. Multistep estimators of the between-study covariance matrix under the multivariate random-effects model for meta-analysis. Stat Med. 2024;43(4).
77. Van Aert RCM, Mulder J. Bayesian hypothesis testing and estimation under the marginalized random-effects meta-analysis model. Vol. 29, Psychonomic Bulletin and Review. 2022.
78. Sotola LK. Garbage In, Garbage Out? Evaluating the Evidentiary Value of Published Meta-analyses Using Z-Curve Analysis. Collabra Psychol. 2022;8(1).
79. Fadhli M, Brick B, Setyosari P, Ulfa S, Kuswandi D. A meta-analysis of selected studies on the effectiveness of gamification method for children. International Journal of Instruction. 2020;13(1).
80. Escobar-Soler C, Berrios R, Peñaloza-Díaz G, Melis-Rivera C, Caqueo-Urízar A, Ponce-Correa F, et al. Effectiveness of Self-Affirmation Interventions in Educational Settings: A Meta-Analysis. Vol. 12, Healthcare (Switzerland). 2024.
81. Zhao L, Yang C, Zuo M, Yang H. Short-term Effects of Opportunistic Salpingectomy on Ovarian Reserve：a Meta-analysis of Randomized Controlled Trials Based on GRADE Evidence Grading System. Chinese General Practice. 2024;27(9).
82. Yu Q, Yu K, Li B. Can gamification enhance online learning? Evidence from a meta-analysis. Educ Inf Technol (Dordr). 2024;29(4).
83. Ahmad IF, Setiawati FA, Prihatin RP, Fitriyah QF, Thontowi ZS. Technology-based learning effect on the learning outcomes of Indonesian students: a meta-analysis. International Journal of Evaluation and Research in Education . 2024;13(2).
84. Qiu X bin, Shan C, Yao J, Fu Q ke. The effects of virtual reality on EFL learning: A meta-analysis. Educ Inf Technol (Dordr). 2024;29(2).
85. Pustejovsky JE, Chen M. Equivalencies Between Ad Hoc Strategies and Multivariate Models for Meta-Analysis of Dependent Effect Sizes. Journal of Educational and Behavioral Statistics. 2024;49(6).
86. Ludwig J, Barbek R, von dem Knesebeck O. Education and suicidal ideation in Europe: A systematic review and meta-analysis. Vol. 349, Journal of Affective Disorders. 2024.
87. Yusuf FA. Total Quality Management (TQM) and Quality of Higher Education: A Meta-Analysis Study. International Journal of Instruction. 2023;16(2).
88. AlWahaibi ISH, AlHadabi DAMY, AlKharusi HAT. Cohen’s criteria for interpreting practical significance indicators: A critical study. Cypriot Journal of Educational Sciences. 2020;15(2).
89. Marier JF, Teuscher N, Mouksassi MS. Evaluation of covariate effects using forest plots and introduction to the coveffectsplot R package. CPT Pharmacometrics Syst Pharmacol. 2022;11(10).
90. Katyeudo KK, de Souza RAC. Digital Transformation towards Education 4.0. Informatics in Education. 2022;21(2).
91. Zhou L. How to Develop 21st Century Skills in Students: The Role of LEGO Education. Science Insights Education Frontiers. 2023;15(2).
92. Nurulita I, Ihtiari P, Filipus Yubeleo DP, Umi K. Optimizing 4C Skills through Team Based Projects Using Product Oriented Modules for Electrical Engineering Education Students. SAR Journal - Science and Research. 2022;
93. Ridho S, Wardani S, Saptono S. Development of Local Wisdom Digital Books to Improve Critical Thinking Skills through Problem Based Learning. Journal of Innovative Science Education. 2021;9(3).
94. Matli W, Ngoepe M. Capitalizing on digital literacy skills for capacity development of people who are not in education, employment or training in South Africa. African Journal of Science, Technology, Innovation and Development. 2020;12(2).
95. Sharma RS, Mokhtar IA, Ghista DN, Nazir A, Khan SZ. Digital literacies as policy catalysts of social innovation and socio-economic transformation: Interpretive analysis from Singapore and the UAE. Sustainable Social Development. 2023;1(1).
96. Chalik AA, Samosir FT. Strengthening Student Digital Literature Capacity in Implementation of E-Learning in the Covid-19 Pandemic (Study in Social Welfare Study Program, Faculty of Social …. … Critics Institute-Journal (BIRCI-Journal). 2022.
97. Lee Y. A Study on the Digital Literacy Capacity and Perception of New College Students. The Korean Society of Culture and Convergence. 2023;45(4).
98. Saetang W, Seksan J, Thongsri N. How Academic Majors In Non-Stem Affect Digital Literacy: The Empirical Study. J Technol Sci Educ. 2023;13(3).
99. Lee H, Lim JA, Nam HK. Effect of a Digital Literacy Program on Older Adults’ Digital Social Behavior: A Quasi-Experimental Study. Int J Environ Res Public Health. 2022;19(19).
100. Fazilla S, Yus A, Muthmainnah M. Digital Literacy and TPACK’s Impact on Preservice Elementary Teachers’ Ability to Develop Science Learning Tools. Profesi Pendidikan Dasar. 2022;9(1).
101. Hawamdeh M, Soykan E. Systematic analysis of effectiveness of using mobile technologies (Mt) in teaching and learning foreign language. Online J Commun Media Technol. 2021;11(4).
102. Andriyani FD, De Cocker K, Priambadha AA, Biddle SJH. Physical activity and sedentary behaviour of male adolescents in Indonesia during the COVID-19 pandemic: a mixed-method case study using accelerometers, automated wearable cameras, diaries, and interviews. Journal of Activity, Sedentary and Sleep Behaviors. 2023;2(1).
103. Adrias, Fitria Y, Ladiva HB, Ruswandi A, Erita Y. The Ability and Readiness of Prospective Elementary School Teachers in Facing Digital-Based Learning Era. International Journal of Elementary Education. 2023;7(3).
104. Muhalim M. Envisioning Online English Teaching in Indonesia: A Digital Autoethnographic Account. Qualitative Report. 2023;28(3).
105. Muslim MHA, Zulfiani Z. Analysis of e-learning readiness level implementation in islamic senior high school south jakarta, indonesia: Review in biological learning. Biosfer. 2023;16(2).
106. Taengetan M, Masengi EE, Tumbel G. Implementation of Teacher Certification Policies at SMK Negeri 1 Bitung. Technium Social Sciences Journal. 2023;39.
107. Batmetan JR, Katuuk DA, Lengkong JSJ, Rotty VNJ. An Investigation of E-Learning Readiness in Vocational High School During the Post Pandemic Covid-19: Case from North Sulawesi. International Journal of Information Technology and Education. 2023;2(3).