Designing and Validating an Evidence-Based Pedagogy Framework: A PRISMA–Delphi–IRR Approach for Technical Education

Arthur Dela Peña

Authors

Arthur Dela Peña National Aviation Academy of the Philippines https://orcid.org/0000-0003-2519-2128

Keywords:

Research Methodology, Instrument Development, Competency-Based Education, Assessment Rubric, Quality Assurance, Technical and Vocational Education and Training (TVET), Delphi Method

Abstract

Instructional quality in aviation maintenance training (AMT) is commonly regulated through content requirements, with limited attention to pedagogy and instructional assurance. This study develops and validates RAMP-ACE, an auditable framework for improving the reliability and transparency of competency-based teaching. A PRISMA-guided systematic review initially identified seven preliminary pedagogical domains. Through regulatory triangulation and Delphi refinement, overlapping standardization/calibration and quality-assurance constructs were consolidated, producing a final six-domain framework. The candidate item pool progressed from 14 indicators in v1 to 23 indicators in v2, with 21 retained after two indicators failed Delphi thresholds. Content validity was examined through a two-round asynchronous Delphi process (Round 1 N = 14; Round 2 N = 13), followed by a pilot audit assessing interrater reliability, rating time, artifact availability, and corrective-action closure. Delphi findings showed strong consensus, with most indicators meeting I-CVI ≥ 0.78, S-CVI/Ave = 0.89, and S-CVI/UA = 0.35. Pilot testing produced κ/ICC values of 0.72–0.86, median dossier rating time of 38 minutes, and artifact availability above 85% for key records. Corrective actions were mostly closed within two weeks, supporting operational feasibility and transferability to technical and competency-based education contexts.

References

Abildgren, L., Lebahn-Hadidi, M., Mogensen, C. B., Toft, P., Nielsen, A. B., Frandsen, T. F., Steffensen, S. V., & Hounsgaard, L. (2022). The effectiveness of improving healthcare teams’ human factor skills using simulation-based training: A systematic review. Advances in Simulation, 7(1), Article 12. https://doi.org/10.1186/s41077-022-00207-2

Basileo, L. D., & Toth, M. (2019). A state level analysis of the Marzano Teacher Evaluation Model: Predicting teacher value-added measures with observation scores. Practical Assessment, Research & Evaluation, 24(1), Article 6. https://doi.org/10.7275/cc5b-6j43

Chen, A. M. H., Kleppinger, E. L., Churchwell, M. D., & Rhoney, D. H. (2024). Examining competency-based education through the lens of implementation science: A scoping review. American Journal of Pharmaceutical Education, 88(2), Article 100633. https://doi.org/10.1016/j.ajpe.2023.100633

Chen, S., Zhang, C., & Li, W. (2022). The effects of competency-based training model in the training of new nurses: A meta-analysis and systematic review. PLOS ONE, 17(11), Article e0277484. https://doi.org/10.1371/journal.pone.0277484

Clare, J., & Kourousis, K. I. (2021). Learning from incidents in aircraft maintenance and continuing airworthiness: Regulation, practice and gaps. Aircraft Engineering and Aerospace Technology, 93(2), 338–346. https://doi.org/10.1108/aeat-06-2020-0114

Edgar, A., & Stack, K. M. (2023). Parallel incorporation and public law. International Journal of Constitutional Law, 21(3), 734–769. https://doi.org/10.1093/icon/moad064

Edge, C., Monske, E., Boyer-Davis, S., VandenAvond, S., & Hamel, B. (2021). Leading university change: A case study of meaning-making and implementing online learning quality standards. American Journal of Distance Education, 36(1), 53–69. https://doi.org/10.1080/08923647.2021.2005414

European Union Aviation Safety Agency. (2023). AMC & GM to Part-147—Issue 2, Amendment 3. https://www.easa.europa.eu/sites/default/files/dfu/amc_gm_to_part-147_-_issue_2_amendment_3.pdf

Federal Aviation Administration. (2022). Aviation instructor’s handbook (FAA-H-8083-9B). U.S. Department of Transportation. https://www.faa.gov/regulations_policies/handbooks_manuals/aviation/aviation_instructors_handbook

Franciosi, C., Di Pasquale, V., Iannone, R., & Miranda, S. (2019). A taxonomy of performance shaping factors for human reliability analysis in industrial maintenance. Journal of Industrial Engineering and Management, 12(1), 115–132. https://doi.org/10.3926/jiem.2702

Haapamäki, E. (2022). Insights into neo-institutional theory in accounting and auditing regulation research. Managerial Auditing Journal, 37(3), 336–357. https://doi.org/10.1108/maj-10-2020-2864

Heinonen, H., Virtaluoto, J., Suomivuori, T., Forsman, K., Kangas, T., & Siltanen, S. (2022). Minimalism for the win: User-centered design for guidance in industrial maintenance. IEEE Transactions on Professional Communication, 65(4), 485–501. https://doi.org/10.1109/tpc.2022.3205468

Hovlid, E., Braut, G. S., Hannisdal, E., Walshe, K., Bukve, O., Flottorp, S., Stensland, P., & Frich, J. C. (2020). Mediators of change in healthcare organizations subject to external assessment: A systematic review with narrative synthesis. BMJ Open, 10(8), Article e038850. https://doi.org/10.1136/bmjopen-2020-038850

International Air Transport Association, & International Federation of Air Line Pilots’ Associations. (2021). Guidance material and best practices for instructor and evaluator training (2nd ed.). https://www.ifalpa.org/publications/guidance-material-for-instructor-and-evaluator-training-edition-2-2633/

International Air Transport Association. (2025). Competency assessment and evaluation for pilots, instructors and evaluators: Guidance material. https://www.iata.org/contentassets/c0f61fc821dc4f62bb6441d7abedb076/competency-assessment-and-evaluation-for-pilots-instructors-and-evaluators-gm.pdf

International Civil Aviation Organization. (2020). Procedures for air navigation services: Training (Doc 9868; 3rd ed.). https://store.icao.int/en/procedures-for-air-navigation-services-training-doc-9868

International Civil Aviation Organization. (2025). Training instructors course (TIC). https://store.icao.int/en/training-instructors-course-tic-en

James, H. K., Chapman, A. W., Pattison, G. T. R., Fisher, J. D., & Griffin, D. R. (2020). Analysis of tools used in assessing technical skills and operative competence in trauma and orthopedic surgical training. JBJS Reviews, 8(6), Article e19.00167. https://doi.org/10.2106/jbjs.rvw.19.00167

Jiménez Sierra, Á. A., Ortega Iglesias, J. M., Cabero-Almenara, J., & Palacios-Rodríguez, A. (2023). Development of the teacher’s technological pedagogical content knowledge (TPACK) from the lesson study: A systematic review. Frontiers in Education, 8, Article 1078913. https://doi.org/10.3389/feduc.2023.1078913

Karunakaran, C. S., & Ashok Babu, J. (2022). Impact of human factors in aircraft accident mitigation and aircraft maintenance training needs in post COVID-19 aviation. Aircraft Engineering and Aerospace Technology, 94(8), 1296–1302. https://doi.org/10.1108/AEAT-10-2021-0300

Küçük Yılmaz, A. (2019). Strategic approach to managing human factors risk in aircraft maintenance organization: Risk mapping. Aircraft Engineering and Aerospace Technology, 91(4), 654–668. https://doi.org/10.1108/AEAT-06-2018-0160

Mazzone, E., Puliatti, S., Amato, M., Bunting, B., Rocco, B., Montorsi, F., Mottrie, A., & Gallagher, A. G. (2020). A systematic review and meta-analysis on the impact of proficiency-based progression simulation training on performance outcomes. Annals of Surgery, 274(2), 281–289. https://doi.org/10.1097/sla.0000000000004650

Mendes, N., Vieira, J. G. V., & Mano, A. P. (2022). Risk management in aviation maintenance: A systematic literature review. Safety Science, 153, Article 105810. https://doi.org/10.1016/j.ssci.2022.105810

Mitchell, E. L. (2025). Assessment in the era of competency-based surgical education. Seminars in Vascular Surgery, 38(2), 122–131. https://doi.org/10.1053/j.semvascsurg.2025.04.006

Morris-Mathews, H., Stark, K. R., Jones, N. D., Brownell, M. T., & Bell, C. A. (2021). Danielson’s framework for teaching: Convergence and divergence with conceptions of effectiveness in special education. Journal of Learning Disabilities, 54(1), 66–78. https://doi.org/10.1177/0022219420941804

Niederberger, M., Schifano, J., Deckert, S., Hirt, J., Homberg, A., Köberich, S., Kuhn, R., Rommel, A., & Sonnberger, M. (2024). Delphi studies in social and health sciences—Recommendations for an interdisciplinary standardized reporting (DELPHISTAR): Results of a Delphi study. PLOS ONE, 19(8), Article e0304651. https://doi.org/10.1371/journal.pone.0304651

Paquibut, R., & Al Naamany, A. (2020). Managing organizational change to meet the research–teaching nexus standard: The case of an HEI in the Sultanate of Oman. International Journal of Educational Management, 34(4), 782–793. https://doi.org/10.1108/IJEM-07-2019-0225

Peiró, J. M., Nielsen, K., Latorre, F., Shepherd, R., & Vignoli, M. (2020). Safety training for migrant workers in the construction industry: A systematic review and future research agenda. Journal of Occupational Health Psychology, 25(4), 275–295. https://doi.org/10.1037/ocp0000178

Piggott, T., Langendam, M., Parmelli, E., Adolfsson, J., Akl, E. A., Armstrong, D., Braithwaite, J., Brignardello-Petersen, R., Brozek, J., Gore-Booth, J., Follmann, M., Leś, Z., Meerpohl, J. J., Neamţiu, L., Nothacker, M., Qaseem, A., Giorgi Rossi, P., Saz-Parkinson, Z., van der Wees, P., & Schünemann, H. J. (2021). Bringing two worlds closer together: A critical analysis of an integrated approach to guideline development and quality assurance schemes. BMC Health Services Research, 21, Article 172. https://doi.org/10.1186/s12913-020-05819-w

Safari, K., McKenna, L., & Davis, J. (2023). The evolution of registration standards for internationally qualified midwives in Australia. Midwifery, 119, Article 103606. https://doi.org/10.1016/j.midw.2023.103606

Sarfati, L., Ranchon, F., Vantard, N., Schwiertz, V., Larbre, V., Parat, S., Faudel, A., & Rioufol, C. (2019). Human-simulation-based learning to prevent medication error: A systematic review. Journal of Evaluation in Clinical Practice, 25(1), 11–20. https://doi.org/10.1111/jep.12883

Scorgie, D., Feng, Z., Paes, D., Parisi, F., Yiu, T. W., & Lovreglio, R. (2024). Virtual reality for safety training: A systematic literature review and meta-analysis. Safety Science, 171, Article 106372. https://doi.org/10.1016/j.ssci.2023.106372

Stievano, A., Caruso, R., Pittella, F., Shaffer, F. A., Rocco, G., & Fairman, J. (2019). Shaping nursing profession regulation through history: A systematic review. International Nursing Review, 66(1), 17–29. https://doi.org/10.1111/inr.12449

Suharno, Pambudi, N. A., & Harjanto, B. (2020). Vocational education in Indonesia: History, development, opportunities, and challenges. Children and Youth Services Review, 115, Article 105092. https://doi.org/10.1016/j.childyouth.2020.105092

Teresi, J. A., Yu, X., Stewart, A. L., & Hays, R. D. (2022). Guidelines for designing and evaluating feasibility pilot studies. Medical Care, 60(1), 95–103. https://doi.org/10.1097/MLR.0000000000001664

Thulasy, T. N., Nohuddin, P. N. E., Nusyirwan, I. F., Rahim, N. A., Amrin, A., & Chua, S. (2022). Skills assessment criteria for aircraft maintenance technician in the context of Industrial Revolution 4.0. Journal of Aerospace Technology and Management, 14. https://doi.org/10.1590/jatm.v14.1286

Tyagi, A., Tripathi, R., & Bouarfa, S. (2023). Learning from past in the aircraft maintenance industry: An empirical evaluation in the safety management framework. Heliyon, 9(11), Article e21620. https://doi.org/10.1016/j.heliyon.2023.e21620

Vlasblom, J. I. D., Pennings, H. J. M., van der Pal, J., & Oprins, E. A. P. B. (2020). Competence retention in safety-critical professions: A systematic literature review. Educational Research Review, 30, Article 100330. https://doi.org/10.1016/j.edurev.2020.100330

Vujičić, S., Hasanspahić, N., Gundić, A., & Hrdalo, N. (2020). Assessment for ensuring adequately qualified instructors in maritime education and training institutions. Athens Journal of Sciences, 7(2), 115–126. https://doi.org/10.30958/ajs.7-2-4

Yusop, S. R. M., Rasul, M. S., Mohamad Yasin, R., Hashim, H. U., & Jalaludin, N. A. (2022). Assessment approaches and learning outcomes in technical and vocational education: A systematic review using PRISMA. Sustainability, 14(9), Article 5225. https://doi.org/10.3390/su14095225

Yusop, S. R. M., Rasul, M. S., Mohammad Yasin, R., & Hashim, H. U. (2023). Identifying and validating vocational skills domains and indicators in classroom assessment practices in TVET. Sustainability, 15(6), Article 5195. https://doi.org/10.3390/su15065195