運算思維能力已日趨重要，世界各地的教育家也開始發展關於運算思維能力培養的課程，而如何有效地評量學生的學習成效也十分重要，目前有許多測驗聲稱能夠評量學生的運算思維能力，但未能說明評量到哪一項概念。因此，本研究以廣為人知的國際運算思維挑戰賽（International Challenge on Informatics and Computational Thinking）為例，利用德菲法進行三回合問卷調查，蒐集專家們的想法，統整出挑戰賽題與運算思維核心概念之關係，並分析各類型專家進行題目評析時的差異，包含哪一類型專家較不受他人影響及其想法較具代表性。
　　本研究礙於題目樣本數及專家人數的關係，將專家分為兩群，評析兩份不一樣的問卷，作為對照，以探討各類專家的差異。
　　實驗結果得知，三回合德菲法問卷調查，可將挑戰賽題依據運算思維五大概念分類，且發現未有任何一題被分類為「分解」這一概念。另外，本研究亦發現Bebras Challenge之命題者在評析題目時也容易受他人影響，並且其填答也未必具代表性，主要是德菲法的匿名性使得評析結果更加客觀。

Computational thinking has become increasingly important. Educators around the world have also begun to develop courses on the development of computational thinking skills. The measurement of students' learning effectiveness is also very important. There are many tests that claim to be able to measure it. But most of them can not be explained which concept is evaluated. Therefore, this study takes the “International Challenge on Informatics and Computational Thinking” as an example. We use Delphi method to conduct a three-round questionnaire survey, collects the ideas of experts, and integrates the core concept of computational thinking and the tasks. We analyzed which types of experts are less affected by others and which are more representative.
In this study, due to the number of questions and the experts, the experts were divided into two groups with two different questionnaires that were evaluated as a comparison to explore the differences among various experts.
The experimental results show that the three-round Dephi questionnaire can classify the tasks according to the concepts of computational thinking, and find that no one is classified as "decomposition". In addition, this study also found that the Bebras Challenge's propositions are also susceptible to others when evaluating the questions, and their answers are not necessarily representative, mainly because the anonymity of Dephi makes the results of the evaluation more objective.

Aho, A.V. (2012). Computation and computational thinking. Computer Journal, 55(7), 833–835. https://doi.org/10.1093/comjnl/bxs074
Andre L. Delbecq, Andrew H. Van De Ven, D. H. G. (1975). Group Techniques for Program Planning: A Guide To Nominal Group and Delphi Processes (Book). Personnel Psychology, 28(4), 598–600. https://doi.org/10.1177/002188637601200414
B, P. H., Hubwieser, E., &Graswald, D. (2016). Informatics in Schools: Improvement of Informatics Knowledge and Perception, 9973, 40–52. https://doi.org/10.1007/978-3-319-46747-4
Barendsen, E., Mannila, L., Demo, B., Grgurina, N., Izu, C., Mirolo, C., …Stupurienė, G. (2015). Concepts in K-9 Computer Science Education. Proceedings of the 2015 ITiCSE on Working Group Reports - ITICSE-WGR ’15. https://doi.org/10.1145/2858796.2858800
Barr, V., &Stephenson, C. (2011). Bringing computational thinking to K-12. ACM Inroads, 2(1), 48. https://doi.org/10.1145/1929887.1929905
Bell, T., Curzon, P., Cutts, Q., Dagiene, V., &Haberman, B. (2011). Overcoming obstacles to CS education by using non-programming outreach programmes. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 7013 LNCS, 71–81. https://doi.org/10.1007/978-3-642-24722-4_7
Bellettini, C., Lonati, V., Malchiodi, D., Monga, M., Morpurgo, A., &Torelli, M. (2015). How challenging are bebras tasks? An irt analysis based on the performance of Italian students. Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE, 2015–June, 27–32. https://doi.org/10.1145/2729094.2742603
Blair, Shirley; Uhl, N. P. (1990). Using the Delphi Method to Improve the Curriculum.
Brooks, &W., K. (1979). Delphi Technique：Expanding Applications. North Central Association Quarterly, v53 n3 p377-85 Win 1979.
Cartelli, A., Dagiene, V., &Futschek, G. (2010). Bebras Contest and Digital Competence Assessment. International Journal of Digital Literacy and Digital Competence, 1(1), 24–39. https://doi.org/10.4018/jdldc.2010101902
Chin-Chia, Y., Seng-Lee, W., &Jih-Hwa, W. (2007). A Study Comparing the Delphi Method and Fuzzy Delphi Method.
Dagiene, V., &Futschek, G. (2013). Bebras A Contest to Motivate Students to Study Computer Science and Develop Computational Thinking. The 10th International Conference on Computers in Education (WCCE 2013), 139–141.
Dagienė Valentina; Pelikis, E. S. G., Dagienė, V., Pelikis, E., Dagiene, V., &Dagienė, V. (2015). Introducing Computational Thinking through a Contest on Informatics: Problem-solving and Gender Issues. Informacijos Mokslai/, 73(January 2017), 55–63. Retrieved from http://search.ebscohost.com/login.aspx?direct=true&profile=ehost&scope=site&authtype=crawler&jrnl=13920561&AN=115175446&h=H%2BQE1mWCVQ4Y8F6DuR79t4LEirCPZzBeWocaPKoEvS2HzMe1BzE%2BL2WGf3hGQGIbMP%2BTdaD6yLY1PxBYFQo80w%3D%3D&crl=c
Dalkey, N. (1969). The Delphi Method: An Experimental Study of Group Opinion. The RAND Corporation. https://doi.org/10.1016/S0016-3287(69)80025-X
Dalkey, N. C. (1975). The methodological worth of the Delphi forecasting technique. Technological Forecasting and Social Change, 7(2), 179–192. https://doi.org/10.1016/0040-1625(75)90057-8
Dalkey, N. C., Corporation, R., Scientific, C. F., &(US), T. I. (1969). The Delphi method: An experimental study of group opinion. Retrieved from http://www.rand.org/content/dam/rand/pubs/research_memoranda/2005/RM5888.pdf
Dalkey, N., &Olaf, H. (1962). An experimental application of the Delphi method to the use of experts.
Google for Education. (2015). Google for Education: Computational Thinking. Retrieved from https://edu.google.com/resources/programs/exploring-computational-thinking/%0Ahttps://edu.google.com/resources/programs/exploring-computational-thinking/%5Cnhttps://edu.google.com/resources/programs/exploring-computational-thinking/#!ct-overview
Gouws, L., Bradshaw, K., &Wentworth, P. (2013). First year student performance in a test for computational thinking. Proceedings of the South African Institute for Computer Scientists and Information Technologists Conference on - SAICSIT ’13, 271. https://doi.org/10.1145/2513456.2513484
Grover, S., &Pea, R. (2013). Computational Thinking in K-12: A Review of the State of the Field. Educational Researcher, 42(1), 38–43. https://doi.org/10.3102/0013189X12463051
Izu, C., Mirolo, C., Settle, A., Mannila, L., &Stupuriene, G. (2017). Exploring bebras tasks content and performance: A multinational study. Informatics in Education, 16(1), 39–59. https://doi.org/10.15388/infedu.2017.03
Lee, G., Lin, Y.-T., &Lin, J. (2014). Assessment of Computational Thinking Skill among High School and Vocational School Students in Taiwan. World Conference on Educational Media and Technology, 2014(1), 173–180. Retrieved from http://www.editlib.org/p/147499/
Lee, I., Martin, F., Denner, J., Coulter, B., Allan, W., Erickson, J., …Werner, L. (2011). Computational thinking for youth in practice. ACM Inroads, 2(1), 32. https://doi.org/10.1145/1929887.1929902
Linstone, H. A.; Turoff, M. (1975). The Delphi method: Techniques and applications.
Lu, J. J., &Fletcher, G. H. L. (2009). Thinking About Computational Thinking Categories and Subject Descriptors. Sigcse, 260–264. https://doi.org/10.1145/1539024.1508959
Lye, S. Y., &Koh, J. H. L. (2014). Review on teaching and learning of computational thinking through programming: What is next for K-12? Computers in Human Behavior, 41, 51–61. https://doi.org/10.1016/j.chb.2014.09.012
Riggs, W. E. (1983). The Delphi Technique An Experimental Evaluation. Forecasting and Social Change, 23, 89–94. https://doi.org/10.1016/0040-1625(83)90073-2
Román-gonzález, M. (2016). Computational Thinking Test : Design Guidelines and Content Validation. Proceedings of EDULEARN15 Conference, (March), 2436–2444. https://doi.org/10.13140/RG.2.1.4203.4329
Selby, C. (2013). Computational Thinking : The Developing Definition. ITiCSE Conference 2013, 5–8.
Todd, R. F., &Reece, C. C. (1989). Desirable Skills and Knowledge Outcomes for an Introductory Educational Research Course: A Delphi Study. Annual Meeting of the American Educational Research Association. https://doi.org/ED305342
Tucker, A., Seehorn, D., Carey, S., Moix, D., Fuschetto, B., Lee, I., …Verno, A. (2011). CSTA K-12 Computer Science Standards Revised 2011 CSTA Standards Task Force The Association for Computing Machinery , Inc .
Walden, J., Doyle, M., Garns, R., &Hart, Z. (2013). An informatics perspective on computational thinking. Proceedings of the 18th ACM Conference on Innovation and Technology in Computer Science Education - ITiCSE ’13, 4. https://doi.org/10.1145/2462476.2483797
Werner, L., Denner, J., &Campe, S. (2012). The Fairy Performance Assessment : Measuring Computational Thinking in Middle School. Proceedings of the 43rd ACM Technical Symposium on Computer Science Education - SIGCSE ’12, 215–220. https://doi.org/10.1145/2157136.2157200
Wing, J. M. (2006). Computational Thinking, 49(3), 33–35. https://doi.org/10.1145/1118178.1118215
Wing, J. M. (2008). Computational thinking and thinking about computing. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 366(1881), 3717–3725. https://doi.org/10.1098/rsta.2008.0118
Parente, F., & Anderson-Parente, J. (1987). Delphi Inquiry Systems. In Wright, G., & Ayton, P. (Eds.), Judgmental Forecasting. NY: John Wiley & Sons.
Lanford, H. W . (1972). Technological Forecasting Methodologies: A Synthesis. New York:American Management Association, Inc.
葉重新（2005）。教育心理學。臺中市：葉重新。
陳陹埅（1996）。國民中學生活科技教師專門能力及其內涵之研究。國立臺灣師範大學工業技術研究所博士論文，未出版，臺北市。
林振春（1992）。德惠法（Delphi Technique）。台北：民意月刊第196期，頁82-101。