以影片等多媒體教材融入教學需要學習者對於畫面中的文字(verbal)和圖像(pictorial)資訊進行同步整合，然而由於大腦的認知資源有限，影片中的圖文呈現方式應考量學習者的認知負荷能力，以達到最有效的影片理解效果。因此，本研究旨在透過操控影片字幕(全字幕、部分字幕)和影像(動畫、關鍵畫面)的呈現模式，探討畫面中圖文資訊量的減少能否降低學習者的認知負荷並促進影片理解。共有62位台灣北部高職應外科的學生參與實驗，研究者參考受試者前兩學期的英語成就測驗表現，將受試者平均分配到四種影片模式：全字幕+動畫、全字幕+關鍵畫面、部分字幕+動畫、部分字幕+關鍵畫面。受試者在觀賞影片後即接受理解測驗，題目設計涵蓋三個層面(全面、局部、推論理解)，以期在整體理解分數下，進一步反映不同的理解深度。測驗結束後，研究者透過量表問卷調查受試者對其影片呈現模式的看法，及其在觀影過程中所感受的認知負荷類型(內在、外在、增生負荷)，為本研究提供相關的質性解釋和理論依據。
研究結果顯示，雖然字幕呈現方式不影響各層面的影片理解，但圖像呈現方式對於影片的「整體理解」具有主要效果。其中，不論搭配全字幕或部分字幕，關鍵畫面皆能以適當的資訊量輔助學習者聚焦於文字處理並掌握影片重點；而受試者雖然較偏好動畫中豐富的視覺刺激，卻也表示這會導致他們無法同時注意及處理影片的字幕。另外，本研究也發現字幕和畫面的呈現方式對於影片的「局部理解」有顯著的交互作用，特別是在關鍵畫面下，搭配全字幕更能引導受試者從完整的文字敘述中汲取特定資訊；相反地，若是搭配部分字幕則會使資訊過於破碎，導致無法串連影片細節。本研究將圖像資訊的影響納入現存針對第二外語學習者的字幕研究，並證實影片中的圖文搭配會間接影響學習者的注意力和影片理解成效。據此，教師在透過影片實施教學時，可視教學目的或學生需要精進的理解層面來選擇或微調影片中的圖文呈現方式。

Teaching and learning through video materials require the learner’s cognitive processing of concurrent verbal and pictorial information. To facilitate L2 learners’ video comprehension, the amount of presented information should thus be compatible with human’s finite cognitive capacity. In light of this, the current study aims to explore whether the reduction on verbal/pictorial information in a video yields lower cognitive load and better comprehension gain. The manipulation on caption mode and pictorial mode resulted in four viewing conditions (i.e., full captions + animation, full captions + static key frames, partial captions + animation, partial captions + static key frames). 62 participants were heterogeneously assigned to one of groups, and were instructed to complete the posttest measuring different aspects of comprehension (i.e., global, local, and inferential), as well as the questionnaire probing their perceived cognitive load (i.e., intrinsic, extraneous, and germane).
The results showed that the pictorial mode of a video had significant effect on L2 learners’ overall comprehension. Specifically, while viewing the video presented in static key frames, the viewers were better able to focus on verbal processing with manageable amounts of visuals; this advantage was not as manifest when viewing the video displayed in the form of animation. The qualitative data further revealed that while the participants enjoyed the rich visual stimuli in animation, its dynamic feature was considered distracting for verbal processing, thus debilitating overall comprehension. Besides the significant effect on overall comprehension, static key frames were particularly beneficial for L2 learners’ local comprehension when presented with full captions. The current study highlights the potential impact of verbal and pictorial presentation modes on L2 video comprehension, which hopefully may serve as a useful guide for designing video materials for specific comprehension goals.

Ayres, P., & Paas, F. (2007). Can the cognitive load approach make instructional animations more effective? Applied Cognitive Psychology: The Official Journal of the Society for Applied Research in Memory and Cognition, 21(6), 811-820.
Baddeley, A. D. (1999). Cognitive psychology: A modular course. Essentials of human memory. Psychology Press/Taylor & Francis (UK).
Baltova, I. (1999). Multisensory language teaching in a multidimensional curriculum: The use of authentic bimodal video in core French. Canadian Modern Language Review, 56(1), 31-48.
Basmmi, A. B. M. N., Abd Halim, S., & Saadon, N. A. (2020, July). Comparison of Web Services for Sentiment Analysis in Social Networking Sites. In IOP Conference Series: Materials Science and Engineering (Vol. 884, No. 1, p. 012063). IOP Publishing.
Betrancourt, M. (2005). The Animation and Interactivity Principles in Multimedia Learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (p. 287-296). Cambridge University Press.
Braverman, B. B., & Hertzog, M. (1980). The effects of caption rate and language level on comprehension of a captioned video presentation. American Annals of the Deaf, 125(7), 943-948.
Buck, G. (2001). Assessing listening. Cambridge University Press.
Calvo, M. (2017). Reflective drawing as a tool for reflection in design research. International Journal of Art & Design Education, 36(3), 261–272. https://doi.org/10.1111/jade.12161
Danan, M. (2004). Captioning and subtitling: Undervalued language learning strategies. Meta, 49(1), 67-77.
Diao, Y., Chandler, P., & Sweller, J. (2007). The effect of written text on comprehension of spoken English as a foreign language. The American journal of psychology (Vol. 120, No. 2, pp. 237-261).
d’Ydewalle, G., & Pavakanun, U. (1997). Could enjoying a movie lead to language acquisition? In New Horizons in Media Psychology. (pp. 145-155). VS Verlag für Sozialwissenschaften, Wiesbaden.
Gass, S., Winke, P., Isbell, D. R., & Ahn, J. (2019). How captions help people learn languages: A working-memory, eye-tracking study. Language Learning & Technology, 23(2), 84-104.
Guillory, H. G. (1998). The Effects of Keyword Captions to Authentic French Video on Learner Comprehension. CALICO Journal (Vol. 15, No. 1-3, pp. 89-108).
Hegarty, M. (1992). Mental animation: Inferring motion from static displays of mechanical systems. Journal of Experimental Psychology: Learning, Memory, and Cognition, 18(5), 1084-1102.
Hegarty, M. (2004). Dynamic visualizations and learning: Getting to the difficult questions. Learning and Instruction, 14(3), 343-351.
Hegarty, M., Kriz, S., & Cate, C. (2003). The roles of mental animations and external animations in understanding mechanical systems. Cognition and instruction, 21(4), 209-249.
Höffler, T. N., & Leutner, D. (2007). Instructional animation versus static pictures: A meta-analysis. Learning and instruction, 17(6), 722-738.
Kam, E. F., Liu, Y. T., & Tseng, W. T. (2020). Effects of modality preference and working memory capacity on captioned videos in enhancing L2 listening outcomes. ReCALL, 32(2), 213-230.
Kang, T., Arvizu, M. N. G., Chaipuapae, P., & Lesnov, R. O. (2019). Reviews of Academic English Listening Tests for Non-Native Speakers. International Journal of Listening, 33(1), 1-38.
Klepsch, M., Schmitz, F., & Seufert, T. (2017). Development and validation of two instruments measuring intrinsic, extraneous, and germane cognitive load. Frontiers in psychology, 8, 1997.
Kruger, J. L., & Steyn, F. (2014). Subtitles and eye tracking: Reading and performance. Reading Research Quarterly, 49(1), 105-120.
Lee, H., & Mayer, R. E. (2018). Fostering learning from instructional video in a second language. Applied Cognitive Psychology, 32(5), 648–654.
Leppink, J., Paas, F., Van der Vleuten, C. P., Van Gog, T., & Van Merriënboer, J. J. (2013). Development of an instrument for measuring different types of cognitive load. Behavior Research Methods, 45(4), 1058-1072.
Lowe, R. K. (2003). Animation and learning: Selective processing of information in dynamic graphics. Learning and Instruction, 13(2), 157-176.
Lowe, R. K., & Schnotz, W. (2014). Animation principles in multimedia learning. In R. E. Mayer (Ed.), Cambridge handbooks in psychology. The Cambridge handbook of multimedia learning (p. 513-546). Cambridge University Press.
Markham, P., Peter, L., & McCarthy, T. (2001). The effects of native language vs. target language captions on foreign language students’ DVD video comprehension. Foreign Language Annals, 34(5), 439-445.
Martin, A.J., & Evans, P. (2018). Load Reduction Instruction: Exploring a framework that assesses explicit instruction through to independent learning. Teaching and Teacher Education, 73, 203-214.
Mayer, R. E. (2002). Multimedia learning. In Brian H. Ross (Ed.), Psychology of learning and motivation (Vol. 41, pp. 85-139). Academic Press.
Mayer, R. E. (2005). Cognitive Theory of Multimedia Learning. In R. E. Mayer (Ed.), The Cambridge Handbook of Multimedia Learning (p. 31-48). Cambridge University Press.
Mayer, R. E. (2010). Techniques that reduce extraneous cognitive load and manage intrinsic cognitive load during multimedia learning. (R. Moreno, Ed.). In J. L. Plass, R. Moreno, & R. Brünken (Eds.), Cognitive load theory (pp. 131-152). Cambridge University Press.
Mayer, R. E. (2014). Cognitive theory of multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (p. 43-71). Cambridge University Press.
Mayer, R. E., Fiorella, L., & Stull, A. (2020). Five ways to increase the effectiveness of instructional video. Educational Technology Research and Development, 68(3), 837-852.
Mayer, R. E., Hegarty, M., Mayer, S., & Campbell, J. (2005). When static media promote active learning: Annotated illustrations versus narrated animations in multimedia instruction. Journal of Experimental Psychology: Applied, 11(4), 256-265.
Mayer, R. E., Heiser, J., & Lonn, S. (2001). Cognitive constraints on multimedia learning: When presenting more material results in less understanding. Journal of Educational Psychology, 93(1), 187-198.
Mayer, R. E., Lee, H., & Peebles, A. (2014). Multimedia learning in a second language: A cognitive load perspective. Applied Cognitive Psychology, 28(5), 653-660.
Meyer, K., Rasch, T., & Schnotz, W. (2010). Effects of animation’s speed of presentation on perceptual processing and learning. Learning and Instruction, 20(2), 136-145.
Mirzaei, M. S., Meshgi, K., Akita, Y., & Kawahara, T. (2017). Partial and synchronized captioning: A new tool to assist learners in developing second language listening skill. ReCALL, 29(2), 178-199.
Mitterer, H., & McQueen, J. M. (2009). Foreign subtitles help but native-language subtitles harm foreign speech perception. PLoS ONE, 13(3), Article e7785.
Moreno, R. (2007). Optimising learning from animations by minimising cognitive load: Cognitive and affective consequences of signalling and segmentation methods. Applied Cognitive Psychology, 21(6), 765-781.
Ng, H. K., Kalyuga, S., & Sweller, J. (2013). Reducing transience during animation: a cognitive load perspective. Educational Psychology, 33(7), 755-772.
Othman, J., & Vanathas, C. (2017). Topic familiarity and its influence on listening comprehension. The English Teacher, 14.
Paas, F., Renkl, A., & Sweller, J. (2003). Cognitive load theory and instructional design: Recent developments. Educational Psychologist, 38(1), 1-4.
Paas, F., Van Gerven, P. W., & Wouters, P. (2007). Instructional efficiency of animation: Effects of interactivity through mental reconstruction of static key frames. Applied Cognitive Psychology, 21(6), 783-793.
Park, M. (2004). The effects of partial captions on Korean EFL learners’ listening comprehension [Doctoral dissertation, The University of Texas at Austin].
Paivio, A. (1990). Mental representations: A dual coding approach. Oxford University Press.
Perez, M. M., Peters, E., Clarebout, G., & Desmet, P. (2014). Effects of Captioning on Video Comprehension and Incidental Vocabulary Learning. Language Learning & Technology, 18(1), 118-141.
Perez, M. M., Peters, E., & Desmet, P. (2014). Is less more? Effectiveness and perceived usefulness of keyword and full captioned video for L2 listening comprehension. ReCALL, 26(1), 21-43.
Perez, M. M., Van Den Noortgate, W., & Desmet, P. (2013). Captioned video for L2 listening and vocabulary learning: A meta-analysis. System, 41(3), 720-739.
Plass, J. L., & Jones, L. C. (2005). Multimedia Learning in Second Language Acquisition. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (p. 467–488). Cambridge University Press.
Price, K. (1983) Closed-captioned TV: An untapped resource. MATSOL Newsletter, 12: 1-8.
Rooney, K. (2014). The impact of keyword caption ratio on foreign language listening comprehension. International Journal of Computer-Assisted Language Learning and Teaching (IJCALLT), 4(2), 11-28.
Soruç, A. (2015). What Makes Redundant Presentation of Multimedia Learning Difficult? Procedia-Social and Behavioral Sciences, 191, 2844-2848.
Soussan, T., & Trovati, M. (2020). Sentiment urgency emotion conversion over time for business intelligence. International Journal of Web Information Systems, 16(5), 519-528.
Spanjers, I., Van Gog, T., & Van Merriënboer, J. (2010). A theoretical analysis of how segmentation of dynamic visualizations optimizes students' learning. Educational Psychology Review, 22(4), 411-423.
Stewart, M. A., & Pertusa, I. (2004). Gains to language learners from viewing target language closed‐captioned films. Foreign language annals, 37(3), 438-442.
Sweller, J. (2005). Implications of Cognitive Load Theory for Multimedia Learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (p. 19–30). Cambridge University Press.
Sweller, J., van Merriënboer, J. J., & Paas, F. (2019). Cognitive architecture and instructional design: 20 years later. Educational Psychology Review, 1-32.
Takacs, Z. K., & Bus, A. G. (2016). Benefits of motion in animated storybooks for children’s visual attention and story comprehension. An eye-tracking study. Frontiers in Psychology, 7, 1591.
Takacs, Z. K., Swart, E. K., & Bus, A. G. (2015). Benefits and pitfalls of multimedia and interactive features in technology-enhanced storybooks: A meta-analysis. Review of educational research, 85(4), 698-739.
Teng, F. (2019). Maximizing the potential of captions for primary school ESL students’ comprehension of English-language videos. Computer Assisted Language Learning, 32(7), 665-691.
Tragant, E. and Pellicer-Sánchez, A. (2019). Young EFL learners’ processing of multimodal input: Examining learners’ eye movements. System, 80, 212-223.
Vandergrift, L. (2007). Recent developments in second and foreign language listening comprehension research. Language Teaching, 40(3), 191.
Vanderplank, R. (1988). The value of teletext sub-titles in language learning. ELT Journal, 42(4), 272-281.
Vanderplank, R. (2010). Déjà vu? A decade of research on language laboratories, television and video in language learning. Language teaching, 43(1), 1-37.
Vanderplank, R. (2013). ‘Effects of’ and ‘effects with’ captions: How exactly does watching a TV programme with same-language subtitles make a difference to language learners? Language Teaching, 49(02), 235-250.
Wilawan, S. (2012). Fostering main idea comprehension among EFL learners through cognitive and metacognitive strategies. International Journal of Humanities and Social Science, 2(14), 46-54.
Winke, P., Gass, S. and Sydorenko, T. (2010) The effects of captioning videos used for foreign language listening activities. Language Learning & Technology, 14(1), 65-86.
Winke, P., Gass, S., & Sydorenko, T. (2013). Factors Influencing the Use of Captions by Foreign Language Learners: An Eye-Tracking Study. Modern Language Journal, 97(1), 254-275.
Yang, J., Chang, C., Lin, Y., & Shih, M. (2010, November). A study of the POS keyword caption effect on listening comprehension. In Proceedings of the 18th International Conference on Computers in Education (ICCE 2010). Putrajaya, Malaysia, 708-712.
Yeldham, M. (2018). Viewing L2 captioned videos: what’s in it for the listener? Computer Assisted Language Learning, 31(4), 367-389.