視網膜前膜(Epiretinal Membrane，ERM)是一種慢性眼疾，肇因於視網膜的表面出現微細缺口，導致黃斑部增生一層纖維薄膜而影響視力。黃斑前膜手術為最典型治療方法，惟部分患者在手術後的視力恢復效果不佳，重要的因素之一是缺乏執行內限界膜(Inner Limiting Membrane，ILM)剝離時機的判斷，而此診斷障礙乃因為缺乏判斷黃斑前膜是否影響視力的標準，而導致醫生無法做出診斷，並於早期進行內限界膜剝離手術以提升術後的視力恢復。

為了解決這個問題，本論文提出多種多流(multi-stream)神經網路，透過光學共輒斷層掃描(optical coherence tomography，OCT)、非侵入性光學共輒斷層血管掃描 (optical coherence tomography angiography，OCTA)、眼底螢光血管攝影(fundus fluorescence angiography，FFA)進行視力預測。我們收集454位患者上述三種影像並標記其視力資訊以訓練我們提出的多流神經網路，並以不同的影像輸入測試網路的效能。實驗結果顯示透過FFA全層、淺層、深層等三種影像在黃斑前膜患者的視力診斷中達到90.19%的準確性。最後，我們利用梯度權重類別活化映射(gradient-weighted class activation map，Grad-CAM) 可視化視力在OCT、 OCTA和FFA之間的特徵，幫助醫生進行診斷。

Epiretinal Membrane (ERM) is a chronic eye disease. It is caused by tiny gaps on the surface of the retina, causing a fibrous membrane to proliferate in the macula, which affects vision. Macular epithelial membrane surgery is the most typical treatment method, but some patients have poor vision recovery after surgery. One of the important factors is the lack of judgment on the timing of performing Inner Limiting Membrane (ILM) peeling. This problem is due to the lack of criteria for judging how ERM will affect the visual acuity, which causes doctors cannot make the diagnosis to perform ILM peeling in early time to improve postoperative vision recovery.
In order to solve this problem, this paper proposes a variety of multi-stream neural networks to find the correlation among visual acuity and optical coherence tomography (OCT), optical coherence tomography angiography (OCTA), fundus fluorescence angiography (FFA) images in ERMs patients. We collect the above three images of 454 patients and label their vision information to train our proposed multi-stream neural network, and then input different images to test the performance of the network. Experimental results show that by using three images of FFA full-thickness, superficial layer, and deep layer, the accuracy of the vision prediction of patients with epidermal membrane can reach 90.19%.
Finally, we apply gradient-weighted class activation map (Grad-CAM) to visualize the characteristics of vision among OCT, OCTA, and FFA to help doctors make a diagnosis.

[1] Xiao W, Chen X, Yan W, Zhu Z, He M. Prevalence and risk factors of epiretinal membranes: a systematic review and meta-analysis of population-based studies. BMJ open. 2017;7(9):e014644.
[2] Sandali O, El Sanharawi M, Basli E, et al. Epiretinal membrane recurrence: incidence, characteristics, evolution, and preventive and risk factors. Retina (Philadelphia, Pa). 2013;33(10):2032-2038.
[3] Novotny HR, Alvis DL. A method of photographing fluorescence in circulating blood in the human retina. Circulation. 1961;24:82-86.
[4] Huang D, Swanson EA, Lin CP, et al. Optical coherence tomography. science. 1991;254(5035):1178-1181.
[5] De Carlo TE, Romano A, Waheed NK, Duker JS. A review of optical coherence tomography angiography (OCTA). International journal of retina and vitreous. 2015;1(1):5.
[6] Suh MH, Seo JM, Park KH, Yu HG. Associations between macular findings by optical coherence tomography and visual outcomes after epiretinal membrane removal. American journal of ophthalmology. 2009;147(3):473-480. e473.
[7] Semeraro F, Morescalchi F, Duse S, Gambicorti E, Russo A, Costagliola C. Current trends about inner limiting membrane peeling in surgery for epiretinal membranes. Journal of ophthalmology. 2015;2015.
[8] Xiao W, Chen X, Yan W, Zhu Z, He M. Prevalence and risk factors of epiretinal membranes: a systematic review and meta-analysis of population-based studies. BMJ open. 2017;7(9): e014644-e014644.
[9] Stevenson W, Prospero Ponce CM, Agarwal DR, Gelman R, Christoforidis JB. Epiretinal membrane: optical coherence tomography-based diagnosis and classification. Clin Ophthalmol. 2016; 10:527-534.
[10] Akil H, Huang AS, Francis BA, Sadda SR, Chopra V. Retinal vessel density from optical coherence tomography angiography to differentiate early glaucoma, pre-perimetric glaucoma and normal eyes. PloS one. 2017;12(2).
[11] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, Dhruv Batra. Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization. arXiv:1610.02391, 2016.
[12] LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015;521(7553):436-444.
[13] Z. Zhao, P. Zheng, S. Xu, and X. Wu, “Object detection with deep learning: A review,” IEEE Trans. on Neur. Net. and Learn. Syst., vol. 30, pp. 3212–3232, 2019.
[14] Varga A, Steeneken HJ. Assessment for automatic speech recognition: II. NOISEX-92: A database and an experiment to study the effect of additive noise on speech recognition systems. Speech communication. 1993;12(3):247-251.
[15] Lai Y-H, Tsao Y, Lu X, et al. Deep learning–based noise reduction approach to improve speech intelligibility for cochlear implant recipients. Ear and hearing. 2018;39(4):795-809.
[16] Michael S. Ryoo, AJ Piergiovanni, Juhana Kangaspunta, Anelia Angelova. AssembleNet++: Assembling Modality Representations via Attention Connections. arXiv:2008.08072, 2020.
[17] Shen D, Wu G, Suk H-I. Deep learning in medical image analysis. Annual review of biomedical engineering. 2017; 19:221-248.
[18] Parmita M, Aaron L, Cecilia L, Magdalena B, Ariel R. Multilabel multiclass classification of OCT images augmented with age, gender and visual acuity data. bioRxiv. 2018.
[19] Kang, Eugene Yu-Chuan et al. “Deep Learning-Based Detection of Early Renal Function Impairment Using Retinal Fundus Images: Model Development and Validation.” JMIR medical informatics vol. 8,11 e23472. 26 Nov. 2020, doi:10.2196/23472
[20] Yim, J., Chopra, R., Spitz, T. et al. Predicting conversion to wet age-related macular degeneration using deep learning. Nat Med 26, 892–899 (2020).
[21] Michael G. Kawczynski, Thomas Bengtsson, Jian Dai, J. Jill Hopkins, Simon S. Gao, Jeffrey R. Willis; Development of Deep Learning Models to Predict Best-Corrected Visual Acuity from Optical Coherence Tomography. Trans. Vis. Sci. Tech. 2020;9(2):51.
[22] Yu-Yi Lin , Bo-Sin Wang , Yi-Ting Hsieh , Chung-Yen Su. A Deep Learning Approach to Predict the Visual Acuity for Epiretinal Membranes Patients based on the Optical Coherence Tomography Images
[23] Tao Hu, Honggang Qi, Qingming Huang, Yan Lu. See Better Before Looking Closer: Weakly Supervised Data Augmentation Network for Fine-Grained Visual Classification. arXiv:1901.09891,2019.
[24] Fan Zhang, Meng Li, Guisheng Zhai, Yizhao Liu. Multi-branch and Multi-scale Attention Learning for Fine-Grained Visual Categorization. arXiv:2003.09150, 2020.
[25] Peiqin Zhuang, Yali Wang, Yu Qiao. Learning Attentive Pairwise Interaction for Fine-Grained Classification. arXiv:2002.10191, 2020.
[26] Simonyan, K. and Zisserman, A. Two-stream convolutional networks for action recognition in videos. CoRR, abs/1406.2199, 2014. Published in Proc. NIPS, 2014.
[27] Christoph F, Axel P, Andrew Z. Convolutional Two-Stream Network Fusion for Video Action Recognition. arXiv:1406.2199,2014.
[28] Park, J.J., Kim, K.A., Nam, Y. et al. Convolutional-neural-network-based diagnosis of appendicitis via CT scans in patients with acute abdominal pain presenting in the emergency department. Sci Rep 10, 9556 (2020).
[29] Panwar, Harsh et al. “A deep learning and grad-CAM based color visualization approach for fast detection of COVID-19 cases using chest X-ray and CT-Scan images.” Chaos, solitons, and fractals vol. 140 (2020): 110190.
[30] Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun. Deep Residual Learning for Image Recognition. arXiv:1512.03385, 2015.
[31] Max Jaderberg, Karen Simonyan, Andrew Zisserman, Koray Kavukcuoglu. Spatial Transformer Networks. arXiv:1506.02025, 2015.
[32] Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, Quoc V. Le. RandAugment: Practical automated data augmentation with a reduced search space. arXiv:1909.13719, 2019
[33] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, Kaiming He. Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour. arXiv:1706.02677, 2017.