本論文主要針對具有彈性自動化的機器人發展與應用，提出了一種基於人類演示學習的機械手臂控制系統，其目的在於降低機械手臂自動化所需要的程式編譯複雜程度，以及增加多種操作功能開發的效率。硬體方面使用六軸串列式機械手臂作為實驗平台，搭配使用一台網路攝影機和一台深度攝影機蒐集影像資訊，在示教學習過程中對於人類演示動作及操作物件進行影像處理。軟體方面，經由深度攝影機偵測之人體骨架資訊，本論文透過正逆向運動學將人類演示的動作轉換成控制機械手臂的關節角度，並建立機械手臂運動控制模組。此外，基於YOLO(You only look once)演算法在多物件偵測具有快速及正確的優點，本論文使用此演算法開發操作物件偵測與辨識，在得出目標物件的類別資訊及所在位置之後，機械手臂可依照物件的種類啟動對應的運動控制模組，目標在於重現人類演示學習所示範的任務。最後，本論文經由所建置之虛擬環境和實際自動取放實驗來驗證所發展之機械手臂演示學習技術之可行性。

In order to contribute to the development and application of flexible automation using robots, this thesis aims to propose a robotic arm control system through learning from human demonstration (LfHD). The developed system is also anticipated to reduce the complexity of program compilation and increase the efficiency of several robot function developments. The hardware uses a six-axis tandem robotic arm as the experimental platform, and uses a webcam and a depth camera to observe the human demonstration for object manipulation. In terms of software, the Kinect 2 depth camera is used to detect the human skeleton, and the demonstrated actions by humans are converted into the controlled angle of the robot arm by forward and backward kinematics, which is established into the motion control module. In addition, based on the YOLO (You only look once) algorithm that is extremely fast and accurate in multiple objects detection, this thesis designs the recognition module to identify the type and position of the target objects while being manipulated in the human demonstration. After that, the system activates the motion control module in accordance with the object type and then completes the imitation task of human demonstration. Finally, through the build virtual reality simulation and the real experiment under the pick and place operation, the feasibility of the proposed LfHD for robotic manipulation can be verified.

[1] B. Albert, “Organisational applications of social cognitive theory,” Australian Journal of management, vol.13, no. 2, pp. 275-302, 1998.
[2] Y. LeCun, L. Bottou, Y. Bengio and P. Haffner, “Gradient-Based Learning Applied to Document Recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278-2324, Nov. 1998.
[3] C. Sang, et al., “Lead-through robot teaching,” 2013 IEEE Conference on Technologies for Practical Robot Applications, July 2013, pp. 1-4.
[4] L. Chen, Z. Wei, F. Zhao and T. Tao, “Development of a virtual teaching pendant system for serial robots based on ROS-I,” In Proc. IEEE International Conference on Cybernetics and Intelligent Systems (CIS) and IEEE Conference on Robotics, Automation and Mechatronics, China, 2017, pp.720-724.
[5] H.-I. Lin and Y.-H. Lin, “A Novel Teaching System for Industrial Robots,” Sensors, vol. 14, no. 4, pp.6012-6031, June 2014.
[6] P. Neto, J. N. Pires and A. P. Moreira, “Accelerometer-based control of an industrial robotic arm,” In Proc. 18th IEEE International Symposium on Robot and Human Interactive Communication, Japan, 2009, pp.1192-1197.
[7] P. Shenoy, K. J. Miller, B. Crawford and R. N. Rao, “Online electromyographic control of a robotic prosthesis,” IEEE Trans. Biomed. Eng., vol. 55, no. 3, pp.1128-1135, Mar. 2008.
[8] T. Yu, C. Finn, A. Xie, S. Dasari, T. Zhang, P. Abbeel and S. Levine, “One-shot imitation from observing humans via domain-adaptive meta-learning,” arXiv preprint arXiv: 1802.01557, Feb. 2018.
[9] C. Finn, T. Yu, T. Zhang, P. Abbeel and S. Levine, “One-shot visual imitation learning via meta-learning,” arXiv preprint arXiv: 1709.04905, Sep. 2017.
[10] M. Everingham, L. Van Gool, C.K.I. Williams, J. Winn and A. Zisserman, “The PASCAL Visual Object Classes Challenge,” Int'l J. Computer Vision, vol. 88, no. 2, pp. 303-338, June 2010.
[11] T.-Y. Lin, et al., “Microsoft COCO: Common objects in context,” In Proc. Eur. Conf. Comput. Vis., 2014, pp. 740-755.
[12] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” In Proc. IEEE Conf. Comput. Vis. Pattern Recognit., USA, June 2009, pp. 248-255.
[13] P. Felzenszwalb, R.Girshick, D. McAllester and D. Ramanan, “Object detection with discriminatively trained partbased models,” PAMI, vol. 32, no. 7, pp. 1627-1645, Sep. 2010.
[14] R. B. Girshick, J. Donahue, T. Darrell and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” In Proc., IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Ohio, Jun. 2014, pp. 580-587.
[15] G. Ross, “Fast R-CNN,” In Proc., of the IEEE international conference on computer vision, Chile, Dec. 2015, pp. 1440-1448.
[16] S. Ren, K. He, R. Girshick and J. Sun, “Faster R-CNN: Towards real-time object detection with region proposal networks,” In Proc. arXiv:1506.01497, Canada, Dec. 2015.
[17] J. Redmon, S. Divvala, R. Girshick and A. Farhadi, “You only look once: Unified real-time object detection,” In Proc. IEEE Conf. Comput. Vis. Pattern Recogn., Lasvegas, July 2016, pp. 779-788.
[18] W. Liu, et al., “SSD: Single shot multibox detector,” In Proc. Eur. Conf. Comput. Vis., Lasvegas, July 2016, pp. 21-37.
[19] A. Jain and C. C. Kemp, “EL-E: An assistive mobile manipulator that autonomously fetches objects from flat surfaces,” Auton. Robots, vol. 28, pp. 45-64, Jan. 2010.
[20] S. Levine, N. Wagener and P. Abbeel, “Learning contact-rich manipulation skills with guided policy search,” in Proc. arXiv preprint arXiv:1501.05611, May 2015.
[21] P. Devendrakumar and S. M. Kakade “Dynamic hand gesture recognition using kinect sensor,” In Proc. 2016 International Conference on Global Trends in Signal Processing, Information Computing and Communication (ICGTSPICC), India, Dec. 2016, pp. 448-453.
[22] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv:1409.1556, Sep. 2014.
[23] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke and A. Rabinovich, “Going deeper with convolutions,” In Proc. IEEE Computer Vision and Pattern Recognition, Boston, June 2015, pp. 1-9.
[24] J. Redmon and A. Farhadi, “YOLO9000: Better, faster, stronger,” In Proc. IEEE Conference on Computer Vision and Pattern Recognition, Hawaii, July 2017, pp. 6517-6525.
[25] A. Cherubini, R. Passama, A. Meline, A. Crosnier and P. Fraisse, “Multimodal control for human-robot cooperation,” In Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst., Japan, Nov. 2013, pp. 2202-2207.
[26] D. Kimura, R. Nishimura, A. Oguro and O. Hasegawa, “Ultra-fast Multimodal and Online Transfer learning on Humanoid Robots,” In Proc. ACM/IEEE International conference on Human-robot interaction, Japan, March 2013, pp. 165-166.
[27] I. E. Makrini, S. A. Elprama, J. Bergh, B. Vanderborght, A. J. Knevels, C. Jewell, F. Stals, G. Coppel, I. Ravyse, J. Potargent, and J. Berte, “Working with walt: How a cobot was developed and inserted on an auto assembly line,” IEEE Robotics & Automation Magazine, vol. 25, no. 2, pp. 51-58, May 2018.
[28] GTX1070顯示卡: https://tw.msi.com/Graphics-card/GEFORCE-GTX-1070-GAMING-X-8G/Gallery
[29] 羅技C922攝影機: https://www.logitech.com/zh-tw/product/c922-pro-stream-webcam
[30] Kinect for Xbox one: http://time.com/3195116/standalone-kinect-xbox-one/
[31] UR3 6軸機械手臂: https://www.universal-robots.com/
[32] UR3 機械手臂控制箱: http://www.nonead.com/intelligence_content/9657.html
[33] Robotiq 2f85平行夾爪: https://robotiq.com/products/2f85-140-adaptive-robot-gripper
[34] qb SoftHand五指夾爪: https://www.youtube.com/watch?v=0oX9eJsnVZc
[35] ]UR5機械手臂運動學: http://rasmusan.blog.aau.dk/files/ur5_kinematics.pdf
[36] J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,” In Proc. arXiv preprint arXiv:1804.02767, Apr. 2018.
[37] V-REP虛擬環境: http://www.coppeliarobotics.com/
[38] Tinkercard: https://www.tinkercad.com/