HOANGVAN NAM MINISTRY OF EDUCATION AND TRAINING HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY - Hoang Van Nam COMPUTERSCIENCE DIFFICULT SITUATIONS RECOGNITION SYSTEM FOR VISUALLY-IMPAIRED AID USING A MOBILE KINECT MASTER THESIS OF SCIENCE COMPUTER SCIENCE 2014B Ha Noi – 2016 MINISTRY OF EDUCATION AND TRAINING HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY Hoang Van Nam DIFFICULT SITUATIONS RECOGNITION SYSTEM FOR VISUALLY-IMPAIRED AID USING A MOBILE KINECT Department : COMPUTER SCIENCE MASTER THESIS OF SCIENCE … SUPERVISOR : Dr Le Thi Lan Ha Noi – 2016 CỘNG HÒA XÃ HỘI CHỦ NGHĨA VIỆT NAM Độc lập – Tự – Hạnh phúc BẢN XÁC NHẬN CHỈNH SỬA LUẬN VĂN THẠC SĨ Họ tên tác giả luận văn : ………………………………… …………… Đề tài luận văn: ………………………………………… …………… .… Chuyên ngành:…………………………… ………………… … Mã số SV:………………………………… ………………… … Tác giả, Người hướng dẫn khoa học Hội đồng chấm luận văn xác nhận tác giả sửa chữa, bổ sung luận văn theo biên họp Hội đồng ngày… .………… với nội dung sau: …………………………………………………………………………………………………… ………… ………………………………………………………………………………………… ………………………… ………………………………………………………………………… ………………………………………… ………………………………………………………… ………………………………………………………… ………………………………………… ………………………………………………………………………… ………………………… …………………………………………………………………………………… Ngày tháng năm Tác giả luận văn Giáo viên hướng dẫn CHỦ TỊCH HỘI ĐỒNG Declaration of Authorship I, Hoang Van Nam, declare that this thesis titled, ’Di cult situations recognition for visual-impaired aid using mobile Kinect’ and the work presented in it are my own I rm that: This work was done wholly or mainly while in candidature for a research degree at this University Where any part of this thesis has previously been submitted for a degree or any other quali cation at this University or any other institution, this has been clearly stated Where I have consulted the published work of others, this is always clearly attributed Where I have quoted from the work of others, the source is always given With the exception of such quotations, this thesis is entirely my own work I have acknowledged all main sources of help Where the thesis is based on work done by myself jointly with others, I have made clear exactly what was done by others and what I have contributed myself Signed: Date: i HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY Abstract International Research Institute MICA Computer Vision Department Master of Science Di cult situations recognition for visual-impaired aid using mobile Kinect by Hoang Van Nam By 2014, according to gures from some organization, here are more than one million people in the Vietnam living with sight loss, about 1.3% of Vietnamese people Although the big impact to the daily living, especially with the ability to move, read, communicate with another, only a small percentage of blind or visually impaired people live with assistive device or animal such as a dog guide Motivated by the signi cant changes in technology have take place in the last decade, especially in the introduction of varies types of sensors as well as the development in the eld of computer vision, I present in this thesis a di cult situations recognition system for visually impaired aid using a mobile Kinect This system is based on data captured from Kinect and using computer vision technique to detect obstacle At the current prototype, I only focused on detecting obstacle in the indoor environment like public building and two types of obstacle will be exploited: general obstacle in the moving way and staircases-which causes a big dangerous to the visually impaired people The 3D imaging techniques were used to detect the general obstacle including: plane segmentation, 3D point clustering and the mixed strategy between depth and color image is used to detect the staircase based on detecting the stair edges and its structure The system is very reliable with the detection rate is about 82.9% and the time to process each frame is 493 ms Acknowledgements I am so honor to be here the second time, in one of the nest university in Vietnam to write those grateful words to people who have been supporting, guiding me from the very rst moment when I was a university student until now, when I am writing my master thesis I am grateful to my supervisor, Dr Le Thi Lan, whose expertise, understanding, gener-ous guidance and support made it possible for me to work on a topic that was of great interest to me It was a pleasure to work with her Special thanks to Dr Tran Thi Thanh Hai, Dr Vu Hai and Dr Nguyen Thi Thuy (VNUA) and all of the members in the Computer Vision Department, MICA Institute for their sharp comments, guidance for my works which helps me a lot in how to study and to research in right way and also the valuable advices and encouragements that they gave to me during my thesis I would like to express my gratitude to Prof Veelaert Peter, Dr Luong Quang Hiep and Mr Michiel Vlaminck at Ghent University, Belgium for their supporting It’s been a great honor to cooperate and work with them Finally, I would especially like to thank my family and friends for their continues love, support they have given me through my life, helps me pass through all the frustrating, struggling, confusing Thanks for everything that helped me get to this day Hanoi, 19/02/2016 Hoang Van Nam iii Contents Declaration of Authorship Abstract i ii Acknowledgements iii Contents iv List of Figures vi List of Tables ix Abbreviations x Introduction 1.1 Motivation 1.2 De nition 1.2.1 Assistive systems for visually impaired people 1.2.2 Di cult situations 1.2.3 Mobile Kinect 1.2.4 Environment Context 1.3 Di cult Situations Recognition System 1.4 Thesis Contributions Related Works 2 11 12 13 14 2.1 Assistive systems for visually impaired people 2.2 RGB-D based assistive systems for visually impaired people 2.3 Stair Detection Obstacle Detection 14 18 19 25 3.1 3.2 3.3 3.4 3.5 Overview Data Acquisition Point Cloud Registration Plane Segmentation Ground & Wall Plane Detection iv 25 26 27 30 32 Contents 3.6 Obstacle Detection v 32 3.7 Stair Detection 3.7.1 Stair de nition 3.7.2 Color-based stair detection 3.7.3 Depth-based stair detection 3.7.4 Result fusion 3.8 Obstacle information representation Experiments 34 34 35 45 46 48 49 4.1 Dataset 4.2 Di cult situation recognition evaluation 4.2.1 Obstacle detection evaluation 4.2.2 Stair detection evaluation Conclusions and Future Works 49 51 51 53 58 5.1 Conclusions 5.2 Future Works Publications 58 59 60 Bibliography 61 List of Figures 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 1.10 1.11 1.12 1.13 1.14 1.15 1.16 2.1 2.2 2.3 2.4 2.5 2.6 A Comprehensive Assistive Technology (CAT) Model provided by [12] A model for activities attribute and mobility provided by [12] Distribution of frequencies of head-level accidents for blind people [18] Distribution of frequencies of tripping resulting a fall [18] A typical example of depth image (A) raw depth image, (B) depth image is visualized by jet color map and the colorbar shows the real distance with each color value, (C) Reconstructed 3D scene A stereo images that taken from OpenCV library and the calculated depth image (A) left image, (B) right image, (C) depth image (disparity map) Some existed stereo camera From left to right: Kodak stereo camera, View-Master Personal stereo camera, ZED, Duo 3D Sensor Time of ight systems from [3] Some ToF cameras From left to right: DepthSense, Fotonic, Microsoft Kinect v2 Structured light cameras From left to right: PrimeSense, Microsoft Kinect v1 Structured light systems from [3] Figure from [16], (A) raw IR image with pattern, (B) depth image Figure from [16] (A) Errors for structured light cameras, (B) Quantization errors in di erent distances of a door: 1m, 3m, 5m Prototype of system using mobile Kinect, (A) Kinect with battery and belt, (B) Backpack with laptop (C)Mobile Kinect is mounted on human body Two di erent environments that I tested with (A) Our o ce build (B) Nguyen Dinh Chieu secondary school Prototype of our obstacle detection and warning system Robot-Assisted Navigation from [17] (A) RFID tag, (B) Robot (C) Navigation NXT Robot System from [6] (A) The system’s Block Diagram, (B) NXT Robot Mobile robot from [22] [21] BrainPort vision substitution device [32] Obstacle detection process from [30] Stair detection from [26] (A) Input image (B)(C)Frequency as a output of Gabor lter (D)Stair detection result vi 4 7 8 9 10 11 12 13 15 16 16 18 20 21 List of Figures 2.7 A near-approach for stair detection in [13] (A) Input image with detected 2.8 2.9 2.10 3.1 3.2 3.3 3.4 3.5 stair region, (B) Texture energy, (C)Input image with detected lines are stair candidates, (D)Optical ow maps in this image, there is a signi cant changing in the line in the edge of stair Example of segmentation and classi cation in [24] Stair modeling(left) and features in each plane [24] Stair detection algorithm proposed in [29] (A) Detected line in the edge image (using color infomation) (B) Depth pro les in each line (red line: pedestrian crosswalk, blue: down stair, green: upstair) Obstacle Detection Flowchart vii 22 23 23 24 26 Kinect mounted on body 27 Coordinate Transformation Process 28 Kinect Coordinate 29 Point Cloud rotation using normal vector of ground plane (while arrow): left: before rotating, right: after rotating 30 3.6 Normal vector estimation algorithms [15] (a) Normal vector of the center point can be calculated by a cross product of two vectors of four neighbor points (red), (b) Normal vector estimation in a scene 31 3.7 Plane segmentation result using algorithm proposed in [15] Each plane is represented by a distinctive color 31 3.8 Detected Ground and Walls plane (ground: blue, wall: red) 33 3.9 Human Segmentation Data by Microsoft Kinect SDK (a) Color Image, (b) Human Mask 34 3.10 Detected Obstacles (a) Color Image, (b) Detected Obstacles 34 3.11 Model of stair 35 3.12 Coordinate transformation models from [7] 36 3.13 Projective chirping: a) A real world object that generate a projection with "chirping" - "periodicity-in-perspective" b) Center raster of image c) Best t projective chirp 38 3.14 A pinhole camera model with stair 38 3.15 A vertical Gabor lter kernel 39 3.16 Gabor lter applied on a color image (a) Original (b) Filtered Image 40 3.17 Thresholding the grayscale image (a) Original (b) Thresholded Image 40 3.18 Example of thinning image using morphological 41 3.19 Thresholding the grayscale image (a) Original (b) Thresholded Image 42 3.20 Six points vote for a line will make an intersection in Hough space, this intersection has higher intensity than neighbor pixels 42 3.21 Hough space (a) Line in the original space (b) Three curves vote for this line in Hough space 43 3.22 Hough space on stair image (a) Original image (b) Hough space 43 3.23 Chirp pattern detection (a) Hough space (b) Original image with detected chirp pattern 44 3.24 Point cloud of stair (a) Original color image (b)Point cloud data created from color and depth image 45 3.25 Detected steps 46 3.26 Detected planes 47 3.27 Detected stair on point cloud 47